first auto cancel of limit order not working RC1

[DaveE]

In RC1 the first time that
EnterLongLimit(int quantity, double limitPrice)
or
EnterShortLimit(int quantity, double limitPrice)
is called, and the order is not filled, the limit order does not seem to be auto-cancelled.
This causes the next limit order to be rejected, by NT's 'Internal Order Handling Rules' (if its in the opposite direction).
Subsequent cases are auto-cancelled

eg EnterLongLimit(int quantity, double limitPrice) one bar (first time no fill), followed by EnterShortLimit(int quantity, double limitPrice) next bar will cause the error

An Enter() method to submit an entry order at...has been ignored. Please search on the term 'Internal Order Handling Rules' in the Help Guide for detailed explanation.

In my opinion NT should change this error to state exactly which of the various rules has been broken. I am assuming in this case it must be this rule (since the position is Flat)

The strategy position is flat and an order submitted by an enter method (EnterLongLimit() for example) is active and the order is used to open a position in the opposite direction

An I am assuming that the rule is broken because the previous limit order is still active, but this is because the 'managed orders' "cancel expired orders' feature has failed to work at the first unfilled order.
This problem might have existed in previous versions, but I did not identify the cause before.

I attach a simple strategy that demonstrates the bug.
This strategy starts with a small distance from close to limit price, so the first few orders are likely to be filled, but as the distance increases each bar you will soon get a limit that is not filled.

Steps to reproduce the bug:
Run the attached strategy (say in Strategy Analyzer) any instrument/TF (I used Forex, 15 minute) so long as there are at least 50 bars in your data.
Check the output (search for 'ignored').

Expected result:
When EnterLongLimit(int quantity, double limitPrice)
or
EnterShortLimit(int quantity, double limitPrice)
is not filled it should be auto-cancelled at the end of the bar (so an order the next bar in the opposite direction will be allowed).
With TraceOrders=true I expect to see 'Cancelled expired order' after each bar where limit order was not filled.

Actual result:
The first time this happens the auto-cancel does not work (subsequent times it does).
(search Output for 'ignored' you will see the error at the first occurrence)

Example output:

Flat at 22 Sep 2016 14:05: EnterLongLimit(100000, 100.431)
22/09/2016 14:05:00 Strategy 'sTestLimitError1/-1': Entered internal SubmitOrderManaged() method at 22/09/2016 14:05:00: BarsInProgress=0 Action=Buy OrderType=Limit Quantity=100,000 LimitPrice=100.43'1 StopPrice=0 SignalName='' FromEntrySignal=''
Flat at 22 Sep 2016 14:30: EnterShortLimit(100000, 100.857)
22/09/2016 14:30:00 Strategy 'sTestLimitError1/-1': Entered internal SubmitOrderManaged() method at 22/09/2016 14:30:00: BarsInProgress=0 Action=SellShort OrderType=Limit Quantity=100,000 LimitPrice=100.85'7 StopPrice=0 SignalName='' FromEntrySignal=''
22/09/2016 14:30:00 Strategy 'sTestLimitError1/-1': Ignored SubmitOrderManaged() method at 22/09/2016 14:30:00: BarsInProgress=0 Action=SellShort OrderType=Limit Quantity=100,000 LimitPrice=100.85'7 StopPrice=0 SignalName='Sell short' FromEntrySignal='' Reason='An Enter() method to submit an entry order has been ignored. Please search on the term 'Internal Order Handling Rules' in the Help Guide for detailed explanation.'
Strategy 'sTestLimitError1/-1': An Enter() method to submit an entry order at '22/09/2016 14:30:00' has been ignored. Please search on the term 'Internal Order Handling Rules' in the Help Guide for detailed explanation.
22/09/2016 14:30:00 Strategy 'sTestLimitError1/-1': Cancelled expired order: BarsInProgress=0, orderId='NT-00006-486' account='Backtest' name='Buy' orderState=Working instrument='USDJPY' orderAction=Buy orderType='Limit' limitPrice=100.431 stopPrice=0 quantity=100,000 tif=Gtc oco='' filled=0 averageFillPrice=0 onBehalfOf='' id=-1 time='2016-09-22 14:05:00' gtd='2099-12-01' statementDate='2016-09-26'
Flat at 22 Sep 2016 14:55: EnterLongLimit(100000, 100.272)

[NinjaTrader_ChelseaB]

Hello DaveE,

I have simplified your script to test the LiveUntilCancelled being set to false when not used and to ensure that the cancel is working.

This script places two orders only and has TraceOrders set to true so that you may see in the Output window if the order is being cancelled.

After the first order is submitted without the liveUntilCancelled overload (so this defaults to false), the second order is placed on the bar after this order will be cancelled (so two bars after the first submission).

Are you finding that if you test this script on your end, that the cancellation does not occur and the first long order remains working or is filled and the short order is ignored?

[DaveE]

I cant see your script as a simplification of mine. Yours sends the 2nd limit order 2 bars after the first. But the error I am reporting occurs when the 2nd limit order is on the next bar.
If you change your code to

firstEntryBar = CurrentBar + [B]1[/B];

i suspect you will see the error.

But did you not try running my strategy?
The reason I did not want to trigger the error straight away is to simulate what often happens in a real strategy, where it may take many limit orders before one does not get filled.

The reason I placed more than 2 orders is because I wanted to highlight the inconsistency between the first occurance (error, order rejected by NT) and subsequent occurance (no error, order accepted by NT). Since the error never occurs more than once in this type of strategy and the error message is obscure, I suspect this bug in the 'managed orders' system may have been around for a long time.

[DaveE]

Clarification of original post based on further testing:

In RC1 the first occurance in a strategy run of

EnterLongLimit(int quantity, double limitPrice) on bar x is not filled, and followed by
EnterShortLimit(int quantity, double limitPrice) on bar x+1

OR

EnterShortLimit(int quantity, double limitPrice) on bar x is not filled, and followed by
EnterLongLimit(int quantity, double limitPrice) on bar x+1

... causes the limit order at bar x+1 to be rejected, by NT's 'Internal Order Handling Rules'
...even though the MarketPosition is Flat for all these orders and the documentation says that unfilled orders are auto-cancelled.

But subsequent occurrences work fine.

[NinjaTrader_ChelseaB]

Hello DaveE,

This is expected behavior, until the bar closes and the order is cancelled and the cancellation is completed, the new order will be ignored.

You could instead wait for the cancellation to be complete and trigger the next order in OnOrderUpdate when the order's OrderState is .Cancelled.

The script I have provided you demonstrates how to place an order in the opposite direction when an order is cancelled at the close of a bar.

You have mentioned that the cancellation at the end of the bar is not taking place. Are you able to demonstrate this behavior using the script I have provided you?

[DaveE]

Here's how I understand your current position

Assumption A:
Use of EnterLongLimit(...) on one bar (not filled), followed by use of EnterShortLimit(...) on next bar, should always result in an error, logged by trace as ' Ignored SubmitOrderManaged() method', causing the EnterShortLimit(...) to be ignored (rejected).

Whereas my position is

Assumption B:
If EnterShortLimit(...) is used then the 'managed orders' system should ensure that any expired order in the opposite direction is first cancelled, before submitting the new order.

But what is currently implemented in RC1 seems to be an inconsistent version of your assumption A. You can see this by running the attached strategy (I used 60 min TF on 24x7). All this strategy does is toggle, on alternate bars, between a EnterLongLimit at a low price that should not be hit, and a EnterShortLimit at a high price that should not be hit.
If your assumption A was correct then the 'expected behavior' would be: Every EnterShortLimit attempt should log an error (and be ignored). But in reality the error only gets logged the first time

Flat at bar 0, 26 Sep 2016 06:00: EnterLongLimit(100000, 0.001)
26/09/2016 06:00:00 Strategy 'sTestLimitError4/-1': Entered internal SubmitOrderManaged() method at 26/09/2016 06:00:00: BarsInProgress=0 Action=Buy OrderType=Limit Quantity=100,000 LimitPrice=0.00'1 StopPrice=0 SignalName='' FromEntrySignal=''
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Submitted
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Accepted
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Working
Flat at bar 1, 26 Sep 2016 07:00: EnterShortLimit(100000, 202.156)
26/09/2016 07:00:00 Strategy 'sTestLimitError4/-1': Entered internal SubmitOrderManaged() method at 26/09/2016 07:00:00: BarsInProgress=0 Action=SellShort OrderType=Limit Quantity=100,000 LimitPrice=202.15'6 StopPrice=0 SignalName='' FromEntrySignal=''
26/09/2016 07:00:00 Strategy 'sTestLimitError4/-1': Ignored SubmitOrderManaged() method at 26/09/2016 07:00:00: BarsInProgress=0 Action=SellShort OrderType=Limit Quantity=100,000 LimitPrice=202.15'6 StopPrice=0 SignalName='Sell short' FromEntrySignal='' Reason='An Enter() method to submit an entry order has been ignored. Please search on the term 'Internal Order Handling Rules' in the Help Guide for detailed explanation.'
Strategy 'sTestLimitError4/-1': An Enter() method to submit an entry order at '26/09/2016 07:00:00' has been ignored. Please search on the term 'Internal Order Handling Rules' in the Help Guide for detailed explanation.
26/09/2016 07:00:00 Strategy 'sTestLimitError4/-1': Cancelled expired order: BarsInProgress=0, orderId='NT-00000-3074' account='Backtest' name='Buy' orderState=Working instrument='USDJPY' orderAction=Buy orderType='Limit' limitPrice=0.001 stopPrice=0 quantity=100,000 tif=Gtc oco='' filled=0 averageFillPrice=0 onBehalfOf='' id=-1 time='2016-09-26 06:00:00' gtd='2099-12-01' statementDate='2016-09-29'
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=CancelPending
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=CancelSubmitted
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Cancelled

But now watch what happens when the same sequence is repeated

Flat at bar 2, 26 Sep 2016 08:00: EnterLongLimit(100000, 0.001)
26/09/2016 08:00:00 Strategy 'sTestLimitError4/-1': Entered internal SubmitOrderManaged() method at 26/09/2016 08:00:00: BarsInProgress=0 Action=Buy OrderType=Limit Quantity=100,000 LimitPrice=0.00'1 StopPrice=0 SignalName='' FromEntrySignal=''
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Submitted
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Accepted
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Working
Flat at bar 3, 26 Sep 2016 09:00: EnterShortLimit(100000, 201.728)
26/09/2016 09:00:00 Strategy 'sTestLimitError4/-1': Entered internal SubmitOrderManaged() method at 26/09/2016 09:00:00: BarsInProgress=0 Action=SellShort OrderType=Limit Quantity=100,000 LimitPrice=201.72'8 StopPrice=0 SignalName='' FromEntrySignal=''
26/09/2016 09:00:00 Strategy 'sTestLimitError4/-1': Cancelled expired order: BarsInProgress=0, orderId='NT-00001-3074' account='Backtest' name='Buy' orderState=Working instrument='USDJPY' orderAction=Buy orderType='Limit' limitPrice=0.001 stopPrice=0 quantity=100,000 tif=Gtc oco='' filled=0 averageFillPrice=0 onBehalfOf='' id=-1 time='2016-09-26 08:00:00' gtd='2099-12-01' statementDate='2016-09-29'
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=CancelPending
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=CancelSubmitted
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Cancelled

So far bars 0 and 1 have worked according to your assumption A, but bars 2 and 3 at first glance seem worked according to my assumption B. Bar 3 did not generate an error, but got ignored anyway. Before I added the OnOrderUpdate code to report OrderState I had assumed that the subsequent EnterShortLimit were all being executed (because no error is produced).

Conclusion: If NT is going to stick with assumption A (rather than moving to my user friendly assumption B) then the bug that needs fixing is failure to log an error (except for the first occurrence).
If an error was logged at every occurrence then this would force users to implement your suggestion

wait for the cancellation to be complete and trigger the next order in OnOrderUpdate when the order's OrderState is .Cancelled.

[NinjaTrader_ChelseaB]

Hello DaveE,

I'd like to clarify, do you still feel that the liveUntilCancelled as false is not causing the first placed order to be cancelled after the bar it is submitted on closes?

Also, the output you have shown (in the 4th quote box) may show that there is a bug or inconsistency and I will further test by calling an order in the opposite direction before the previous order is cancelled.

If you have included the complete output, this shows that the sell order never entered a submit, or accepted, or working state.

Order is submitted as buy limit

26/09/2016 08:00:00 Strategy 'sTestLimitError4/-1': Entered internal SubmitOrderManaged() method at 26/09/2016 08:00:00: BarsInProgress=0 Action=Buy OrderType=Limit Quantity=100,000 LimitPrice=0.00'1 StopPrice=0 SignalName='' FromEntrySignal=''
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Submitted
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Accepted
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Working

Order is submitted as sell limit though buy limit is still working

Flat at bar 3, 26 Sep 2016 09:00: EnterShortLimit(100000, 201.728)
26/09/2016 09:00:00 Strategy 'sTestLimitError4/-1': Entered internal SubmitOrderManaged() method at 26/09/2016 09:00:00: BarsInProgress=0 Action=SellShort OrderType=Limit Quantity=100,000 LimitPrice=201.72'8 StopPrice=0 SignalName='' FromEntrySignal=''

Buy limit cancellation is triggered after the sell limit is submitted

26/09/2016 09:00:00 Strategy 'sTestLimitError4/-1': Cancelled expired order: BarsInProgress=0, orderId='NT-00001-3074' account='Backtest' name='Buy' orderState=Working instrument='USDJPY' orderAction=Buy orderType='Limit' limitPrice=0.001 stopPrice=0 quantity=100,000 tif=Gtc oco='' filled=0 averageFillPrice=0 onBehalfOf='' id=-1 time='2016-09-26 08:00:00' gtd='2099-12-01' statementDate='2016-09-29'

Buy limit is cancelled

OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=CancelPending
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=CancelSubmitted
OnOrderUpdate 29 Sep 2016 13:22:11 order: Name=Buy, OrderState=Cancelled

Sell limit is never accepted.

I appreciate your patience while I look further into this. If I am able to reproduce orders being submitted in the opposite direction before the previous order is cancelled I will submit a bug report about this to our development department.

[DaveE]

Hello Chelsea,
re

do you still feel that the liveUntilCancelled as false is not causing the first placed order to be cancelled after the bar it is submitted on closes?

I never mentioned liveUntilCancelled, It was you who introduced that.

Regarding Bar 3 (same for all subsequent odd numbered bars)
re

this shows that the sell order never entered a submit, or accepted, or working state.

Yes. But but before I had explicitly added code to track OrderState this was not noticable.
Because the TraceOrders only shows

Entered internal SubmitOrderManaged() method at 26/09/2016 09:00:00: BarsInProgress=0 Action=SellShort OrderType=Limit ...

But it does not log that the order is ignored (neither does it put an error in Log). So in my original post I assumed it had been processed.

Re

If I am able to reproduce orders being submitted in the opposite direction before the previous order is cancelled I will submit a bug report about this to our development department.

I am no longer saying that limit orders in the opposite direction are being processed, I was assuming that because of the lack of subsequent error reporting.
So the bug is the failure to report subsequent 'order reversal ignored' errors.

Let me give an example (more realistic) scenario of how serious this is:
User takes the SampleMACrossOver saves it as SampleMACrossOverLimit and simply changes the EnterLong to EnterLongLimit (perhaps price a few ticks below), and changes the EnterShort to EnterShortLimit (perhaps price a few ticks above).
User runs this live with say a few hundred bars historical. Its likely that at some point in the historical bars there will have been a crossover one way followed by a crossover the opposite way the next bar. If the first limit was not filled this will cause the error 'Ignored SubmitOrderManaged()...' to be logged. Since this error only gets logged once the user might assume its just a one-off glitch and otherwise their strategy is doing fine (since no more errors are reported). Say when live there is another crossover up then down, and the EnterLongLimit was not filled. The user will expect a EnterShortLimit to be processed, but nothing will happen since NT has silently ignored the 'error' (without recording the error in any way). This error should be recorded by TraceOrders and in the Log just like the first occurrence.

[NinjaTrader_ChelseaB]

Hello DaveE,

Below is what I am responding to from post 1. (Trying to take this one step at a time. As this revolves around whether the cancellation works or does not work, I feel we should address this first.)

Expected result:
When EnterLongLimit(int quantity, double limitPrice)
or
EnterShortLimit(int quantity, double limitPrice)
is not filled it should be auto-cancelled at the end of the bar (so an order the next bar in the opposite direction will be allowed).
With TraceOrders=true I expect to see 'Cancelled expired order' after each bar where limit order was not filled.

Actual result:
The first time this happens the auto-cancel does not work (subsequent times it does).
(search Output for 'ignored' you will see the error at the first occurrence)

Are you not referring to the autocancellation that should happen if the order is submitted with liveUntilCancelled as false and does not fill before the bar it was submitted on closes?

Is this behavior no longer happening and we are addressing a new issue?

Currently, I have not yet had time to test to see if subsequent orders are experience misbehavior with the internal handling rules and allowing orders to be placed in the opposite direction before the previous order has been cancelled.
(Or if the error message that should be appearing is not appearing as you have mentioned)


*Note that NinjaTrader 8 beta items fall secondary in priority to NinjaTrader 7 inquiries.
Per the NinjaTrader 8 beta help guide: Bug reports will be reviewed by our support and development staff inside of a 48 hour period.
See the link below on Bug reports and providing feedback.
http://ninjatrader.com/support/helpG...n-us/?beta.htm

[DaveE]

My original post was based on my Assumption B.
When I said in the original post

The first time this happens the auto-cancel does not work

I meant that I assumed the EnterLongLimit(int quantity, double limitPrice)
would be automatically cancelled the next bar if not renewed, AND that if the next bar requested a limit in the opposite direction then this auto-cancellation would be done before the reversal limit order was processed.
So from the point of view of Assumption B the auto-cancel is not working in the correct sequence, even though it does work in the correct bar.
When I said

(subsequent times it does)

This was an assumption based on the lack of subsequent error reporting from NT.

I think it might be best to finish with this thread and I may raise:

• A new post just reporting the lack of subsequent errors as a bug.
• A new 'discussion' post requesting that switching Limit order direction be made more user friendly in the 'managed orders' environment.