Overfill

[tradetree]

I tried getting this handled via support but there appears to be a problem with which email address I use, as I use two of them.

I have been receiving overfill errors in my strategy. Here is a piece of log (I pulled the account numbers):

10/12/11 12:21 PM Strategy An over fill was detected on order 'Order='742054610/U' Name='Sell' State=Filled Instrument='TF 12-11' Action=Sell Limit price=0 Stop price=0 Quantity=1 Strategy='Tra******et2' Type=Market Tif=Day Oco='' Filled=1 Fill price=696 Token='bc90046fb35f4fab8b3add8a8decd24b' Gtd='12/1/2099 12:00:00 AM'' generated by strategy 'Tra******et2/1f9809de0e66485ca08774b272abe1a5' : This strategy will be disabled and NinjaTrader will attempt to cancel/close any strategy generated orders and positions. Please check your account orders and positions and take any appropriate action.
10/12/11 12:21 PM Execution Execution='TF 12-11/0000d7eb.4e94e8fc.01.01' Instrument='TF 12-11' Account='U' Exchange=Nybot Price=696 Quantity=1 Market position=Short Operation=Insert Order='742054610' Time='10/12/2011 12:21:51 PM'
10/12/11 12:21 PM Order Order='742054610/U' Name='Sell' New state=Filled Instrument='TF 12-11' Action=Sell Limit price=0 Stop price=0 Quantity=1 Type=Market Filled=1 Fill price=696 Error=NoError Native error=''
10/12/11 12:21 PM Order Order='742054610/U' Name='Sell' New state=Working Instrument='TF 12-11' Action=Sell Limit price=0 Stop price=0 Quantity=1 Type=Market Filled=0 Fill price=0 Error=NoError Native error=''
10/12/11 12:21 PM Order Order='742054610/U' Name='Sell' New state=Accepted Instrument='TF 12-11' Action=Sell Limit price=0 Stop price=0 Quantity=1 Type=Market Filled=0 Fill price=0 Error=NoError Native error=''
10/12/11 12:21 PM Default Order Event Warning:Attribute 'Outside Regular Trading Hours' is ignored based on the order type and destination. PlaceOrder is now being processed. 2109

I have several questions, all for "managed" orders:
1) If you perform an ExitLong and then another ExitLong on the same bar, does that trigger the overfill? (and please don't send me to the canned documentation on overfill as I've read all that).
2) If you perform an ExitLong and then an ExitLongLimit does that trigger an overfill?
3) Why do I get a Warning about "Outside Regular Trading Hours" during the regular hours of the day?

[NinjaTrader_RyanM1]

Hi tradetree,

Overfills happen anytime there are orders resting near the market, an order is marked for cancellation, and then it is filled before it can be cancelled.

1) If you perform an ExitLong and then another ExitLong on the same bar, does that trigger the overfill?

No, shouldn't be an overfill here because the strategy checks its position before submitting multiple closing orders. The second time you place it, should receive message
Ignored PlaceOrder() Reason='This was an exit order but no position exists to exit'

2) If you perform an ExitLong and then an ExitLongLimit does that trigger an overfill?

Yes, you could have an overfill here if exit long is submitted at same time a long limit order is working near the market. The limit order can be filled before it can be properly cancelled by the strategies internal order management.

3) Why do I get a Warning about "Outside Regular Trading Hours" during the regular hours of the day?

This message comes from IB and essentially means they they ignore certain attributes about the order. Most likely the TimeInForce setting in this case. IB will have more details on this message.

[tradetree]

Ok, I think #2 was taking place in my code. Can I test my fix with market replay, will that catch the same condition?

[NinjaTrader_RyanM1]

Yes, overfills can take place in market replay as well. The timing on overfills is usually when PendingCancel > Filled happens in the same second, so it may be challenging finding the exact conditions for it.

[marcow]

I also get a lot of overfills when I run a strategy on fast Renko bars, eg. 2 Renko
This strategy only uses managed orders.

consider the following scenario:
strategy is in Long position, indicator(s) give a short signal. According to the strategy code the following steps are taken all on close of bar.
1) Bar x: Exit long position with ExitLong()
2) Bar x+1:Enter Short position with EnterShort()

step 1 will create a sell order for 1 contract
step 2 will create a sell order for 1 contract (to exit long position), and another sell order for 1 contract (for short entry)

Problem is that with 2 Renko bars, it is very likely - in rapid price movements - that multiple bars will be created at the almost same time.This results in step 1 and step 2 beiing executed at almost the same time. This is when I get the overfill errors. Probably because step 2 tries to exit a long position which doesn't exist anymore because of step 1. It appears that the strategy thinks it is still in a long position and that this information is not updated fast enough.

As a solution I put a delay in my strategy code so that step 1 and step 2 are always 5 seconds apart.

In my opinion this is a bug in NinjaTrader managed orders.

step 2 should only be executed when the strategy position information is fully updated.So after step 1 strategy position information should be "Flat". NinjaTrader should wait for this updated strategy position before it executes step 2, no matter how fast they are triggered. Maybe a solution for this would be to hardcode a certain delay between 2 consecutive orders ?

Marco

[tradetree]

I agree. I have coded state machines that handle just this type of scenario and it is not that hard. But as soon as you call something a bug, the walls go up and the cooperation ends. :-) I can understand NT can't take on every wish, but with something as important as order execution it sure is a priority for traders. What they (NT) really want is for you to do your own unmanaged order execution as soon as you have any alternate situation. The problem I have is that it is hard enough handling the strategy code without also performing order handling. The managed case should be able to say, "I have a long and two sells, put the first sell against the long, and the second sell is a short."

[NinjaTrader_Brett]

Will forward your comments to development.

Thanks very much guys.

[brima]

Is there any news on that issue?

Because this overfill subject bothers us too with fast Renko bars.

[NinjaTrader_RyanM1]

brima, Unfortunately there is no news here. Overfills are considered a strategy coding issue. If you are not already, consider working with advanced handling to provide more control on order submission, modification and cancellation.

[marcow]

Overfills are considered a strategy coding issue

So if a someone without C# - programming knowledge creates a very simple and basic strategy with the NinjaTrader strategybuilder, the strategybuilder generates incomplete or faulty code ?

Let me explain; In the strategybuilder create a strategy with a ZLEMA which goes Long at crossAbove level A, exits Long at crossBelow level B, enters short at crossBelow level C and exits short at crossAbove level D. (conditions : A < B , C > D , A and B > C and D ) Run this strategy on for example a 2 Renko chart and you will probably get the overfills (of course only on shorter periods of ZLEMA )

If the overfills are indeed a "coding issue" please update the strategy builder so it generates code that doesn't generate the overfill errors on faster Renko charts.

[NinjaTrader_RyanM1]

marcow,

Thank you for the feedback. One thing with the strategy wizard that may help with overfills is making sure that you never have, for example, ExitLong, EnterShort in the same set. Since Enter methods handle the reversal automatically, the ExitLong could contribute to overfills because you're submitting duplicate orders.

If you have a scalping style strategy with orders frequently resting near the market, then you are going to find the control offered by the strategy wizard to be limiting, and can work with code and advanced handling to have more control.

[marcow]

thank you Ryan,

I fully agree that these overfill errors can be avoided by clean coding.
My solution in the simple strategy mentioned in post #10 is to instead of having 4 if- conditions, replace the 2nd and 4th if-condition with an else - if - condition.
This way, it is impossible to have an EnterLong and ExitShort in short succession, thus avoiding the possible overfills.

 
// Enter Long
if (CrossAbove(level A)
{
EnterLong(DefaultQuantity, "");
}
 
// Exit Short
[B]else if[/B] (CrossAbove(level D)
{
ExitShort("", "");
}
 
// Enter Short
if (CrossBelow(level C)
{
EnterShort(DefaultQuantity, "");
}
 
// Exit Long
[B]else if[/B] (CrossBelow(level B)
{
ExitLong("", "");
}

Since Enter methods handle the reversal automatically, the ExitLong could contribute to overfills because you're submitting duplicate orders

However, shouldn't the NinjaTrader "core" have some logic incorporated that detects these duplicate exits ? When it is processing the ExitLong while a new EnterShort comes in, it should first execute the ExitLong, then update the strategy position information which will now be "Flat", and only then start with executing the EnterShort.

In my opinion this is a big error in the NinjaTrader managed order execution. It shouldn't execute incoming orders like a machine gun without first updating strategy position information between each order.


Marco

[brima]

Will forward your comments to development.

Did development already look into this issue? The question here is if it makes sense to wait for a solution ...

Thank you
brima

[NinjaTrader_RyanM1]

Thanks for the feedback. There is logic in place that checks this and is based on position, but position is not updated in time if you send both exit long and enter short at the same time. It's not about "quick succession" but including them in the same code block, or before the position update can be received for the first order. It's up to the strategy developer to decide whether they want 1) a standard exit, or 2) a reversal, and then structure the orders accordingly.

[NinjaTrader_Brett]

Hello,

Development will consider this and may or may not make a change here in the next major release. However this is still sometime down the road. Would recommend you coding your own solution with the tips Ryan provided.