Fill event misalignment in backtesting

[anachronist]

I'm backtesting something using using the unmanaged approach, and I noticed a weird result.

If the next bar after the current bar would fill a limit order, the fill event fires off on the current bar rather than the next bar. In other words, a fill event for a limit order doesn't align with the bar that filled it.

This misalignment messes up the logic for adjusting stops and targets on the current bar if the entry order's status has already been set to Filled before it actually happens, resulting in occasional errors in the backtest results.

I got around this by delaying stoploss and target processing until the bar following the fill event (which is the bar that actually caused the fill).

If others are accounting for this misalignment in their code and NinjaTrader corrects this behavior, it might break some things.

-Alex

[NinjaTrader_Bertrand]

Thanks for the report Alex, I'm not aware of similiar issues seen sofar so would like to investigate with your help if possible.

Is there any way you could mask your proprietary logic and send me your script for testing to support at ninjatrader dot com Attn Bertrand?

Please include datafeed, instrument / expiry and timeframe / chart type as well.

Thanks much,

[anachronist]

It isn't a problem in my script. This should be easy to duplicate with a simple script. I was on a 2 minute chart of TF using a Zen-Fire data feed through Mirus Futures, with trades executed on a 20-second secondary data series.

• Find any bar in the history that makes a significant pivot low, several ticks below the prior several bars, on both time frames.
• Identify the bar number of this new low (I have a simple indicator that plots the value of CurrentBar on every bar, so I can view the bar number in the data window).
• Run a simple script that places an unmanaged limit order a few bars beforehand, to buy 1 or 2 ticks above the pivot low price. e.g. if (CurrentBar==pivotbar-3) SubmitOrder(...).
• In OnOrderUpdate(), print the primary and secondary bar numbers to the output window when the Fill event is captured. Also print the low of the current bar.
• You will find that the low of the current bar would not fill the order, but the low of the next bar would. The bar number printed for the fill bar will be 1 bar prior to the bar that would fill the order.

I've read other messages from support staff on this forum saying that the backtesting logic looks ahead to the next bar to determine if a limit order is filled. That makes sense, but it seems that the fill event isn't delayed until the next bar but instead gets fired off as soon as a fill is detected on the next bar.

-Alex

[NinjaTrader_Bertrand]

Alex, thanks - so this would be different for you in a managed script, is this correct?

I would be surprised since accessing bar info in the event order / execution method would not be a valid check for your situation, since the executions in backtesting are proccessed before the bar update events.

[anachronist]

It is no different in a managed script.

Here's an example. TF 09-11 contract loaded from ZenFire, 2-minute bars, Default 24/7 session template, 2 days ending on 7/4/2011.

• Place order on bar 296 (this is 08:28 Pacific time) to sell at 839.3 limit.
• Trading halts on bar 298, the 08:30 bar (this is a holiday). Note that trading has halted without the limit order being filled (price never reached it).
• Nevertheless, the fill event fires off on bar 298, even though price never touched it.
• The actual fill is hours later when trading resumes, on bar 299 at 17:02.
• Another problem observed: Even though ExitOnClose=false, the order is exited on the session close anyway. Why?

The script below prints the message:
7/4/2011 8:26:00 Entered internal PlaceOrder() method at 7/4/2011 8:26:00: BarsInProgress=0 Action=SellShort OrderType=Limit Quantity=1 LimitPrice=839.3 StopPrice=0 SignalName='test' FromEntrySignal=''
Filled on bar 298 at 839.3 where high=839 and low=838.8

Regardless of whether I use the managed or unmanaged approach, the fill event fires for 1 bar in the future, not on the bar where the order got filled.

PHP Code:


        #region Variables
        // Parameters for TF 09-11 2 minute, Default 24/7 template, 2 days ending 7/4/2011
        private int barNum = 296; // Default setting for BarNum
        private double limitPrice = 839.3; // Default setting for LimitPrice
        private IOrder entry;
        #endregion

        protected override void Initialize() {
            CalculateOnBarClose = true;
            ExitOnClose = false;
            TraceOrders = true;
        }

        protected override void OnOrderUpdate(IOrder o) {
            if (o == entry && o.OrderState == OrderState.Filled)
                Print("Filled on bar "+CurrentBar.ToString()+" at "+o.AvgFillPrice.ToString()+" where high="+High[0].ToString()+" and low="+Low[0].ToString());
        }

        protected override void OnBarUpdate() {
            if (CurrentBar == barNum)
                entry = EnterShortLimit(0, true, 1, limitPrice, "test");
        } 


I would be surprised since accessing bar info in the event order / execution method would not be a valid check for your situation, since the executions in backtesting are proccessed before the bar update events.

That's good to know. However, it still messes things up. Notwithstanding the example above, I don't access the bar information in the OnOrderUpdate() method. I access the bar information in the OnBarUpdate() method.

The problem is that I need to check order states in OnBarUpdate(), so that I can adjust stops and targets for any positions running. The order state is being set to "Filled" and the position is created before the bar that actually fills the entry.

-Alex

[NinjaTrader_Bertrand]

Thanks for the clarifications Alex, this would be expected in backtesting and it would not be different for managed vs unmanaged approach how the fill simulation would work - your CurrentBar print comes from the OnOrderUpdate() and it would report the last seen state, at the point your print this CurrentBar would not have advanced as OnBarUpdate() for the next bar has not yet been called. In simulation with CalculateOnBarClose = false this should align as you expext.

For the ExitOnClose issue - so is the position exited or the order expired at session boundary? For order expiration, please try with a TIF of GTC.

http://www.ninjatrader.com/support/h...imeinforce.htm

[anachronist]

Regarding ExitOnClose, you're right, the order was a day order and had expired.

Indeed, for the managed approach, CalculateOnClose=false aligns the fill with the correct bar.

The fact remains, however, that fills are still misaligned with the unmanaged approach regardless of the CalculateOnClose setting.

Here's the same example using the unmanaged approach and a secondary series, this time printing out the bar and fill information from OnBarUpdate() instead of OnOrderUpdate(). For some reason the fill is reported two bars early. I have attached a chart showing the problem.

PHP Code:

        #region Variables
        // Parameters for TF 09-11 2 minute, Default 24/7 template, 2 days ending 7/4/2011 
        private int barNum = 296; // Default setting for BarNum
        private double limitPrice = 839.3; // Default setting for LimitPrice
         private IOrder entry = null;
        bool printed = false;
        #endregion

        protected override void Initialize() {
            Add(PeriodType.Minute, 1);
            Unmanaged = true;
            CalculateOnBarClose = false; // doesn't matter
            ExitOnClose = false;
            TraceOrders = true;
        }

        protected override void OnBarUpdate() {
            if (BarsInProgress != 0) return;
            if (CurrentBar == barNum)
                entry = SubmitOrder(1, OrderAction.Buy, OrderType.Limit, 1, limitPrice, 0, "", "test");
            if (!printed && entry != null && entry.OrderState == OrderState.Filled) {
                Print("Filled on bar "+CurrentBar.ToString()+" at "+entry.AvgFillPrice.ToString()+" where high="+High[0].ToString()+" and low="+Low[0].ToString());
                printed = true;
            }
        } 


Output window says:
7/4/2011 8:26:00 Entered internal SubmitOrder() method at 7/4/2011 8:26:00: Action=Buy OrderType=Limit Quantity=1 LimitPrice=839.3 StopPrice=0 OcoId='' Name='test'
Filled on bar 297 at 839.3 where high=838.9 and low=838.8

The attached picture should make the problem clearer. I am open to suggestions about how to manage exit orders if my fills are occurring too soon. The only approach I have come up with is to record the bar number of the fill and delay any further processing until the next bar -- but here I have a case where the fill is two bars early.

-Alex

[NinjaTrader_Bertrand]

Thanks Alex, let's take it a step back - so this would be a Buy Limit order you place above the market?

If yes, then what you see is expected - as it's a marketable limit order, it would fill immediately here in your case. Basically a limit order says, this 'price or better' > for the buy the lower (current) price is better, so it fills. If you would like to park an entry at this level, please try a StopLimit order type.

[NinjaTrader_Bertrand]

I also just saw: you annotated the chart as Sell Short order, yet the code submits a Buy Limit entry. What's the desired action here for your test case?

Thanks,

[anachronist]

Aargh! This is what happens when I'm rushing to write code for this thread before the market opens. I make mistakes that waste your time, and I apologize for that.

You're right, I got confused. It was supposed to be a sell. If I correct the code, everything works properly. The fill event fires before CurrentBar is updated, but if I print things out in OnBarUpdate() the fill shows up on the correct bar.

Now my problem is figuring out why, in my more complex strategy, OnBarUpdate() was showing me fills one bar before they actually happen. Even the chart showed this. This is 700 lines of code that attempts to emulate the behavior of ATM strategies in backtesting, so that's a bit much to post here; it's my problem to solve.

It helps to know that in backtesting, OnOrderUpdate() occurs before OnBarUpdate(). That should help me figure out what's going on.

-Alex

[NinjaTrader_Bertrand]

No worries Alex, glad to assist. Recreating the ATM's for backtesting can definitely be a challenging task, if you would like I can give your code a run here on my end to check into and see if anything stands out for you to investigate, no guarantees though - you're right this 700 line script would be involved to debug, but would be happy to offer a second set of eyes here. This is a MultiSeries script as well, correct? If so I would first take out those aspects and let it run as single series script (just for isolating out) - same discrepancy witnessed?

All the best,

[anachronist]

Oh, the ATM backtesting recreation is all done. There's just this niggling issue about the bar in which an order is filled. My methods called from OnBarUpdate() and OnOrderUpdate() print out the same values of the CurrentBars[] array after a fill, and these values are always 1 bar before the actual fill on both data series.

Aha. I managed to modify my simple test posted earlier to reproduce the problem. It does seem to be an issue with multiple time frames. I need to know the actual bars of the fill for all data series. Therefore I can't perform the test only when BarsInProgress==0.

Initial condition: Zen-fire TF 06-12, 2-minute 24/7 template, 1 day ending 5/14/2012. Order is placed on bar 376. Should get filled on bar 377. Instead it gets filled on 376 (series 0), 631 (series 1) according to the CurrentBars values in OnBarUpdate(). Here's the script.

PHP Code:

        #region Variables
        // Parameters for Zen-Fire TF 06-12 2 minute, Default 24/7 template, 1 day ending 5/14/2012 
        private int barNum = 376; // Default setting for BarNum
        private double limitPrice = 780.3; // Default setting for LimitPrice
         private IOrder entry = null;
        bool printed = false;
        int barSeries = 0;
        #endregion

        protected override void Initialize() {
            Add(PeriodType.Minute, 1);
            Unmanaged = true;
            CalculateOnBarClose = false; // doesn't matter
            ExitOnClose = false;
            TraceOrders = true;
        }
        protected override void OnStartUp() { ClearOutputWindow(); }

        protected override void OnBarUpdate() {
            if (!printed && entry != null && entry.OrderState == OrderState.Filled) {
                Print("Buy order filled on bar "+CurrentBars[0].ToString()+","+CurrentBars[1].ToString()+" at "+entry.AvgFillPrice.ToString()+" where high="+High[0].ToString()+" and low="+Low[0].ToString());
                printed = true;
            }
            if (BarsInProgress != 0) return;
            if (CurrentBar == barNum)
                entry = SubmitOrder(barSeries, OrderAction.Buy, OrderType.Limit, 1, limitPrice, 0, "", "test");
        } 


Here's the output.

5/14/2012 6:06:00 Entered internal SubmitOrder() method at 5/14/2012 6:06:00: Action=Buy OrderType=Limit Quantity=1 LimitPrice=780.3 StopPrice=0 OcoId='' Name='test'
Buy order filled on bar 376,631 at 780.3 where high=781.2 and low=781

Given the fill price and the high and low reported, it isn't logically possible for that bar.

-Alex

[NinjaTrader_Bertrand]

Thanks Alex, you've limited yourself with the Printed bool, as your code would execute for both BarsInProgress (you don't filter it) yet the only one BIP would print it - if you remove this and print for each call you can see -

5/14/2012 9:08:00 AM BIP 0 Buy order filled on bar 377, 633 at 780.3 where high=781 and low=780

which is the correct BIP 0 (2min frame) bar you submitted to for a fill, this info would only be valid to access in BIP0 which the above print does.

From reviewing your code and fill behavior seen I do not see a misreporting.

[anachronist]

Aha. I think I get it. CurrentBars[x] isn't valid unless x==BarsInProgress. Right?

In my ATM backtest framework, I need to store the fill bar in each data series. So instead of what I've been doing in OnBarUpdate():

if (order.OrderState == OrderState.Filled && atm.fillbar[0] == 0) {
    // record the bar in each data series on which the order filled
    for (i = 0; i < CurrentBars.GetLength(0); ++i)
        atm.fillbar[i] = CurrentBars[i];
}

I'd do something like this in OnBarUpdate():

if (order.OrderState == OrderState.Filled && atm.fillbar[BarsInProgress] == 0) {
   // record the order fill bar
   atm.fillbar[BarsInProgress] = CurrentBars[BarsInProgress];
}

Then if anything in my code needs atm.fillbar[x], that part of the code should be skipped if atm.fillbar[x]==0. Hopefully that will fix things. Thanks.
-Alex

[anachronist]

OK, something is still fishy here. I changed my script to record the fill bar only only when BarsInProgress matches the data series. The fill bar for data series 0 is reported correctly, but the fill bar for data series 1 is still incorrect.

Here's the new test script. Same initial conditions, Zen-Fire TF 06-12, 2 minute data, 24/7 session template, 1 day of data ending 5/14/2012.

PHP Code:


        #region Variables
        // Parameters for Zen-Fire TF 06-12 2 minute, Default 24/7 template, 1 day ending 5/14/2012 
        private int barNum = 375; // Default setting for BarNum
        private double limitPrice = 780.3; // Default setting for LimitPrice
         private IOrder entry = null;
        private int barSeries = 0;
        private int [] fillbar;
        #endregion

        protected override void Initialize() {
            Add(PeriodType.Minute, 1);
            Unmanaged = true;
            CalculateOnBarClose = false; // doesn't matter
            ExitOnClose = false;
            TraceOrders = true;
        }
        protected override void OnStartUp() {
            ClearOutputWindow();
            fillbar = new int[CurrentBars.GetLength(0)];
        }

        protected override void OnBarUpdate() {
            if (entry != null && fillbar[BarsInProgress] == 0 && entry.OrderState == OrderState.Filled)
                fillbar[BarsInProgress] = CurrentBars[BarsInProgress];
            if (BarsInProgress != 0) return;
            if (CurrentBar == barNum)
                entry = SubmitOrder(barSeries, OrderAction.Buy, OrderType.Limit, 1, limitPrice, 0, "", "test");
            if (CurrentBar == barNum+10 && entry != null) {
                Print("Filled at "+entry.AvgFillPrice.ToString());
                for (int i = 0; i < fillbar.GetLength(0); ++i)
                    Print("\ton series "+i.ToString()+" bar "+fillbar[i].ToString());
            }
        } 


The output window shows:


5/14/2012 6:06:00 Entered internal SubmitOrder() method at 5/14/2012 6:06:00: Action=Buy OrderType=Limit Quantity=1 LimitPrice=780.3 StopPrice=0 OcoId='' Name='test'
Filled at 780.3
on series 0 bar 377
on series 1 bar 631

377 is correct for series 0. However, the fill bar on series 1 should be 632. The bar 631 is nowhere near the limit order price.

Any ideas? I really need a consistent way to identify the correct fill bar on all data series.

-Alex