Sequence of events in backtesting

[Maese]

Hi,

I am coding a simple strategy that does the following:
1. Sets and adjusts a stoploss in the OnBarUpdate() section:
SetStopLoss(CalculationMode.Percent, 1)

2. Evaluates a certain condition in the OnBarUpdate() section and enters short if met:
EnterShort(1000,"GoingDown");

3. Evaluates another condition in the OnBarUpdate() section and exits the short if met:
ExitShort("GoingDown close", "GoingDown");

The conditions are such that they cannot be met simultaneously within the same bar. The idea is that in CurrentBar=n I could enter the short position and in CurrentBar>n I check if I should exit.
The strategy sets CalculateOnBarClose=true;

It all works fine when I backtest this with a single instrument. The sequence of events seems to be for each new bar: OnBarUpdate -> OnOrderUpdate -> OnExecution

However, when I adapt the strategy to multiple instruments then the sequence of events for the non-primary series seem different: OnOrderUpdate->OnExecution->OnBarUpdate

As a consequence the strategy executes the stoploss order before I can evaluate my second condition that would otherwise close the position at a different exit price.

Is there a hardcoded sequence of events for each instrument when backtesting multi-instrument strategies? What would that sequence be?

If not, how can I force the evaluation of my exit-condition before the stoploss is filled? Note this is not a problem with my single-instrument strategy because events seem to always follow the same sequence.

thanks

[NinjaTrader_CalH]

Hello Maese,

Thank you for your note.

The OnBarUpdate() has not synchronization between the OnOrderUpdate() and OnExecution().

However, between OnOrderUpdate() and OnExecution(), OnOrderUpdate() will process first for order state of the filled order then OnExecution() will be processed for the execution of the filled order.

You may need to include a BarsInProgress check for your OnBarUpdate() so that orders are being processed through the correct OnBarUpdate()

[Maese]

Thank you for the quick reply.

I do have BarsInProgress checks at the beginning of my OnBarUpdate() code.
I have verified that OnExceution() always run right after OnOrderUpdate() (as explained in the help guide).
But what I am observing is that the sequence between Bar updates and Order updates seems different in a single-instrument strategy and a multi-instrument strategy.

Let me try to describe it in more detail:
- I am running a simple strategy on a single instrument GBPUSD.
On bar 521 a condition is met whereby a SetStopLoss and an EnterShort commands are launched. This is what I see in the output screen with some Print commands at different points of the code:

OnBarUpdate 04/01/2006 22:00:00
Bar: 521 BarsInProgres: 0 Instrument: $GBPUSD Default Position: Flat
Open: 1.746 Close: 1.7583 High: 1.7617 Low: 1.745
OnOrderUpdate 04/01/2006 22:00:00
Order='NT-00019/Backtest' Name='GoingDn' State=PendingSubmit Instrument='$GBPUSD'
Order='NT-00019/Backtest' Name='GoingDn' State=Accepted Instrument='$GBPUSD'
Order='NT-00019/Backtest' Name='GoingDn' State=Working Instrument='$GBPUSD'
Order='NT-00020/Backtest' Name='Stop loss' State=PendingSubmit Instrument='$GBPUSD'
Order='NT-00020/Backtest' Name='Stop loss' State=Accepted Instrument='$GBPUSD'
Order='NT-00020/Backtest' Name='Stop loss' State=Working Instrument='$GBPUSD'
Order='NT-00019/Backtest' Name='GoingDn' State=Filled Instrument='$GBPUSD'
OnExecution 04/01/2006 22:00:00
Execution='NT-00016' Instrument='$GBPUSD' Account='Backtest' Name='GoingDn' Exchange=Default Price=1.7583 Quantity=0.74M Market position=Short Commission=0 Order='NT-00019' Time='05/01/2006 22:00:00' Multiplier=1E-05
OnPositionUpdate 04/01/2006 22:00:00
Bar: 521 BarsInProgres: 0 Instrument: $GBPUSD Default Position: Short

On the next bar the exit condition is verified inside the OnBarUpdate() code and an ExitShort command is launched. Note the OnBarUpdate() section also contains a SetStopLoss command that adjust the previously filled stop orders:

OnBarUpdate 05/01/2006 22:00:00
Bar: 522 BarsInProgres: 0 Instrument: $GBPUSD Default Position: Short
Open: 1.7583 Close: 1.7555 High: 1.7586 Low: 1.7484
OnOrderUpdate 05/01/2006 22:00:00
Order='NT-00020/Backtest' Name='Stop loss' State=PendingChange Instrument='$GBPUSD'
Order='NT-00020/Backtest' Name='Stop loss' State=Accepted Instrument='$GBPUSD'
Order='NT-00020/Backtest' Name='Stop loss' State=Working Instrument='$GBPUSD'
Order='NT-00021/Backtest' Name='GoingDn close' State=PendingSubmit Instrument='$GBPUSD'
Order='NT-00021/Backtest' Name='GoingDn close' State=Accepted Instrument='$GBPUSD'
Order='NT-00021/Backtest' Name='GoingDn close' State=Working Instrument='$GBPUSD'
Order='NT-00020/Backtest' Name='Stop loss' State=PendingCancel Instrument='$GBPUSD'
Order='NT-00020/Backtest' Name='Stop loss' State=Cancelled Instrument='$GBPUSD'
Order='NT-00021/Backtest' Name='GoingDn close' State=Filled Instrument='$GBPUSD'
OnExecution 05/01/2006 22:00:00
Execution='NT-00017' Instrument='$GBPUSD' Account='Backtest' Name='GoingDn close' Exchange=Default Price=1.7555 Quantity=0.74M Market position=Long Commission=0 Order='NT-00021' Time='06/01/2006 22:00:00' Multiplier=1E-05
OnPositionUpdate 05/01/2006 22:00:00
Bar: 522 BarsInProgres: 0 Instrument: $GBPUSD Default Position: Flat

As you can see the sequence of events suggests that the Bar update is processed before the Order update. As a result my position is exited at the close price of bar 522 which is exactly what I wanted.

Now I run the same logic but in a multi-instrument strategy where the primary series is EURUSD and the second series is GBPUSD (i.e. the same as before).
At the beginning of bar 521 both instruments are flat and the entry condition is only met by GBPUSD. I get very much the same output as in the single-instrument strategy:

OnBarUpdate 04/01/2006 22:00:00
Bar: 521 BarsInProgres: 0 Instrument: $EURUSD Default Position: Flat
Open: 1.2017 Close: 1.2117 High: 1.2144 Low: 1.2005
OnBarUpdate 04/01/2006 22:00:00
Bar: 521 BarsInProgres: 1 Instrument: $GBPUSD Default Position: Flat
Open: 1.746 Close: 1.7583 High: 1.7617 Low: 1.745
OnOrderUpdate 04/01/2006 22:00:00
Order='NT-00019/Backtest' Name='GoingDn' State=PendingSubmit Instrument='$GBPUSD'
Order='NT-00019/Backtest' Name='GoingDn' State=Accepted Instrument='$GBPUSD'
Order='NT-00019/Backtest' Name='GoingDn' State=Working Instrument='$GBPUSD'
Order='NT-00020/Backtest' Name='Stop loss' State=PendingSubmit Instrument='$GBPUSD'
Order='NT-00020/Backtest' Name='Stop loss' State=Accepted Instrument='$GBPUSD'
Order='NT-00020/Backtest' Name='Stop loss' State=Working Instrument='$GBPUSD'
Order='NT-00019/Backtest' Name='GoingDn' State=Filled Instrument='$GBPUSD'
OnExecution 04/01/2006 22:00:00
Execution='NT-00016' Instrument='$GBPUSD' Account='Backtest' Name='GoingDn' Exchange=Default Price=1.7583 Quantity=0.74M Market position=Short Commission=0 Order='NT-00019' Time='05/01/2006 22:00:00' Multiplier=1E-05
OnPositionUpdate 04/01/2006 22:00:00
Bar: 521 BarsInProgres: 1 Instrument: $GBPUSD Default Position: Short

On bar 522 I would have expected something similar to the single-instrument strategy, but I get some surprises:

OnBarUpdate 05/01/2006 22:00:00
Bar: 522 BarsInProgres: 0 Instrument: $EURUSD Default Position: Flat
Open: 1.2117 Close: 1.2108 High: 1.212 Low: 1.2066
OnOrderUpdate 05/01/2006 22:00:00
Order='NT-00020/Backtest' Name='Stop loss' State=Filled Instrument='$GBPUSD'
OnExecution 05/01/2006 22:00:00
Execution='NT-00017' Instrument='$GBPUSD' Account='Backtest' Name='Stop loss' Exchange=Default Price=1.76 Quantity=0.74M Market position=Long Commission=0 Order='NT-00020' Time='06/01/2006 22:00:00' Multiplier=1E-05
OnPositionUpdate 05/01/2006 22:00:00
Bar: 522 BarsInProgres: 1 Instrument: $GBPUSD Default Position: Flat
OnBarUpdate 05/01/2006 22:00:00
Bar: 522 BarsInProgres: 1 Instrument: $GBPUSD Default Position: Flat
Open: 1.7583 Close: 1.7555 High: 1.7586 Low: 1.7484

As you can see, for the second series (GBPUSD) the update of the stop order is processed before the update of the bar. As a matter of fact the stop at 1.76 is executed although that price is not within the price range of bar 522. How is this possible?
As a result I get exited at the stop price instead as the close price which I got in the single-instrument strategy.

I'm sorry to go into so much details but I cannot figure out a simpler way to explain my problem.
Any insights?
thanks

[NinjaTrader_CalH]

Maese,

This is because you are using a SetStopLoss() method which is applied to the primary bar for processing.

The Set() is still show the 'Correct' instrument but will be processed on the primary data series.

The workaround for this would be to use your own exit logic order instead of the Set() methods.

Let me know if I can be of further assistance.

[Maese]

Thanks for your feedback,

I also thought it could be some processing of the stop order across the series. But in the example I sent you the primary series is flat at the time the stop order is filled. So even if the stop loss was processed on that series there is nothing to close.
Besides, the stop price at 1.76 is not within the price range of the primary series on that bar.

I have tried a couple of things to manage in more detail each series:
a. I have explicitly pass the signal name in the SetStopLoss(). And this name is specific to each series:

SetStopLoss("GoingUp"+BarsInProgress, CalculationMode.Price,pLongStopPrice,false);

b. I have adapt the code to be Unmanaged so I have full control on the orders. I submit the stops now with explicit reference to the BarsInProgress index:

pStopShort=SubmitOrder(BarsInProgress,OrderAction. BuyToCover,OrderType.Stop,740000,0,pShortStopPrice,"","StopGoingDn"+BarsInProgress);

But I am still getting the same undesired results: the stop order is filled at 1.76 in a bar that never crosses that level.

Regarding your suggestion to use my own exit logic, this is what I have tried with the unmanaged approach. But it doesn't seem to work.

Any other ideas?

thanks

[NinjaTrader_CalH]

Maese,

The SetStopLoss() will still be attached the instrument that the entry was placed with. However, Set() methods will get processed on the primary bar update. The EURUSD didn't have any positions, but the GBPUSD did and the Set() order was working at that time.

For the fill price, can you attach a screen shot of the test and where you are getting filled at?

To send a screenshot press Alt + PRINT SCREEN to take a screen shot of the selected window. Then go to Start--> Accessories--> Paint, and press CTRL + V to paste the image. Lastly, save as a jpeg file.
Click Reply -> Go Advanced -> Click the Paperclip icon on the toolbar

For detailed instructions please visit the following link

http://take-a-screenshot.org/

[Maese]

"Set() methods will get processed on the primary bar update. The EURUSD didn't have any positions, but the GBPUSD did and the Set() order was working at that time."

Does this mean that a working order for GBPUSD is filled based on the price of EURUSD?

The order to enter short GBPUSD is filled at 1.7583 on update of bar 521:

Order='NT-00019/Backtest' Name='GoingDn1' State=Filled Instrument='$GBPUSD' Action=SellShort Limit price=0 Stop price=0 Quantity=0.74M Strategy='MyCoinTossMulti2' Type=Market Tif=Gtc Oco='' Filled=740000 Fill price=1.7583 Token='d27431cccf704de8a62db013af4c1fa7' Gtd='01/12/2099 00:00:00'

The stop order is filled at 1.76 on update of bar 522

Order='NT-00020/Backtest' Name='Stop loss' State=Filled Instrument='$GBPUSD' Action=BuyToCover Limit price=0 Stop price=1.76 Quantity=0.74M Strategy='MyCoinTossMulti2' Type=Stop Tif=Gtc Oco='NT-00016-37' Filled=740000 Fill price=1.76 Token='1d9833e9ce19456d9360c4a6602f4482' Gtd='01/12/2099 00:00:00'

Neither series crossed 1.76 on that day:

Bar: 522 BarsInProgres: 0 Instrument: $EURUSD Default Position: Flat
Open: 1.2117 Close: 1.2108 High: 1.212 Low: 1.2066
Bar: 522 BarsInProgres: 1 Instrument: $GBPUSD Default Position: Flat
Open: 1.7583 Close: 1.7555 High: 1.7586 Low: 1.7484

So, even if the stop order was processed on the EURUSD series instead of the GBPUSD how can it possibly get filled?

In any case, note that I have also tried the unmanaged version of this strategy without using SetStopLoss() and I still seem to have the same problem.

Regarding the screenshots, when I run the multi-instruments strategy the chart of the backtest only shows the dataseries for EURUSD for which no positions are taken. Is there a way to also chart the GBPUSD series and the entries/exists generated by the strategy?

thanks

[Maese]

Hi Cal,

I've attached a screenshot of the bars in my example together with a list of the price levels in each bar.
I've been investigating this in more detail and here is a summary of what I am observing:


In a single-instrument strategy on GBPUSD (managed approach):

• On close of bar 521 the OnBarUpdate() is processed and there is code there that places a market order to go short which it gets filled at 1.7583

• OnExecution() contains code to check the short was filled and places a Buy To Cover Stop at 1.7600. It gets to the working status.

• On close of bar 522 the OnBarUpdate() is processed and there is code there that places a market order to close my position which gets filled at 1.7555 as desired.

In a multi-instrument strategy (managed approach):

• My primary series is EURUSD and my second instrument is GBPUSD

• On close of bar 521 I place a market order to go short on the second instrument and it gets filled at 1.7583

• On execution of this order I place a Buy to Cover Stop at 1.7600 on GBPUSD and it gets to the working status.

• On close of bar 522 the OnBarUpdate() is processed for EURUSD but before it is processed for GBPUSD I get the stop order filled at 1.7600. Note in the chart and table that bar 522 for GBPUSD never crosses 1.7600 (nor does EURUSD)

I have found that if I set the stop price above 1.7720 then the multi-instrument strategy does not fill the stop. This suggests that for the non-primary series upon closing of bar 522 the stop order is processed based on the prices of bar 523 instead of those of bar 522.
I have tried:

• Managed approach with EnterShort(), ExitShort() and SetStopLoss() methods

• Managed approach with EnterShort(), ExitShort() and ExitShortStop() methods

• Unmanned approach with SubmitOrder(), ChangeOrder() and CancelOrder() methods

All behave the same way: in single-instrument strategy they work as desired but in multi-instrument strategy the stop is filled on the close of a bar based on the prices of the following bar.

Is there any way to fix this?

thanks

[NinjaTrader_CalH]

Maese,

Does this mean that a working order for GBPUSD is filled based on the price of EURUSD?

No. The StopLoss is working on the GBPUSD and will get filled based on the price of GBPUSD. Since, this is a Set() method for StopLoss, these get processed by the bar update of the primary instrument. Meaning when the bar of the primary instrument is closed, all working Set() orders will get processed for their respective instruments.

Everything you have listed in the last post is expected behavior has I have pointed out.
All SET() orders are processed on the primary bar update, but will be worked for their respective instruments.

Additionally, yes in a backtest, when an order is processed, it is checked at the next bar if it will get filled at the price the order is set to.

[Maese]

Hi Cal,

thanks for the detailed explanation on how the Set() methods work in multi-instrument strategies.
I understand the in backtest mode the orders are checked against the next day price but what I am observing is that in multi-instrument strategies this works different:

- Single-instrument strategy:
Close of 04JAN: I place a stop order at 1.76
Close of 05JAN: the order is not filled
Note that the price range on 05JAN does not cross 1.76. It seems the system is checking 1.76 against the price range of 05JAN.

- Multi-instrument strategy:
Close of 04JAN: I place the same stop order at 1.76
Close of 05JAN: the order is filled at 1.76
Note that the price range on 05JAN does not cross 1.76, but the price range on 06JAN does.
It seems that now the system is checking 1.76 against the price range of 06JAN.

The correct behaviour to me should be the one I am observing with the single-instrument strategy. How can I get the same behaviour wit the multi-instrument?

thanks

[NinjaTrader_CalH]

Maese,

I will need to test the strategy on my end to check the behavior that you are seeing.

Can you send me the script to support? Help -> Mail to PlatformSupport

Put ATTN Cal in the subject and reference this thread in the body.

[Maese]

I've tried to send the email using Help -> Email support but for some reason I get an error "Cannot start Microsoft Outlook. The command line argument is not valid. Verify the switch you are using."

So I have sent it directly from my yahoo account to support@ninjatradercom with ATTN Cal on the subject and the scripts attached.

Please let me know if you have received it.

thanks a lot

Maese

[NinjaTrader_CalH]

Thanks Maese,

I have received it and looking into the matter.

[NinjaTrader_CalH]

Maese,

I wanted to update this.

I am still looking into this and working with development on the sequence of events and what is expected here.

I will report back when I have an answer to provide you with.

Thank you for your patience.

[Maese]

Thanks Cal. I'll wait for your update.

btw. I've just seen another thread by "ElMurcielago" that is reporting issues that could be related to the one I raised. Here is this other thread: "Strategy backtest has some wrong dates and prices "