Inconsistent backtest results

[ctrlbrk]

Hello,

I am still struggling with what seems to be some backtesting problems. I have narrowed down one such issue as much as I can to make it as simple as I can and need some help.

Run strategy with backtest period of 1/1/08 - 1/10/08
Result is 3 trades taken.

If I run the same strategy and add additional days, such as 12/1/07 - 1/10/08, now only 2 trades are taken from 1/1/08 to 1/10/08. One trade is missing, on 1/9/08.

I have TraceOrders enabled. There is no mention of why the missing trade was not taken.

The missing trade in this example is this one (showing output log):
1/9/2008 1:59:00 PM Entered internal PlaceOrder() method at 1/9/2008 1:59:00 PM: Action=Buy OrderType=Limit Quantity=1 LimitPrice=1400.25 StopPrice=0 SignalName='long 1 es' FromEntrySignal=''
1/9/2008 1:59:00 PM Entered internal SetStopTarget() method: Type=Stop FromEntrySignal='long 1 es' Mode=Price Value=1394.45766889127 Currency=0 Simulated=False
1/9/2008 1:59:00 PM Entered internal SetStopTarget() method: Type=Target FromEntrySignal='long 1 es' Mode=Price Value=1406.04233110873 Currency=0 Simulated=False
1/9/2008 2:06:00 PM Cancelled pending exit order, since associated position is closed: Order='NT-00001/Back101' Name='Stop loss' State=Working Instrument='ES ##-##' Action=Sell Limit price=0 Stop price=1394.5 Quantity=1 Strategy='live_ES_smacross_long1' Type=Stop Tif=Gtc Oco='NT-00000' Filled=0 Fill price=0 Token='df49e7e2c58348128024035f84fa2672' Gtd='12/1/2099 12:00:00 AM'
1/9/2008 2:06:00 PM Cancelled OCO paired order: BarsInProgress=0: Order='NT-00001/Back101' Name='Stop loss' State=Cancelled Instrument='ES ##-##' Action=Sell Limit price=0 Stop price=1394.5 Quantity=1 Strategy='live_ES_smacross_long1' Type=Stop Tif=Gtc Oco='NT-00000' Filled=0 Fill price=0 Token='df49e7e2c58348128024035f84fa2672' Gtd='12/1/2099 12:00:00 AM'
1/9/2008 2:06:00 PM Cancelled expired order: BarsInProgress=1: Order='NT-00001/Back101' Name='Stop loss' State=Cancelled Instrument='ES ##-##' Action=Sell Limit price=0 Stop price=1394.5 Quantity=1 Strategy='live_ES_smacross_long1' Type=Stop Tif=Gtc Oco='NT-00000' Filled=0 Fill price=0 Token='df49e7e2c58348128024035f84fa2672' Gtd='12/1/2099 12:00:00 AM'

The more days I add, the more missing trades that occur.

I've attached a video illustrating the missing 1/9/08 trade, since it is the simplest example.

I should also mention that if days are added AFTER the period, instead of before, the issue doesn't exist. For instance, if I run the period of 1/1/08 to 11/26/08, the total number of trades are 99.

However, if I run 1/1/07 (one year prior) to 11/26/08 (same end date), the total number of trades are 93 --- so six trades go missing over various time frames.

This increases more if I add an extra year, and so on.

Video:
http://www.mgbi.net/ninja/missingtrade.wmv - Video 1m 39s, 8.72MB

Thank you -- and Happy Thanksgiving!

[ctrlbrk]

I'd also like to mention (and this can be seen in the video) that I have added some print statements as well, to let me know when I am 'inside' a certain part of the script.

For the missing trade in question, the print statement doesn't go off at all, which is basically telling me that Ninja thinks the data isn't the same as it was when the number of days was different.

My strategy uses 1 minute, 2 minute, and 14 minute time frames and nothing goes back to the previous trading day as for moving averages or anything like that.

[ctrlbrk]

I started going line by line removing conditions for a trade, and found that the indicator ParablocSAR is responsible for the inconsistency.

Now the question is to why.

Why, when more historical data is added, would this indicator change enough over the period of one full year in which it would not trigger 6 out of 99 trades that were otherwise triggered before adding more historical data.

I guess that is question I will try to answer, unless someone else can help.

I started thinking the minimum required bars might be in order -- but changed it from 20 to 50 to 100 to even 200 bars and there was no change in data.

[ctrlbrk]

Sorry for talking to myself -- lol -- but it is 6am on turkey day and I'm sitting here optimizing strategies!

I do really try to avoid asking for help, I prefer to figure things out on my own. But sometimes the process of going through the motions to present my case on a forum helps me see the gaps in logic.

I am going to try to re-write the logic to use ATRTrailing instead of ParabolicSAR. They seem to be similar for what I am wanting to accomplish, in fact ATRTrailing probably makes more sense.

[NinjaTrader_Ray]

A simple test would be to throw the SAR indicator on two different changes of the same data, and compare plot values, I don't fully understand how SAR works but I think once there is enough data to normalize it, it should plot the same values.

[ctrlbrk]

Thanks Ray. I basically did that (different approach) and I couldn't figure out why the numbers were different. So, instead my energy will be focused on using a different indicator. I felt like the SAR indicator was a bit to curve fitted anyway, so this is good.

Have a great Thanksgiving with your family!

[NinjaTrader_Ray]

Thanks, Happy Thanksgiving to you and yours as well.

[ctrlbrk]

Argh. Ok I spent the last couple hours re-writing the strategy to eliminate SAR which was what was causing the initial problems.

However, the problem has now come back.

I am using only:
LeaderOfMACD
RSI
SMA

Those are the only indicators. The problem is same as before, more historical data changes trade values days and months away.

The largest barsarray/time frame is 14 minutes, and the longest period is 26 bars (12,26,9).

I have to stop working for now but will try to hunt this down some more when I return.

[ctrlbrk]

If the strategy is executed on a 1 minute time frame, and the largest barsarray time frame is 14 minutes, and the longest period of any indicator is 26 bars...

14*26 = 364 1 minute bars

Am I correct in assuming the "min bars required" should be set to at least 364 for this to work?

Is using the (if CurrentBar < 364) return; approach in a strategy the same as manually typing in "364" in the dialog box?

Last, when running the strategy real-time live, will it wait 364 bars from the time I hit 'start', or does it load 364 historical bars and is ready to trade immediately?

[NinjaTrader_Ray]

If the strategy is executed on a 1 minute time frame, and the largest barsarray time frame is 14 minutes, and the longest period of any indicator is 26 bars...

14*26 = 364 1 minute bars

Am I correct in assuming the "min bars required" should be set to at least 364 for this to work?

Is using the (if CurrentBar < 364) return; approach in a strategy the same as manually typing in "364" in the dialog box?

Last, when running the strategy real-time live, will it wait 364 bars from the time I hit 'start', or does it load 364 historical bars and is ready to trade immediately?

I believe you might need min 364 14 min bars, you would need to check with this. CurrentBar < 364 would work. It will run from the 1st bar on your chart if running from a chart or it will load the default # of days (from Tools > Options) when running from the Strategies tab.

[ctrlbrk]

Alright -- so no charts involved, if I have CurrentBar < 364 return in my script and I 'start' my script at 9:00am, and there is a trade that should fire at 9:01am, it will? It is loading the 364 from the past, not waiting to accumulate 364 bars since pressing 'start' ?

And same is true for strategy analyzer? Even if dialog box says '20' min bars, if the script has CurrentBar < 364 it will load 364 historical bars before proceeding, but won't wait 364 bars to accumulate new data before executing a trade?

I just need to be extra clear. I tried looking at the on-line help and couldn't find anything that spelled out how this one item works.

[NinjaTrader_JoshP]

364 is determined from the start of your chart, not from real-time bars.

CurrentBar is related to the amount of bars on your chart and has nothing to do with real-time bars. If you want to limit trading till 364 real-time bars have passed you would want to run your own counter in conjunction with this line of code

if (Historical)
     return;