backtesting vs. forward testing like night vs. day

[andrewbee]

Hi,

I have a strategy coded that seems to lose money in forward testing, but makes money in backtesting. (It works on ES). For instance, in the 11-hour period up to 11:00 am today, forward testing indicates a loss of $845, while backtesting indicates a profit of $1200! In 11 hours!

Now, I understand that back vs. forward testing might not always give exactly the same results, but this is ridiculous. The fill type is set to 'Liberal' in both cases, and the data feed is the same (OpenTick). So, the fills should theoretically be about the same, but they aren't. I'm beginning to think the backtester is useless for all practical purposes. And no, I don't believe it's my code. Any thoughts, anyone?

[NinjaTrader_Ray]

Here are some -

http://www.ninjatrader-support.com/HelpGuideV6/DiscrepanciesReal-TimeVsBacktest.html

[mwyatt]

I am experiencing the same thing as Andrew, and have read the link provided. Is it a ninja trader issue that the back testing is done on the OHLC rather than tick by tick, or is a data issue (i.e. esignal data not precise enough)? It seems rather restrictive on the types of back testing strategies we can write that will actually have some use (i.e. trailing stops in backtesting useless)...

[NinjaTrader_JoshP]

What some people have done in the past is generate there strategy as a multi-time framed strategy using 1tick as one of the time frames. This is a very CPU/memory intensive way of trying to backtest like a forward test and is limited by the amount of tick data you can actually get.

Other than that, there is no real way NinjaTrader or any other program can backtest more realistically simply because of the resolution limitation of the data available.

[mwyatt]

Thanks for the quick reply Josh. Next Question. Who has the best data? I have heard the mention of tickdata.com and tradestation. Any ideas?

cheers
Mat

[NinjaTrader_JoshP]

Here is the list of all the supported data providers: http://www.ninjatrader-support.com/H...ricalData.html

[andrewbee]

I have written off NT's backtesting as completely useless for my purposes, since it does not support tick-by-tick testing. I had already written my own backtester anyway, in Java, so I just had to re-code my strategy in Java and use that.

[NinjaTrader_Dierk]

Curious: Why is NT backtesting useless for your purposes? NT supports tick by tick backtesting.

[KBJ]

I don't understand this either, as I have found NinjaTrader's tick-by-tick backtesting to be very useful in developing strategies.

The only part of it that I find difficult is this... If I backtest with a tick-based timeframe selected, and when done I switch to a chart to see where the trades occurred, if I then change the timeframe of the chart so I can better visualize what's happened, all the trades go away off the chart. (Please feel free to add this to your list of areas for requested improvements.)

But other than that, I find the tick-by-tick backtesting very useful.

[NinjaTrader_JoshP]

Thanks for the suggestion as always KBJ.

[mwyatt]

Thanks for that Josh and KBJ, thats an interesting approach.

But. Which provider offers enough historical tick data that we can test on? I use eSignal and they only offer 2 weeks of historical tick data... I want atleast 1 years worth.

[NinjaTrader_JoshP]

I believe OpenTick has a bunch. I am not too familiar with the rest so cannot comment.

[NinjaTrader_Dierk]

Note: 1 tick based testing has to deal with memory issues. 150,000 ticks per day * 220 trading days per year * ~25 bytes per tick gives you an idea about NT mem usage. Unfortunately there is no way around it for now. We'll look into potential optimizations for NT7 (end of this year)

[NinjaTrader_Ray]

The only part of it that I find difficult is this... If I backtest with a tick-based timeframe selected, and when done I switch to a chart to see where the trades occurred, if I then change the timeframe of the chart so I can better visualize what's happened, all the trades go away off the chart. (Please feel free to add this to your list of areas for requested improvements.)

What I do is have another chart open with the different time frame. Then enable Global Crosshair on both the Strategy Analyzer chart and the new chart. You can then scroll in the SA chart and the other time frame chart will scroll to that point. This is in NT 6.5.

[mwyatt]

Using esignal as the data source and using the continuous contract symbols in ninja, i can manage to get about 2 weeks tick data for ES and ER.

Using open tick i can manage to get 1 week historical tick data for the ES, and about 2 weeks for the ER.

What am i doing wrong? Is this all the tick data that is on offer?