Crashes on cache memory limit

[nightriderx]

I have been working with NT7 for over 6 months. The only reliable way to backtest and optimize a strategy is by using tick data (unless you calculate on bar close). However, because NT relies on the .NET framework to cache all price data into memory, which NT never clears even after a strategy is disabled/cleared, the memory quickly maxes out and causes NT to crash after brief testing. As my strategies have evolved a bit I am now at a point where I can only test a few times before NT crashes or I have to restart it.

my questions:
1) Does any of the NT community actually backtest/optimize their strategies? or do people really only use NT for custom charting? If so has this issue never been a concern as NT is now in its 7th generation???

2) Can I expect to find the ability to clear the cache to be added anytime soon or should I just stop wasting my time and find a platform that is actually tested and reliable for automated trading and testing?

3) Is it possible to backtest/optimize without caching all price data?

[NinjaTrader_RyanM]

Hello nightriderx,

For sure backtesting large amounts of tick data can tax your memory. There are currently no plans to add a cache clearing feature but you should be able to release memory by closing the strategy analyzer window.

[nightriderx]

What about just regular enabling of strategies to trade live with historical data? Or is that just not recommended to backtest in regular control center mode?

[NinjaTrader_Chip]

Hello nightriderx,

Yes – you can enable trading strategies to trade live with historical data. You may want to see if at some level you begin to experience performance issues; e.g. when having enabled multiple charts, requiring large amounts of historical tick data – or something along those lines.

For the second part or your post, please clarify if you are trying to run the backtest concurrent with Live trading.

Thank you,

[nightriderx]

Yes, so if I were to enable a strategy in the control center to trade live, but also select it to show 30 days of historical backtesting results on tick basis (so to compare with live results) that is when the memory starts getting filled up quickly never clears

[NinjaTrader_Chip]

Hello nightriderx,

Agreed – 30 days of historical tick data is significant especially for high volume contracts like the ES. Are you performing this backtest/live trading approach with multiple charts?

[nightriderx]

I would like to but NO. I have one strategy that I cannot even run the whole month with tick data due to memory max out - this is absolutely ridiculous! Didnt you guys ever think that a realistic tick data backtest would max out the memory? I just cannot understand how this is possible????? What are you guys testing? anything? minutes? which are ofter completely irrelevant to real live data??

[NinjaTrader_RyanM]

There are much different requirements running a backtest versus running a strategy live.

If you are running into memory limits running a strategy live: Reduce the amount of historical data loaded. This is controlled with the days to load when creating a new strategy.

[nightriderx]

I have tried that. But even when running the optimizer, with no variation, just to back test a month of tick data still causes NT to crash after a while...

[NinjaTrader_RyanM]

What type of hardware are you using? Is there any room for improvement there? One month of tick data can end up taking a lot depending on the instrument you're trading. Trying to reproduce what you're seeing here - I'm able to optimize 6 months of ES tick data without running into memory limits.

We have performance tips you can review to make sure you're reducing NinjaTrader's resource requirements as much as possible. If you're frequently running into memory limits, then reducing the amount of historical data requests is what you'll have to do. Make shorter tests until it works successfully and then aggregate the results.

http://www.ninjatrader.com/support/helpGuides/nt7/performance_tips2.htm

[nightriderx]

the problem is in the fact that all the memory does not get cleared out at the end of the testing. If you run that ES test on tick basis over and over throughout the day without restarting while running live strategies - you will no doubtingly see what I mean... the link does not seem to address the issue. I am running a quad core pc, with 4G of ram, though on a 32 bit os. But even if i were to switch to a 64 bit os, over time the memory would still get overfilled

[NinjaTrader_RyanM]

Shutdown and reopen the strategy analyzer window in-between memory-intensive tests.