Back-testing is just extremely slow vs 6.5...

[heech]

I posted a thread about this maybe a year ago, when I first updated to Ninja 7 from 6.5... and now I'm back, because it's reached a point where the performance is just too poor for me to continue using this.

I know this is kind of a long shot, but I'm really hoping someone at Ninja can do some creative thinking and help me figure out what might be causing the slow-down. I'm talking about probably a 10x-20x degredation in performance from Ninja 6.5 to Ninja 7, on a machine identical to what I was using previously. (A test that used to take 1 minute will often take 15 minutes now... a test that used to run in 5 minutes, well, I can't wait that long.)

My strategy has some quirks in it:

- I run on two instruments, with 3 different time series (1/5/30 minute bars).
- on any given bar, I can place up to 20 different orders. Most of these are stop orders. (So, yes... single instrument, perhaps bracketed by 5-10 stop orders on each side, with prices updated every 1-5 minutes.)
- I have custom OnExecution() code which does some processing upon a fill.
- I do not use any Ninja-based order handling (targets/OCOs/stops). I exclusively use unmanaged order functions.

I am not using any of the strategy analyzer's optimize functions, just doing a straight back-test over a roughly 30-50 day time period.

So... help! What am I doing wrong?

[heech]

Just FYI, I tried the exact same test using managed orders... (I have both managed/umanaged versions wrapped up in a layer, so it's easy to switch between the two.) And performance results are identical, so that wasn't the issue.

I have run dotTrace in order to figure out what was happening, but didn't gain a lot of insight. 3 of the worker threads did no work at all, with seemingly all back-testing work in the main thread. I really wasn't able to decipher any more than that. If anyone at NT has suggestions on performance profiling tools that might be effective, I'd love to hear about that as well.

[heech]

I've tried getting rid of the great majority of my order submissions entirely... this sped things up (as you'd expect), by about a factor of 2. Even so, we are looking at probably a 5x slow down to where things were in version 6.5... so still exceedingly painful to work with.

There must be something else happening here. Is it possible something as fundamental as accessing bar high/low has become extremely inefficient? Every bar I loop through quite a bit of logic, and might end up accessing bar/session data quite a bit.

If bar data was no longer a very fast read, for example... I could just cache that data in some local state. And that's exactly the kind of insight I'm looking for: what has changed that might affect performance substantially, that I should try some hacks to work around in my own code?

Btw, I'm running NT7 64-bit on Windows 7 Professional, 16 GB SRAM, and SSD.

[heech]

One thing I've noticed which I wonder is significant...

On 6.5, I used to see my NinjaTrader process consume up to 1 GB at the peak of this backtest. Now on 7, I rarely see it break 300 MB even with a similar sized test.

Now, that's hardly an apple/apple comparison... but I'm wondering if this suggests a lot less program memory is being used to cache frequently used data? And if so, could that be an issue? Is there any way for me to tweak that setting... consume more memory + faster performance?

[heech]

Wait, I just noticed this from the NT6.5/NT7 migration pdf.

"DataSeries objects now default to storing only the last 256 data points with
MaximumBarsLookBack.TwoHundredFiftySix"

I don't have any custom DataSeries in my code, but I use High/Low (and just general Bars stuff) extensively, and quite often might look back > 256 data points. Are High/Low also storing/caching just the last 256 data points??

[NinjaTrader_Matthew]

I have run dotTrace in order to figure out what was happening, but didn't gain a lot of insight. 3 of the worker threads did no work at all, with seemingly all back-testing work in the main thread. I really wasn't able to decipher any more than that. If anyone at NT has suggestions on performance profiling tools that might be effective, I'd love to hear about that as well.

I assume this was on a quad core processor. Backtesting should be fully multi-core efficient and we should expect all 4 threads to work. I suspect this is the root of what you're experience.

If you run the same test on the SampleMACrossOver strategy, do you see the same results, or do you see 4 threads working?

Regarding your specific strategy, can you detail what type of user defined inputs you are using if any? Are these doubles/ints or are you using any enumerations or bools?

I don't have any custom DataSeries in my code, but I use High/Low (and just general Bars stuff) extensively, and quite often might look back > 256 data points. Are High/Low also storing/caching just the last 256 data points??

To improve memory performance, the HighLow would also be stored to just 256 bars. You can always set this to MaximumBarsLookBack.Infinite to see if that improves the HighLow calculations you're doing on the primary and added data series, but you will definitely see a significant memory hit in this case.

[heech]

I assume this was on a quad core processor. Backtesting should be fully multi-core efficient and we should expect all 4 threads to work. I suspect this is the root of what you're experience.

I will do the test you suggested later this afternoon, and see if profiling shows me anything for the sample strategy. But why, if Ninja 6.5 was single-threaded (that's my understanding)... would being stuck on one thread be the root of the problem?

For what its worth, I use a lot of lock() statements within my code to prevent race conditions / sync issues. If it turns out multi-threading is the issue/solution, I can try to look at whether I can lock at finer-granularity stuff instead... do you think that would help?

Regarding your specific strategy, can you detail what type of user defined inputs you are using if any? Are these doubles/ints or are you using any enumerations or bools?

This code's been slowly growing for 3+ years, and I use the same core code but "select" between back-testing/live-trading/perofrmance tracking, etc, etc... so there are a *lot* of user defined inputs. Ints, DateTime, enumerations, bools... I probably have 15-20 different user inputs. Why might this be an issue?

To improve memory performance, the HighLow would also be stored to just 256 bars. You can always set this to MaximumBarsLookBack.Infinite to see if that improves the HighLow calculations you're doing on the primary and added data series, but you will definitely see a significant memory hit in this case.

I was very excited to hear this might be a cause, and ran out to try it immediately... but sadly, no joy. Changing 'MaximumBarsLookback' to infinite had no effect on time whatsoever.

[NinjaTrader_Matthew]

At this point, I couldn't say why 6.5 to 7 has slowed down, but for now I'd just like to focus on why it's working on a single thread as this does suggest the backtesting engine is not working as we designed in NT7 for your strategy.

I bring up the inputs because we are aware enums/bools will prevent the optimizer from using multiple threads. I haven't seen this to be the case with just a straight backtest - however this would be good to know if this is what you're running into.

This limitations is discussed in greater detail in the following thread:

http://www.ninjatrader.com/support/forum/showthread.php?t=31816

I'd advise re-coding some of these properties as per post #13 suggestions to see if this has any effect on the backtesting performance you're seeing.

[NinjaTrader_Matthew]

Sorry, I misspoke in this regard. Straight backtesting will always run from a single core.

To help determine exactly what is taking so long to process, can you Print(DateTime.Now) at the beginning and end of each event handler you're using?


For example, the beginning and end of OnBarUpdate() and also the beginning and end of OnExecution()

[heech]

Okay... I'll go through and try to change things up. A *lot* of changes, so don't expect a response from me for a while.

Can you tell if multi-core is also disabled for live strategy runs?? That seems like a pretty drastic "fix" for an optimizer problem.

[NinjaTrader_Matthew]

No, this only applies to optimization. There was a misunderstanding on my end on this limitation and I don't believe this to be the problem at this point.

[heech]

Sorry, I misspoke in this regard. Straight backtesting will always run from a single core.

To help determine exactly what is taking so long to process, can you Print(DateTime.Now) at the beginning and end of each event handler you're using?


For example, the beginning and end of OnBarUpdate() and also the beginning and end of OnExecution()

Ah, okay.. well, you just saved me a lot of work by letting me that was the case.

I can do as you suggest, but I'm not sure that kind of crude profiling is going to buy us a whole lot.. since it doesn't tell us how much time stuff is *supposed* to take (vis-a-vis what was happening in 6.5).

One thing we should think about.. I noticed this in the 6.5->7.0 code breaking guide:

All pending orders are processed after OnBarUpdate() now. Before only orders targeting that particular BarsInProgress would process.

Is there anything tricky about the number of orders submitted for a bar, that might be an issue...? Like I said, after OnBarUpdate() finishes... I could very likely have 20+ orders queued up to be processed. Is the way that 7.0 handles this very differently from the way that 6.5 handled it?

[NinjaTrader_Matthew]

I appolgoize again for the miscommucation, since this is backtsting all will be single threaded.

Keep this in mind that you would not be able to get simultaneous OnBarUpdate and OnExecution calls.

I do not believe that the change to when orders are processed with create this large of delay.

Printing the OnBarUpdate and OnExecution calls should give us a better idea of where we need to focus.

Reviewing this thread and you previous threads, have you tried removing/comment out your Lock() statements? Do you see any change in the performance?

[koganam]

... For what its worth, I use a lot of lock() statements within my code to prevent race conditions / sync issues. If it turns out multi-threading is the issue/solution, I can try to look at whether I can lock at finer-granularity stuff instead... do you think that would help?

That is where I would focus initially. Make sure you are not creating deadlocks that are being later released by side-effects.

[heech]

Success!

The problem is definitely related to the usage of lock. Commenting them out (and ignoring the sync issues)... everything speeds up by a factor of 5x.

I'm extremely weary about changing lock behavior in my live code. After all, it's been running well for many years + millions of dollars on the line.

Now, since we know backtesting is single-threaded... should I just change the code around to get rid of locks entirely in back-testing, and then switch to monitors in live code?

EDIT: Nevermind... speed is definitely up, but putting back in locks one by one hasn't made it slow down either. I suspect there must be some other (unintended) change I made in my code... I'll update once I figure out whats going on. (Thank goodness for version control.)