Looking for performance tips

[egan857]

Hi, I have a strategy that is a resource hog and I'm looking for general performance tuning tips. I'm looking for specific recommendations to improve my strategy, but I'd also really like to eventually see a support thread in the Tips section or a chapter in the documentation on performance tuning and/or memory management.

First, I'll describe my hog. This thing is now 6000+ lines, has almost 300 variables, some of them arrays. It issues long and short entries based on 68 entry criteria (roughly 34 long strategies and 34 short strategies, more or less mirrors of each other). Of these, I consider 16 to be production strategies and the rest to be in development, which means I'm constantly tweaking their entry and exit criteria until they give me the desired results for a while. I send a whole lot of messages to the output window: every time I enter a position, exit a position, exactly 10 minutes after exiting, and at other interesting points in between. A typical day has 1000-3000 lines sent to the output screen, which I manually copy and paste into a spreadsheet about once every 10-15 minutes.

The program uses 4 timeframes: 1-minute, 1-minute on symbol DIA, 1-day, and 3-minute. Most of the logic is in OnMarketData, not OnBarUpdate. Many of the variables are only calculated on a print update or once a minute, and the daily-bar calculations like average daily volume are done just once a day.

I have a 100-symbol Kinetick license and I run 200 instances of this program during the day: 100 for the Sim101 account and the same 100 symbols for my IB account. The program's smart enough to know not to buy into the Sim101 account if it is also buying the symbol in the IB account for the same reason.

In general, this program almost never uses over 1% CPU, but it can spike to 5%. The problem is memory. I have 8GB RAM, and really only about 6.2GB truly usable space for NT before NT slows to a crawl. It'll start off using only 1.2GB and will go up at a slightly increasing rate. It takes about 4-5 hours to hit 6.2GB depending on time of day. It doesn't seem to make much difference if I have 100 instances of the program running or 200 instances running for 100 symbols.

So, here are the questions:

1. All prints to the output window are retained in memory. Will I save memory by printing shorter lines, or are all the lines stored at a fixed length?

2. Does it add to memory usage to play sounds?

3. How much memory space is used up for each trade?

4. How much space would be allocated for a 10-item DateTime array? Is it all allocated at once, or does it take less space if only one element of the array contains data?

5. How much space would be allocated for a 10-item String array?

6. Is there any reason why running through a FOR loop would eat up memory? Specifically, I was looping once a minute from 1 to barsSinceSession (390 iterations at the end of the day) looking through the minute bars to do some calculations. I stored the answers in regular variables, not arrays, so it theoretically shouldn't have taken more memory to loop through 390 minute bars than to loop through 30 minute bars, but I think it did. I changed the logic to accumulate the data each minute instead of looping through all the minute bars each minute, and the memory usage seemed to go down.

7. Does it eat any extra memory to make calls to get account information? I still need the information, but if it saves memory I'll request the information less often.

8. Do the math operations (Math.Min, Math.Max, Math.Abs, etc.) eat up memory that doesn't get released?

9. As mentioned earlier, it seems that having two instances of the program running against the same symbol doesn't cost much more in memory. Why is that?

10. Are there any other tips you might have on memory management? I'm a newbie to C#, so please don't assume I know something that seems obvious to a pro.

[NinjaTrader_Bertrand]

Hi egan857, unfortunately that is an area which could not be easily generalized for increasing performance / optimizing code and we would not have specific memory allocations for the operations you perform. One thing that stands out is removing your Print statement to a minimum for runtime, so non debug use, one way to achieve this is have a ShowPrints bool for example as public input so you can toggle between debug and a more efficient runtime mode.

Given the more complex logging you do, you might be better off overall to move the logs to txt files instead.

Also, if you draw with your strategy - those objects would be at best kept to minimum as well.

Generally any action you do (PlaySound, Email, log, account info retrieval) will result in a performance 'hit' for executing your code - which section / part is the major factor in the used memory growth seen would need to be isolated by step by step debugging.

The memory handling in C# is managed by the internal 'garbage' collection, so for custom resources being used it would be important to correct dispose them to allow this process to free up resources taken.

http://www.ninjatrader.com/support/h...ermination.htm

[egan857]

I was hoping for more detailed info but that's OK. I will continue to experiment with different actions to get the memory usage down, including sending output to text files instead of the output window. Thanks!

[NinjaTrader_Bertrand]

I can understand, it's unfortunately not really possible with such an involved script to be more detailed other than giving general directions how to work stepwise on a more optimized version. For your collections do you work with Array's or Lists?

The biggest improvement will be likely seen when you ensure properly disposing any custom resources and minimizing print / drawing objects. Those are the common 'bottlenecks' users run into with their scripts.

[egan857]

If I ever use a collection, it's a one-dimensional array, and if I need to keep track of two things at once I use two one-dimensional arrays and keep both updated at the same time. I have 26 one-dimensional arrays, the biggest has 10 items, and most have 2 or 5.

I don't draw anything and the only printing is to the output window, usually details of the entry or exit.

I actually don't have a problem with garbage collection. I found that when you disable a strategy and then remove the symbol from the Strategies tab, all its memory is released in about a minute. This is what I do when I run out of memory and still have some trades working. When I only have simulated trades active and want to clear memory, I shut down NT, clear out the cache directories and then restart NT. That only takes about a minute, and it frees me to select a different list of 100 symbols to trade.

[NinjaTrader_Bertrand]

Sounds fine to me Egan, Arrays are great and performance efficient if you know how many items you need to keep track of...since you run Sim and Live trading on the same PC, it might be good to consider splitting this up in the future to take load away from the live trading setup...

[egan857]

Yep, I may end up using two servers. But first, I'm going to start writing the output to text because I want to expand to 200 symbols and be able to run them all for a whole day. I can't even run the production strategies alone for 100 symbols for a whole day because the output is stored in memory. That may not be the whole problem, but it is at least a part of the problem.