Memory efficiency

[redwoood]

Hi,

I noticed that NT can be quite a memory hog in some situations.
In particular, when you load a lot of historical data, NT seems to allocate a lot of memory.
(And when I say a lot, I mean it can blow up until it crashes)

If I had to guess, I would say you try to download all the data to memory and then dump it to disk afterwards (as is done in other places as well, e.g. when exporting data).
Now, I don't know if this is done by design or mistake, but clearly, fillling a limited-size buffer and dumping it to disk every once in a while when it fills would be a better strategy (especially considering the sequential nature of the data.

redwoood

[NinjaTrader_Jessica]

Thank you for the suggestion.

Here are some performance tips that may help:
http://www.ninjatrader-support.com/HelpGuideV6/PerformanceTips1.html

[redwoood]

Thank you for the suggestion.

Here are some performance tips that may help:
http://www.ninjatrader-support.com/HelpGuideV6/PerformanceTips1.html

Thanks, but that really doesn't apply in this case.

Just for clarification: when I say loading historical data, I mean from the tools, not inside a chart.
i.e. I just want to get the historical data from my data provider into NT's database.

Of course I could limit the size of my historical data requests, but
a) I don't have an accurate way of telling what request translates to what size
b) There really should be no reason for NT to keep all this data in memory. Buffering is quite
a standard technique in such a case.

So just a suggestion for Ray et al to look into the way historical data is handled. Shouldn't be all that much work to improve the implementation.

Thanks

redwoood

[NinjaTrader_Jessica]

Thanks Redwood,

Sorry those tips weren't of much help in your situation, but I will forward your suggestion for our list of future considerations.

Enjoy your weekend and happy trading.

[redwoood]

[...] I will forward your suggestion for our list of future considerations. [...]


Uhm, I'm not sure if this has become clear or not, but this is a bug report, not a feature request.
NT can crash when historical data requests exceed RAM size. That's a bug in my book.
My "suggestion" is how to fix the bug.

redwoood

[NinjaTrader_Dierk]

>> NT can crash when historical data requests exceed RAM size
Unfortunately there is no fix, since you always could overload your system by requesting too large a series.

[redwoood]

>> NT can crash when historical data requests exceed RAM size
Unfortunately there is no fix, since you always could overload your system by requesting too large a series.

Sorry, if an application crashes, it's a bug not a feature ;-)
You need to handle this more gracefully.

In the use case described above, ie reloading historical data through the tools menu, it should never have to run out of memory,

redwoood

[NinjaTrader_Dierk]

Thanks for your suggestion. We'll add it to the list of future considerations.

[nicko9]

Dierk, I think there is more to it than that. I have tried everything I can think of to avoid the repeated crashes by NT due to out of memory overflow errors.
I have discovered that by increasing RAM and processing power I get little incremental benefit. I find that typically when NT reaches a RAM usage of approx 1.3-1.5 Gb it will crash regardless of the system it is running on and the available memory resources. (this was the case running on a machines with 1-4gb of RAM).

Installed now on a machine running 8gb of RAM in a 64 bit OS, it will still crash at the same point (Ram usage/#instruments and look-back bars).

Is NT limiting itself in some way, regardless of the resources available to it? My machine is hardly breaking a sweat but still NT crashes due to insuffuicient memory. Or, maybe could the error message in the log/trace that is always referring to the memory issue, be misguided and it is isn't a memory problem after all?

thanks

Nick

[NinjaTrader_Dierk]

>> I find that typically when NT reaches a RAM usage of approx 1.3-1.5 Gb
Correct, since no 32bit process can allocate more than 1.5GB or so. I read something to that effect recently (since I checked). Unfortunately don't have a link at hand right now.