Most efficient way to reference indicators when working with large instrument lists

[kbeary33]

Hi,

I am working on a strategy that ranks a great many instruments based on momentum, and buys the highest ranked. As part of this strategy, I need to call various indicators for each instrument (ATR, ROC, etc).

When I am doing optimization runs, the memory usage gradually explodes, until it has used up all 12GB of RAM.
> I am running a strategy with ~300 instruments, on daily data, from 1/1/2000 to 1/1/2010
> After ~400 runs, the optimizer has used up all of the memory.

I have read that every time you change the inputs in an indicator, NinjaTrader creates a new instance of it, but it doesn't dispose of the old one. Is that what is causing my memory leak?

For example, say I had 100 instruments added to my strategy:
- If my first optimization run called ATR(10) for each instrument, there would be 100 indicators stored in memory
- If my second optimization run called ATR(15) for each instrument, there would now be 200 indicators stored in memory
-... and so on


I have read that the best way to limit memory usage by a strategy, and improve calculation times, it is best to add an instance of the indicator that you manually reference. Please see the following thread:

http://www.ninjatrader.com/support/forum/showthread.php?t=38831

In Initialize()
private ATR myATR = ATR(10);
...

In OnBarUpdate()
double currentATR = myATR[0];
...

I have been able to make this work for a strategy with only a single (or small number) of instruments, but I'm not sure how to approach this in a strategy with 500+ stocks that are being evaluated.

What would be the best way to perform calculations and reference indicators, such that the memory usage would be kept under control?

Is there some way that I could iterate through the list, and create an instance of the indicator for each instrument, without typing it out manually?

[B]For example, coding manually:[/B]
private SMA sma0 = SMA(BarsArray[0],smaPeriod)
private SMA sma1 = SMA(BarsArray[1],smaPeriod)
...
(500 Lines)
...
private SMA sma500 = SMA(BarsArray[500],smaPeriod)

[B]For example, what I would like to somehow do:[/B]
for (int i = 0, i < numberInstruments; i++)
{
private SMA sma[i] = SMA(BarsArray[i], smaPeriod
}

[NinjaTrader_PatrickH]

Hello kbeary33,

Thank you for your post.

I am looking into this matter and I will follow up with you when I have additional information.

[NinjaTrader_PatrickH]

Hello kbeary33,

Thank you for your post.

This concept will not work for all bar series (assigning the values to a double in OnBarUpdate()).

You can instead limit the MaximumBarsLookBack to 256 rather than Infinite: http://www.ninjatrader.com/support/h...ookback256.htm

You can also use the 64 bit version of NinjaTrader during optimization, which will allow the application access to more system resources meaning you could see improvement in performance.

[kbeary33]

Hi Patrick,

I am already running NT in 64bit mode, and my MaximumBarsLookBack is already to 256. This isn't the source of the issue.

Is my other assumption correct - that NT doesn't release indicators from memory once they've been generated?

[NinjaTrader_PatrickH]

Hello kbeary33,

Thank you for your response.

Correct, MaximumBarsLookBack and 64 bit version are likely not the items of concern. These are suggestions to improve the performance of NinjaTrader, if you are already implementing these then it is time to scale back on the number of instruments loaded into your strategy.

During an optimization the indicators values for each instrument tested on will need to be stored in memory and then this will be again stored in memory for the next combination of parameters.

[kbeary33]

Hi Patrick,

Thanks for the reply.

To confirm:
- Optimization run #1 -> NT saves all the indicator values for the strategy
- Optimization run #2 -> NT saves all of the indicator values for the strategy (based off of different paramters than in run #1), but does not release the values that were saved in run #1

Is this correct? If so, is there any way to force NT to dispose of the saved indicator values from the first run? If I'm evaluating a new set of parameters, I obviously don't want to keep the old values hanging around. My strategy requires me to test over a large number of instruments, and so just cutting instruments out of my strategy isn't an option.

[NinjaTrader_PatrickH]

Hello kbeary33,

Thank you for your response.

We should clarify on Optimization run here; is the optimization run you refer to each time you click 'Run Optimization'? Or each run through of the parameters the Optimization does?

I look forward to your response.

[kbeary33]

Hi Patrick,

Sorry, to clarify: I mean for each combination of parameters within a given optimization run

Ie, optimizing
longSMA = 10;20;1
shortSMA = 2;9;1

The 10-2 combination would store one set of dataseries for the indicators, the 10-3 combination would store another, the 10-3 another, etc.

Is this right? Or am I not understanding this correctly?

[NinjaTrader_PatrickH]

Hello kbeary33,

Thank you for your response.

That is correct, each combination would be saved to store in the results.

[kbeary33]

Hi Patrick,

To clarify - I'm not talking about the trades collection, or system performance (I wasn't clear on your response whether we're on the same page)

Just to confirm:
- Say that I have 100 instruments in my strategy, and the strategy calls SMA(10) and SMA(20) for each of the instruments (Meaning 200 indicator calls each time the strategy runs)
- Will the dataseries related to those indicators be stored for each combination of parameters?

Sorry for dragging out this point, but this sort of stuff is a bit of a black box for me. All of the threads on this forum that I've read relating to memory management say that the .net garbage collector will automatically remove unused resources. However, it's also my understanding that if any objects maintain any sort of reference to a memory location, the garbage collector won't empty it.

- What I'm thinking is that, because the different parameter runs might have some overlapping data, NT decides to keep references to all of it.

Is this correct? Or am I way off base?

[koganam]

Hi Patrick,

Thanks for the reply.

To confirm:
- Optimization run #1 -> NT saves all the indicator values for the strategy
- Optimization run #2 -> NT saves all of the indicator values for the strategy (based off of different paramters than in run #1), but does not release the values that were saved in run #1

Is this correct? If so, is there any way to force NT to dispose of the saved indicator values from the first run? If I'm evaluating a new set of parameters, I obviously don't want to keep the old values hanging around. My strategy requires me to test over a large number of instruments, and so just cutting instruments out of my strategy isn't an option.

You can try explicitly Dispose()ing the the named instances when (CurrentBar == Count - 1).

You likely have to create and populate those named instances manually. You can always write a CMD file to write all the lines for you, then cut and paste them into your code.

[kbeary33]

Hi koganam,

Thanks once again for your help. This is a bit beyond my capabilities - do you happen to have a code example that I could work off of?

In the course of troubleshooting this, I have already set up named instances of each indicator using the code below:

// In Variables
private List<SMA> SMACollection;
private bool init = false;
private SMA sMACalc;
...

// In Initialize
SMACollection = new List<SMA>();
...

// In OnBarUpdate
if(!init)
{
	for (int i = 0; i <= numInstruments - 1; i++)
	{
		sMACalc = SMA(Inputs[i], SMAPeriod);
		SMACollection.Add(sMACalc);
         }
	init = true;
}
...

// In Termination
SMACollection = null;

Then, when I want to call an SMA for particular instrument, I just use:

double currentSMA = SMACollection[instrumentIndex].Value[0];

Is this heading in the right direction? Or will this not do anything beyond just calling the indicators in line?

[koganam]

Hi koganam,

Thanks once again for your help. This is a bit beyond my capabilities - do you happen to have a code example that I could work off of?

In the course of troubleshooting this, I have already set up named instances of each indicator using the code below:

// In Variables
private List<SMA> SMACollection;
private bool init = false;
private SMA sMACalc;
...
 
// In Initialize
SMACollection = new List<SMA>();
...
 
// In OnBarUpdate
if(!init)
{
    for (int i = 0; i <= numInstruments - 1; i++)
    {
        sMACalc = SMA(Inputs[i], SMAPeriod);
        SMACollection.Add(sMACalc);
         }
    init = true;
}
...
 
// In Termination
SMACollection = null;

Then, when I want to call an SMA for particular instrument, I just use:

double currentSMA = SMACollection[instrumentIndex].Value[0];

Is this heading in the right direction? Or will this not do anything beyond just calling the indicators in line?

No, I do not have specific code that you can reference.

So it seems this is a multi-instrument strategy? Where and how are you adding the instruments?

[kbeary33]

Yes, this is a multi-instrument strategy (currently ~300). I add the instruments in the Initialize function (manually right now, because I ran into issues tracking down the start dates for each of them when I just added the list)

int instrumentCounter = 0;

/* Code for adding using instrument List (not used right now)
foreach (Instrument i in currentInstrumentList.Instruments)
{
	Add(i.FullName, PeriodType.Day, 1);
	instrumentCounter++;
}
*/
				

Add("A",PeriodType.Day,1);       /*  Nov 18, 1999   ||   Agilent Technologies Inc  */    instrumentCounter++;
Add("AA",PeriodType.Day,1);       /*  Jan 02, 1962   ||   Alcoa Inc  */    instrumentCounter++;
Add("ABC",PeriodType.Day,1);       /*  Apr 04, 1995   ||   Amerisource Bergen Corp  */    instrumentCounter++;
Add("ABT",PeriodType.Day,1);       /*  Apr 06, 1983   ||   Abbott Laboratories  */    instrumentCounter++;

...
...

[kbeary33]

For anyone that runs into a similar issue as me, I have found that moving all of my indicator functions inside of my strategy (ie, not making any external indicator calls) solved this memory issue.

Unfortunately, I was never able to find another way around it, but at least NinjaTrader doesn't bring my computer to its knees anymore...