Two Pass Scanning of Data - Possible?

[tazatek]

Is is possible to scan through the data twice before plotting?

Or possible to get all the contents of all the OHLCV data in the initialize() method?

Why?

Some strategies I am creating will change previous values in a custom dataseries multiple times, depending on fluctuations in the future.

The final values in the dataseries would not be known until the last bar is evaluated, and then this custom dataseries would be used to plot the final indicator.

Is this reasonable to achieve in NT?

[NinjaTrader_Austin]

tazatek, could you please clarify what you mean? Data isn't plotted until .Set() is called for the DataSeries, so you could put off plotting as long as you want.

For your second question, it is not possible to get OHLCV data in the Initialize() method because Initialize() is called before any bars have any data.

If you wish to change values in a DataSeries (click here for DataSeries help guide entry) after it is set, you can also set the value for historical bars by including a "barsAgo" value that represents the number of bars ago that you want the double value to be stored at:

// to set the data point 3 bars ago of Plot0 to the value of 10
Plot0.Set(3, 10);

[tazatek]

I'm trying to remember how NT handles plots, so let me give an example that might clarify...

Let's say I have 3 bars... and that I create my own DataSeries object in initialize() called D, and will have 3 values in them D[1], D[2] & D[3]

On bar 1:
D[1]=3

On Bar 2:
D[1]=23
D[2]=2

On Bar 3:
D[1]=2
D[2]=12
D[3]=33


Not too tricky yet, just assigning past bars with new values in the DataSeries...

But now is the part I'm not sure of....

I need to start at Bar 1 and feed into bar 1 the value of D[1] into the algorithm, of which I can't know the value of until I've run through the last bar.

Which is why I was hoping there was a way to either reset the bar number to 1, or have early access to the OHLVC data in initialize()

When you run through a data set twice, it's called a two pass scan - at least in compiler theory it was

I'm sure there's a creative way to do this, but could use some help in figuring it out.

Thanks

Matt


PS - just thought of this...

If I'm in the final bar (of historical data) do I know that I'm in the last bar? and if so, could I set all the plot values in one swoop, like:

if(finalbar) {
Plot0.Set(0, D[3]);
Plot0.Set(1, D[2]);
Plot0.Set(2, D[1]);
}
else {}

This might get me around my complication as well.

Thanks again

[NinjaTrader_JoshP]

I am not quite following you. On Bar 1, just access index 1 should be no problem. On Bar 0 you will run into issues because there is no bar before that so you cannot access an index of 1.

if (Historical) tells you if you are on historical data. The moment that switches is when you know you just finished processing historical data.

[tazatek]

I am not quite following you. On Bar 1, just access index 1 should be no problem. On Bar 0 you will run into issues because there is no bar before that so you cannot access an index of 1.

if (Historical) tells you if you are on historical data. The moment that switches is when you know you just finished processing historical data.

I'm still not making myself clear.

The concept here is that bar1 would use inputs that are based on some bar in the future.

So what I'm looking for is a way to go through once, to populate my own dataseries, and then combine my dataseries data with the original OHLVC data, and then plot it.

Here's some pseudo code for what I'm trying to accomplish. Maybe this will clarify my position. If you're familiar with neural nets, where outputs are re-fed in as inputs, it's essentially what I'm doing...

int passes=1;//init pass to first pass
private DataSeries myDataSeries; // Declare a DataSeries variable

protected overide void onbarupdate() {

// This section is run on the 1st pass, and populates
// myDataSeries with various data
//
if(passes==1){

// this example just uses close from 2 days ago, but will be 
// almost random data by the time the 
// custom dataseries is populated
  myDataSeries.Set(Close[2]);

 // check if this is the last bar
  if(LastBarInSession) // (Made Up var)
  {
   // if this is the last bar, we set ourselves to run the second pass
  //  and reset our bar number to number 1
  // and re-run OnBarUpdate()
    passes=2;  // set to 2nd pass
    resetBars(); // Re-Start onBarUpdate() at bar 1 (made up method)
  }

}
// Begin 2nd pass functions...
// it can do anything, but I need it to take my custom dataseries
// and combine it with the original OHLCV data
//
else if (passes==2) { // Start the second pass

  // trivial example that combines my dataseries with close
  plot0.set(close[0]*myDataSeries[0]);
}

}

I made up some functions/variables to highlight what I'm trying to do, not to imply they really exist.

I'm not seeing any methods that reset the bar number, nor to check that we're on the last bar.

Also, the example could be done without 2 passes, but my real indicator cannot.

Did that help?

Any thoughts on feeding outputs from pass 1 as inputs into pass 2?

Is there a method or variable that will check if I'm on the last bar of historical data?

Thanks for your time.

Matt

[NinjaTrader_JoshP]

Matt,

You sound like you want to work with arrays instead of DataSeries. DataSeries objects are tied to the bars. Arrays can be whatever you need them to be. Unfortunately this would be outside the programming support we can offer though, but there should be plenty of resources on google for making and using arrays in C#.

Again, there is no check of "if this is the last bar". You know a switch from historical to live happened with if (Historical).

[tazatek]

Matt,

You sound like you want to work with arrays instead of DataSeries. DataSeries objects are tied to the bars. Arrays can be whatever you need them to be. Unfortunately this would be outside the programming support we can offer though, but there should be plenty of resources on google for making and using arrays in C#.

Again, there is no check of "if this is the last bar". You know a switch from historical to live happened with if (Historical).

Arrays/DataSeries ... in my case they serve the same purpose, but the arrays would have less overhead...

But I still need to access the OHLCV data twice... sounds like I can't do that with NinjaTrader....

Is there a method that will tell me how many bars are loaded that I can expect to process?

This would allow me to know when I'm on the last bar, and I can reset the plot values by hand

The indicator I'm working on will only work on historical data.. never live data.

Is there an open ticket for such a request? This sort of thing would be a huge boost to programming complex indicators.

Thanks for looking into this guys.

[Ralph]

Hi Matt,

here is an idea regarding the last bar available:

You could implement that part of the code in the Plot() method. When this method is called the first time, then you have all bar-data available until the last bar visible on the chart.

Under OnBarUpdate() you collect your date (as they come in) in a list and under Plot() you re-iterate the list and store the calculation results in another data structure.

Regards
Ralph

[tazatek]

Hi Matt,

here is an idea regarding the last bar available:

You could implement that part of the code in the Plot() method. When this method is called the first time, then you have all bar-data available until the last bar visible on the chart.

Under OnBarUpdate() you collect your date (as they come in) in a list and under Plot() you re-iterate the list and store the calculation results in another data structure.

Regards
Ralph

Ralph, this sounds interesting... how do I go about doing this?

Is this a simple matter of overriding plot() and calling the base plot()?

Is there an indicator that uses this? Could you provide a pseudo (or real) code example?

Thanks

Matt

[Ralph]

I used that concept time ago to create a colored indicator (EMA). I stored the coordinates for the colored segments in a list as bars did progress. And under Plot() I calculated and painted the line segments for the bar-range visible. Not the same problem you have and I don't know what a neural network is. But here is a proposal for what I think you described:

[FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]private[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] List<double> listPre  = [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]new[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] List<double>();[/SIZE][/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]private[FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] List<double> listPost = [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]new[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] List<double>();[/SIZE][/FONT][/SIZE][/FONT][/SIZE][/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2][/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]protected[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]override[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]void[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] OnBarUpdate()
{
[/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2][/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff] if[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] (FirstTickOfBar)
 {
  barCurrent = CurrentBar;
  listPre[CurrentBar] = CurrentBar > 1 ? Close[2] : 0.0;
  listPost[CurrentBar] = 0.0;
 }
}
[FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2][/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]public[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]override[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]void[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] Plot(Graphics GraphicsContext, Rectangle Canvas, [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]double[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] MinPrice, [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]double[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] MaxPrice)[/SIZE][/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]{[/SIZE][/FONT][/SIZE][/FONT]
 if (CalculateCondition)
 {
[FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff][FONT=Courier New][SIZE=2][COLOR=#0000ff]  for[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] (int i = [/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][COLOR=#800080][FONT=Courier New][SIZE=2][COLOR=#800080][FONT=Courier New][SIZE=2][COLOR=#800080]0[/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][/COLOR][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]; i < barCurrent; i++)[/SIZE][/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]  {[/SIZE][/FONT][/SIZE][/FONT]
   listPost[i] = listPre[i] * Close[0];
[FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]  }
[/SIZE][/FONT][/SIZE][/FONT][FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2] }[/SIZE][/FONT][/SIZE][/FONT]
[FONT=Courier New][SIZE=2][FONT=Courier New][SIZE=2]}
[/SIZE][/FONT][/SIZE][/FONT][/SIZE][/FONT][/SIZE][/FONT][/SIZE][/FONT][/SIZE][/FONT]

If you don't take over the plotting itself then, yes, you need to add a base.Plot(...).

Here are some problems:
I implemented a variable barCurrent, because CurrentBar can have a different value in Plot() than in OnBarUpdate().
I inserted a condition named CalculateCondition. Here you should define something when to perform the calculation in the for-loop (maybe when barCurrent has changed). I guess you don't want to execute the loop every time Plot() is invoked.

Hope that helps
Ralph

[tazatek]

Excellent Ralph!

I think that'll do what I need. I'll give that a try and see what happens.

Matt

[tazatek]

Here are some problems:

I inserted a condition named CalculateCondition.

Here you should define something when to perform the calculation in the for-loop (maybe when barCurrent has changed).


Ralph

Looking over this, I'm still in the same boat...

I don't know how to determine CalculateCondition... I know that it needs to run after all bars have been updated, but I'm at a loss to know when that happens.

This should only be called on the last plot() call (in essence the last onbarupdate()) ...

surely there's a way to do this... ( and quite calling me Shirley)

Matt

[Ralph]

I think it is a different boat now, Matt.

If I understand correctly, you would like to see your neural results on the chart, as soon as something is plotted on the chart, and that is when Plot() is called first time. And perhaps you would like to recalculate the results when the visible part on the chart moves on into the future (last bar visible has increased, no matter if the last historical bar has been reached or not). That's the way I calculated the segments for the indicator mentioned.

Regards
Ralph

[tazatek]

Thanks for your help Ralph.

I've submitted an Enhancement Request for this functionality.

I see power in it, but we'll wait and see what the NT dev's think.

Thanks again.

Matt

[Ralph]

http://www.ninjatrader-support2.com/vb/showpost.php?p=128053&postcount=3

Have you seen that post, Matt? It is the solution to your problem.

Regards
Ralph