Difference between live and historical indicator plot

[molecool]

This one won't be easy to explain so let me show you guys first. Take a look at the two enclosed PNGs. Both are the very same chart, with one exception: The one titled 'live_data' has been running against live data for the past day or so. The second one titled 'historical_only' was launched just now - very same indicator, same settings, etc. In other words - if I go to the chart labeled 'live_data' and reload the data it will suddenly look like 'historical only'.

Now part of what I am doing in my indicator is to measure single tick variations (yes, I am using 1 tick periods as part of a multiple instrument indicator) and those become part of my equation. But that data should still be there when reloading the chart as those tick variations are in my historical data, right? (Kinetick)

Unfortunately I cannot share my code here, but based on what I am describing can anyone suggest what may cause these plot differences? This is the first time this is happening to me after years of developing indicators for NT, and either it's a bug, a delta between live/historical data, or something I screwed up when writing this thing.

ANY input would be very appreciated.

Cheers,

Michael

[kdoren]

It's hard to say without knowing more about your algorithm, but I suspect the issue might be relative timing of the ticks from the different instrument.

With live tick data, OnBarUpdate will be called tick-by-tick in the order the ticks arrive.

The ticks are stored in the NT7 database with 1-second resolution. If ticks from 2 different instruments arrived within 1 sec of each other, they could have the same timestamp.

When processing historical tick data, NT7 may call OnBarUpdate in a different order than what happened originally, because it doesn't know which tick came first. If your code is sensitive to that, it could cause you trouble.

-Kevin

[molecool]

Awesome reply

Kdoren - thanks for responding to my thread again - you continue to offer valuable insights.

Yes, that's exactly what's happening. When looking at the two plots in comparison it is clear that all the extreme swings have been averaged out, so it appears that pulling historical data somehow slows everything down to 1-sec resolution.

I was wondering if it may be possible to externalize my live indicator plots - either to the NT7 database or to a file and then load that instead. This way what I see plotted as the result of live data can be recovered upon application relaunch. Has this ever been done? Are there hooks in NT7 to store indicator plots?

Thanks!

Michael

[NinjaTrader_Bertrand]

Thanks Kevin - yes that's likely the 1 sec resolution limitation you're running into Molecool - I believe you could look into the GomRecorder framework being discussed here on the forums as it deals with working around that current limitation (which is on our suggestion to look into for the future).

[molecool]

Thank you for your input Bertrand - could you perhaps point me to the right URL and the resources? I would like to start implementing this asap.

Best,

Michael

[NinjaTrader_Bertrand]

Hi Michael, you could start looking into this thread here - http://www.ninjatrader.com/support/f...ad.php?t=23283

[molecool]

Thanks a bunch. Quick question - will GomRecorder be able to persist my indicator data down the level of accuracy required?

On the other hand there is no issue with having a 1-sec resolution, my indicator is running on a 4000 tick chart and as long as the live indicator signal can be recorded it should work just fine.

Of course I will somehow have to figure out how to mix Kinetick live and historical data with GomRecorder's persisted data. Meaning - my indicator should plot via the data feeds as right now and only pull data from GomRecorder if it is available. Do you have an example of mixing both? Please let me know if this doesn't make sense.

[NinjaTrader_Bertrand]

Michael, honestly I would not know in your exact scenario - this was merely an idea how to attempt working around the limitation you unfortunately hit. If you could record the realtime indicator signal with the Gom framework then you could potentially reconstruct on reload as it would not have to use the slightly changed source data to recalculate it.

[molecool]

I just looked at this page:

http://www.bigmiketrading.com/wiki/trading-wiki/GOMRecorder

From what I'm reading this does not seem to be what I'm looking for. GOMRecorder is designed to store bid/ask data - what I am looking to store is simply an indicator signal. When you guys pull a data feed and make it available to indicators/strategies via Closes[0] for instance you are storing this data in the NT database, right? And it stays in there until it is being cleared/removed. Now - is it possible for me to write my indicator feed to the NT database my own 'data feed'? This way I would not have to do anything on relaunch - NT would simply look for historic indicator data in its database.

Does this make sense and can it be done?

[NinjaTrader_Brett]

Hello,

Unfortunately this is not supported.

About the only thing you could is store this yourself in your own text file.

1. Using StreamWriter to write to a text file - also applicable to strategies
2. Using StreamReader to read from a text file - also applicable to strategies
3. Using System.IO File properties to write to and read from a text file - also applicable to strategies

[molecool]

Hi Brett - yes, that is what I was thinking. I could of course use IO to write to disk - my only concern would be how to recreate the data and then merge it with NinjaTrader's internal data plots (i.e. Plot(0), Plot(1), etc).

Would I simply store the time stamp/value pairs and after parsing the data file try to set each historic bar to a particular value?

I also thought about a C# persistence framework - used to that with Java back when. This way I could persist the entire Plot object to disk. But I don't know if I would run into internal problems with NT7 if I tried to clear Plot(0) for instance and replace it with my own. Does that make sense?

[NinjaTrader_Brett]

Hello,

Yes this would be the only way I could think of to do it.

This is simply a sandbox for you to play in and add in any functionality you need however. How you do this would be unsupported since this is more c# programming instead of NInjaScript. However it can be done.

[gomifromparis]

With the GomFramework you can get your 1 tick resolution.
If you only have to put your tick logic in the GomOnMarketData() method, and this method will be called with actual tick data even on historical data.
Latest version of the framework on bigmiketrading forum is able to read directly from ninja tick data, so you don't have to maintain a gom file if you don't need bid/ask info.

example : intrabar volume profile, on historical data

[molecool]

Thanks for the response, Gomi. When you say 'my tick logic' - what are you referring to? Currently I have an indicator that does some tick logic in the onBarUpdate() method (which calls other internal methods on each bar update). I basically combine several instruments at tick intervals to produce my indicator signal which goes to Plot(0).

Would I have to significantly change my indicator to implement this? Would it be semi-transparent in that my indicator will either pull from the local file if there's data and if not go to my data feed?

My apologies if I don't completely grasp the concept of GOMFramework. My impression is that it's used to store bid/ask activity.