IB-BackFill-Format Data series

[MrBaffalo]

Thanks Dierk
I usually adjust contracts after rollover due to IB's lack of TBT data

When you say "presumably incomplete" NT does assume it's incomplete? or it does reload the last day in any way?
Sorry for this continous bouncing but I need to understand!
Thanks
Marce

[NinjaTrader_Dierk]

>> When you say "presumably incomplete" NT does assume it's incomplete?
Correct

[zoltran]

Originally Posted by NinjaTrader_Dierk

>> why as soon as I connect to IB (or other feed, it doens't change, esignal does the same) NT updates older days (days included yet in my "reliable" db).
NT will NOT do that but only loads from the last (presumably incomplete) day in the DB to the current day.

NT does overwrite.

Example ... using IB for R/T feed
1 - I have been collecting from my RT feed all day long. My quote database is nice and clean, and reflects my real-time experience
2 - Then I get an internet/tws glitch and NT disconnect/reconnects to TWS
3 - NT will now request the whole days worth of quotes .. overwriting all those good quotes that I already collected. Sometimes .. and I don't know the conditions .. it seems to get more than 1 day.

This is particularly troublesome with IB, as it's R/T feed is very different from any historical provider, even IB's own history data.

As we've discussed elsewhere .. NT is pretty unique in this regard, most other chart programs will not over-write previously collected data unless the user makes a manual request for it. They will usually just start where they left off.. sometimes leaving a gap if necessary. The user then would make a request to reload historical.

This is particularly important at contract roll-over times. Both the program and user needs to be careful about requesting to much data, as it's very easy to corrupt your newly merged 'front month' contract

[MrBaffalo]

Thanks Dierk and thanks Zoltran...I'm not ALONE!!!!
Just to add, sometimes if while reloading you reach the max IB data offers and charts are not fully reloaded, sometimes, days (chart) come out incomplete and NT writes only what IB offered.

[zoltran]

Marce

I think this happens because NT reloads the whole days data each time it does a backfill. IB will throttle to many requests .. and they have a pretty low threshold. Reloading the whole day for a number of instruments will certainly cause this to happen.

Dierk has explained why NT does this.
But it certainly causes other issues, such as triggering the IB throttle limits, and over-writing existing data.


Hopefully the great folks at NT will find a solution.

[NinjaTrader_Dierk]

>> NT will now request the whole days worth of quotes .. overwriting all those good quotes that I already collected.
This is true for current (!) day only, but not for previous days.

To verify by yourself: next NT6.5 holds additional trace info like
2008-01-10 06:23:44:765 (YAHOO) Data.Bars.GetBarsNow: instrument='NVDA' from=10.01.2007 00:00:00 to=10.01.2008 00:00:00 period=Daily

which gives you an idea what data actually is requested from the provider.

[MrBaffalo]

This happens only if you have a clean chart (W/O indicators) or if you have an average, let's say a 200 periods on a 15 minutes, it goes back to collect data?
Thanks
Marce
p.s. will try on monday the trace trick

[MrBaffalo]

Ciao Dierk,
here is the snapshot if what we are talking about
ciao
marce

p.s. I did nothing on the workspace, it as I closed on friday

[NinjaTrader_Dierk]

The error text is very clear: You hit an IB internal limitation on # of requests for historical data. There is nothing NT can do.

[MrBaffalo]

Dierk the first image shows a backfill for more than friday (1 day back)

[NinjaTrader_Dierk]

We will publish a document shortly which will describe what data is loaded from the provider. Please be patient.