Historical data still not loaded correctly

[Harry]

NinjaTrader 7.0.0.21 with TWS 906

I am now loading daily data exclusively from Kinetick, so only loading intraday data from IB. I have a workspace with 3 intraday charts. There is still a specific problem when loading data from IB that only occurs with CL. So it is possibly connected to the known issue of the contract month, which is handled via symbol mode CL||||1.

When switching the workspace to CL 10-10, NT starts suspicious loading that results in data throttling. Today, it always loaded data for the front month 10-10 and the month 08-10, which is two months back. Causes data throttling, and sometimes had to abandon the loading.

Excerpt of trace file for intraday data loading for three connected minute charts below

2010-09-15 16:42:06:325 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 08-10' from='2010-07-16' to='2010-07-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-09-15 16:42:09:106 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=18.07.2010 instrument='CL 08-10' from='2010-07-16' to='2010-07-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:09:106 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201008;0;;;NYMEX;;USD;;20100719 00:00:00;1 min;5 D;0;TRADES'
2010-09-15 16:42:11:747 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6973 instrument='CL 08-10' from='2010-07-16' to='2010-07-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:11:794 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='^TICK' from='2010-09-14' to='2010-09-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='US Equities RTH'
2010-09-15 16:42:11:794 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=14.09.2010 instrument='^TICK' from='2010-09-14' to='2010-09-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:11:794 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='TICK-NYSE;IND;;0;;;NYSE;;USD;;20100915 00:00:00;1 min;5 D;0;TRADES'
2010-09-15 16:42:12:325 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=1950 instrument='^TICK' from='2010-09-14' to='2010-09-14' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:12:341 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 10-10' from='2010-09-15' to='2010-09-15' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-09-15 16:42:13:809 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=15.09.2010 instrument='CL 10-10' from='2010-09-15' to='2010-09-15' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:13:809 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201010;0;;;NYMEX;;USD;;;1 min;5 D;0;TRADES'
2010-09-15 16:42:15:247 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6583 instrument='CL 10-10' from='2010-09-15' to='2010-09-15' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:15:294 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='^TICK' from='2010-09-15' to='2010-09-15' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='US Equities RTH'
2010-09-15 16:42:15:825 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=15.09.2010 instrument='^TICK' from='2010-09-15' to='2010-09-15' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:15:825 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='TICK-NYSE;IND;;0;;;NYSE;;USD;;;1 min;5 D;0;TRADES'
2010-09-15 16:42:16:716 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=1633 instrument='^TICK' from='2010-09-15' to='2010-09-15' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:17:434 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 08-10' from='2010-07-16' to='2010-07-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-09-15 16:42:20:106 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=18.07.2010 instrument='CL 08-10' from='2010-07-16' to='2010-07-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:20:106 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201008;0;;;NYMEX;;USD;;20100719 00:00:00;1 min;5 D;0;TRADES'
2010-09-15 16:42:20:106 (Interactive Brokers) IB.HistoricalDataRequest.Send: waiting 5 seconds
2010-09-15 16:42:26:497 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6973 instrument='CL 08-10' from='2010-07-16' to='2010-07-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:27:622 (Interactive Brokers) Data.Bars.GetBarsNow: instrument='CL 08-10' from='2010-07-16' to='2010-07-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 session='Nymex Metals / Energy ETH'
2010-09-15 16:42:31:122 (Interactive Brokers) IB.HistoricalDataRequest.Send: last=18.07.2010 instrument='CL 08-10' from='2010-07-16' to='2010-07-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0
2010-09-15 16:42:31:122 (Interactive Brokers) IB.HistoricalDataRequest.Send: request='CL;FUT;201008;0;;;NYMEX;;USD;;20100719 00:00:00;1 min;5 D;0;TRADES'
2010-09-15 16:42:31:122 (Interactive Brokers) IB.HistoricalDataRequest.Send: waiting 10 seconds
2010-09-15 16:42:42:466 (Interactive Brokers) IB.HistoricalDataRequest.Process: itemCount=6973 instrument='CL 08-10' from='2010-07-16' to='2010-07-18' period=1 Min splitAdjusted=False dividendAdjusted=False bars=0 

[Harry]

Explanation of the bug

Got further with this. If you look at the tracefile below, you will find that NT 7 tries to load data for the week-end July 16 (Friday) to July 18 (Sunday) for the long expired contract CL 08-10.

This unnecessary request causes the data throttling and the popups.

I checked my database. It is fine and contains all necessary data, but here comes the catch:

-> July 19 is rollover date, and the data base contains data for July 19 for the new contract CL 09-10. which is needed to display the mergebackadjusted data -> correct
-> Also the database contains no data for July 19 for the old contract -> correct, this is not needed
-> The problem occurs, when rollover data falls on a Monday, NinjaTrader tries to load the old contract 08-10 data for the week-end over and over again, although the last session prior to rollover day is complete.

Of course there is no data, so this only causes data throttling and error popups.

[NinjaTrader_Brett]

Hello,

Thank you for your patience.

We will respond shortly.

Curius, is this the paid Kenitick or the free end of day kinetick?

[NinjaTrader_JoshP]

Thanks. We will take a look and get back to you.

[Harry]

It is the free EOD Kinetick, which I use. I am very happy with this datafeed, no problems whatsoever so far! And also I will not complain anymore that NT cannot read IB's daily data correctly, as this is no more required. I connect to Kinetick first for daily data and then second connect to other datafeeds for intraday data.

This is an important improvement of NT7 versus NT 6.5

Hello,

Thank you for your patience.

We will respond shortly.

Curius, is this the paid Kenitick or the free end of day kinetick?

[Harry]

Just to confirm that the loading of historical data for CL hangs every day, because NT tries to load that non-existent week-end data for CL 08-10 prior to rollover to CL 09-10. Rollover date on Monday as desribed below.

[NinjaTrader_Dierk]

Got further with this. If you look at the tracefile below, you will find that NT 7 tries to load data for the week-end July 16 (Friday) to July 18 (Sunday) for the long expired contract CL 08-10.

This unnecessary request causes the data throttling and the popups.

I checked my database. It is fine and contains all necessary data, but here comes the catch:

-> July 19 is rollover date, and the data base contains data for July 19 for the new contract CL 09-10. which is needed to display the mergebackadjusted data -> correct
-> Also the database contains no data for July 19 for the old contract -> correct, this is not needed
-> The problem occurs, when rollover data falls on a Monday, NinjaTrader tries to load the old contract 08-10 data for the week-end over and over again, although the last session prior to rollover day is complete.

Of course there is no data, so this only causes data throttling and error popups.

Harry,

We looked into to understand why there are the repetitive requests for data on Fri-Sun. Long story short (would not want to get into the last details here): trouble is NT has no logic like "ok, I tried to pull data from the provider for Sun, but there was none and for that reason I will no longer go to the provider trying to pull data for Sun".

The issue is exposed since the CL rolls on Mon and NT needs data on the day before (=Sun) to properly calculate offsets.

-> unfortunately this is a limitation

Hope this makes sense. Sorry for the bad news.

[Harry]

This has the following consequence:

If my connected charts (lookback 60 too 100 days) show CL and I connect to NinjaTrader -> data throttling and empty charts.

So I have to switch my workspace to another symbol first , then connect. Then use the historical data manager to load backfill and then switch to CL.

Using NinjaTrader is sometimes like flying an airplane in the beginning of the 20th century.

Harry,

We looked into to understand why there are the repetitive requests for data on Fri-Sun. Long story short (would not want to get into the last details here): trouble is NT has no logic like "ok, I tried to pull data from the provider for Sun, but there was none and for that reason I will no longer go to the provider trying to pull data for Sun".

The issue is exposed since the CL rolls on Mon and NT needs data on the day before (=Sun) to properly calculate offsets.

-> unfortunately this is a limitation

Hope this makes sense. Sorry for the bad news.

[NinjaTrader_Dierk]

I hear you. Only way around would be having a rollover date which would NOT be Mon.

[Harry]

I hear you. Only way around would be having a rollover date which would NOT be Mon.

LOL. Instruction in the help file of NinjaTrader: "Do not select any Monday as a rollover date with Interactive Brokers data feed ."