Historical Data Handling & NT Servers Vs. Providor Servers

[TradeThePlan]

Hi, I was recently told that, starting with a completely cleared data cache, if I open NinjaTrader 7 and then connect to my data feed, AND THEN open a chart, the data that will populate that chart with historical bars will come from data stored on NinjaTrader's servers, and then only real-time data will come directly from my actual data provider. And that if I want historical data directly from my data provider I have to use the Historical Data Manager to download it.

I have spent the past couple of hours trying to verify this, but I cannot find a solid answer. It seems to me that it depends on the data provider.

I read the following about Kinetick and Rithmic:

Kinetick's data servers are located in Omaha, Nebraska, which serves as a central location for traders in North America in close proximity to the exchanges in Illinois. Rithmic's servers are located directly in Aurora, Illinois. Both feeds will provide unfiltered tick data, including market depth data as reported by the exchanges. Both feeds will also provide an extensive resource of historical tick and minute data.

And I read the following in User-Guide:

NinjaTrader loads data from your data provider whenever it determines it could potentially not have all the data pertaining to the requested time period. To minimize the need to load data and to speed up chart load times, NinjaTrader maintains an internal cache of your prior accessed data. When data is in this cache, NinjaTrader will use it to populate your charts instead of loading from your data provider.
Note: The internal cache is managed by the .NET framework so it is unpredictable when the .NET runtime will clear it. In most cases, as long as you have at least one chart of your instrument open, the instrument’s data cache will persist.

And in a response to a question about a CQG connection, I found this post: https://ninjatrader.com/support/foru...ad.php?t=87466

The initial chart would load the data from our historical data servers, all subsequent charts would access cached historical data.

The historical data servers for CQG / Continuum / NinjaTrader Continuum data feeds with NinjaTrader are managed by NinjaTrader and are stored in Chicago.

You can check which server you're connecting to via the Control Center's log tab, when connected you should see "Using HDS" -- this will also list the IP address and server (remove the numbers after the / symbol if you need to run tests to the IP, also please do not post the server IP on the forum for security purposes)

My questions are the following:

1) Is there a difference between the historical data that would be downloaded to fill a blank chart (starting with an empty data cache) by simply opening a chart with N days back on it versus the historical data that would be manually downloaded via the Historical Data Manager? Or are both of these data coming from the same servers? (Essentially, can historical data come from NinjaTrader's servers for certain "download methods", and then for other "download methods" come directly from the actual data provider?)

2) Are only CQG's and Continuum's historical data managed by NinjaTrader's servers in Chicago, or does NinjaTrader's servers also manage historical from other data providers, such as Kinetick and/or Rithmic?

3) If NinjaTrader stores the historical data previously accessed in the "db" cache, then the only purposes for checking the option (in Tools>Options>Data) "Save chart data as historical" would be to 1) store the data that came in LIVE since the last time you downloaded historical data, or 2) to begin recording live incoming data as historical to bypass having to download historical data from NinjaTrader's servers, thereby potentially guarding against corrupted historical data. Are there any other reasons for setting this "Save chart data as historical" that you can think of? And if you are saving chart data as historical, if any chunk of that data is missing (for instance due to an internet outage), will the data for that day be redownloaded and replaced after connecting and opening a chart using that data? Or does data get downloaded iff and only if the full day is missing?

I know this is a lot to digest, and I apologize for that, but I am thinking that my understanding of how historical data is handled may be faulty and I have been unable to find the answers to these questions anywhere else.

My problem has been with my historical data getting corrupted and then missing huge chunks of data (sometimes small chunks that are less noticeable), and it would appear this problem was much more intense when I was using CQG, and clearing the cache and restarting didn't always fill in the gaps. But with Kinetick, clearing the cache and restarting DOES seem to fill in the gaps. But if the historical data I'm downloading just simply by opening NinjaTrader 7 with a bunch of workspaces and then connecting to Kinetick is somehow coming from NT's servers and not Kinetick's servers, I would like to know about it.

Thanks for taking the time to read this.

Best,

Daniel Roe

[TradeThePlan]

Question about Cache system

I also wanted to understand the caching system. All data is saved in the Documents/NinjaTrader 7/db folder, correct? Each data type gets stored in its own subfolder named 2) Data (market replay data), 3) Day, 4) Minute, 5) Tick. And the 1) Cache folder in this 'db' folder is for storing the data that is sorted according to session templates, correct?

When I read the following about how the data cache is managed and can be unpredictably cleared, I'm a bit confused:

NinjaTrader loads data from your data provider whenever it determines it could potentially not have all the data pertaining to the requested time period. To minimize the need to load data and to speed up chart load times, NinjaTrader maintains an internal cache of your prior accessed data. When data is in this cache, NinjaTrader will use it to populate your charts instead of loading from your data provider.
Note: The internal cache is managed by the .NET framework so it is unpredictable when the .NET runtime will clear it. In most cases, as long as you have at least one chart of your instrument open, the instrument’s data cache will persist.

So the data in this 'db' folder can get deleted somehow? Or is this referring to the OTHER cache folder located at Documents/NinjaTrader 7/cache (or some other internal caching system I cannot readily access)? I cannot find any info on this 'cache' folder. I know that when I compile any .cs Ninjascript, new DAT files are generated in this folder, and 'AccountStats' are saved in this folder, but that's about all I know. Are real-time data also stored in this 'cache' folder somehow, and then later cleared?

If this info is written in the user-guide somewhere, I could not find it.

Best,

Daniel Roe

[NinjaTrader_ChelseaB]

Hello Daniel,

This is referring to the Documents\NinjaTrader 7\db\cache folder which is generated by .NET.

This is created so that as you scroll the chart the data on the chart isn't building as you scroll. The data in the \db\cache folder is written and re-written often and is deleted by .NET if no chart is using for a long time.

[TradeThePlan]

Thanks for the reply, Chelsea. I suppose I should have put that question in a separate thread. The more important questions I had were in the post below this one. Could you take a look at that one, please? It's a big one

[NinjaTrader_ChelseaB]

Hello Daniel,

Historical data can be removed from within NinjaTrader.

We do not recommend deleting or modifying any files within the Documents\NinjaTrader 7\ folder unless directed by our staff.

If you are connected to a connection that provides historical data, historical data will automatically be downloaded for the period of the "Days to load" in the charts Data Series window. This is saved to the computer so that it will be quickly loaded in the future and is not downloaded every time you open a chart. This is saved to the Documents\NinjaTrader 7\db\tick, minute, and day folders.

You can also download historical data in the Historical Data Manager window on the Download tab as well if you would like.

There is no difference in this data.


If you are connected to a provider that does not provide historical data, or does not provide historical tick data specifically (such as Interactive Brokers) you can save the data from a real-time feed using the Tools>Options>Data) "Save chart data as historical" option.

This is saved as tick data to the same location that all historical data is saved to: Documents\NinjaTrader 7\db\tick.

This only records for instruments that are loaded in charts, superdoms, maket analyzers, that are open. Meaning you would need to leave a window with that instrument loaded and your connection always connected at all times or there will be gaps in the data.

I would not recommend this setting unless you do not have access to historical tick data and you need to record this.


Both CQG/Continuum based connections provide historical tick data from NinjaTrader's historical data servers for up to 360 days back (any time of day).

With Kinetick, when requesting tick data with Kinetick between the hours of 9:30am ET to 4:30pm ET you will have access to the last 8 days of historical tick data. If you request the data outside those hours the full last 180 days available. After you request the data, it will be downloaded to your historical database and cache, and you will be able to access this data at any time.

If you have the Tools>Options>Data) "Save chart data as historical" turned off (disabled so that historical tick data will come from the provider) and you right-click a chart and select Reload All Historical Data, and there are gaps, this would be coming from the provider's servers and should be reported to the provider.

What instrument and expiry are you loading?
When you say gaps do you mean jumps in price? Or do you mean there is data missing from the chart?
Is Tools>Options>Data>"Get data from server (if available)" checked and enabled?

[TradeThePlan]

Thanks, Chelsea! Great answer. Is there a way to remove just the historical data and data cache without having to use the "Reset DB" button and then lose all your ATM templates, etc.? I always just deleted the folders in the db folder, and then restarted NinjaTrader 7. The missing data seems to be random. It just occasionally gets corrupted, and then sections of data will simply be missing from the charts. Starting with brand new data from Kinetick fixes my problems, though, so I'm good there.

Do you guys have a standard size for the NinjaTrader.sdf that once larger, it can cause slower performance and should be reset?

I just had one last question about splits,which I wrote here:

If you, or someone else, could help me with that, I would be very grateful.

Best,

Daniel

[NinjaTrader_ChelseaB]

Hello Daniel,

Resetting the database will not erase the historical data files.

The database contains your trade information and atm strategies. It does not contain historical data.

To remove historical data open the Historical Data Manager.
Go to the Edit tab.
Right-click the data you want to delete -> click Delete.

To confirm, right-clicking the chart and selecting Reload All Historical Data does not correct the chart?

Also, I am unsure if you are also recording historical data which would cause this issue.

[TradeThePlan]

Ok thanks. So you can delete historical data only one instrument at a time then. Sorry, I thought Reset db also deleted cache. Right-clicking and Reloading data makes all my charts go blank about 50% of the time. I imagine it would clear up the gaps, though. And no, I am not saving chart data as historical.

One other question, do you guys have a standard size for the NinjaTrader.sdf that once larger, it can cause slower performance and should be reset?

[TradeThePlan]

I guess I'm not sure what the differences are between Resetting and Repairing the database. I can't find very much info on these. Can you shed a little light on these for me, please?

[NinjaTrader_ChelseaB]

Hello TradeThePlan,

The Database has nothing to do with historical data.

The database contains the trade history, atm strategies, and the instrument settings.

Resetting this will clear your trade history that you would view on the Account Performance tab and will delete any Atm Strategies you have set up.

The Repair is used if trades are not being entered correctly. Sometimes when trades are being mismatched this can correct further trades so the matching is correct.

Neither of these actions will do anything with historical data.