Barchart Data

[monti1a]

Hi-

I am using Barchart.com as my data provider for stocks.

Whenever, I refresh an intraday chart (ex., 1-minute), my indicator signals that were present during real-time earlier, suddenly disappear upon refresh.

I thought that barchart provided native timestamping for data and this should not occur. Am I correct, and/or is there is a solution to correct this problem?

[NinjaTrader_Jason]

Hello monti1a,

Can you please tell me which exact indicators disappear upon refresh.

Do they disappear completely or do they start over?

I suspect these indicators are real-time indicators only. Please see the link below for more information.
http://www.ninjatrader-support.com/HelpGuideV6/PowerVolumeIndicators.html

[monti1a]

Hello monti1a,

Can you please tell me which exact indicators disappear upon refresh.

Do they disappear completely or do they start over?

I suspect these indicators are real-time indicators only. Please see the link below for more information.
http://www.ninjatrader-support.com/HelpGuideV6/PowerVolumeIndicators.html


Jason-


My indicators are "custom" written indicators that are ONLY based on bar CLOSE. They are not real-time indicators, and they only reference the CLOSE of the bar in their logic.

I should also mention that Barchart.com is not my primary connection. It is my second connection.

Sometimes they just disappear upon refresh, other times they are replotted on different bars than before.

[NinjaTrader_Jason]

Hello monti1a,

Does the issue occur with any of the default indicators available in NinjaTrader?

[monti1a]

Hello monti1a,

Does the issue occur with any of the default indicators available in NinjaTrader?


Jason-

The issue is not related to whether indicators are correctly performing calculations. The indicators are working fine.

This issue appears to be caused by how ninjatrader is building the chart BECAUSE upon refresh the O,H,L, or C of a particular price bar sometimes changes.

According to ninjatrader's documentation in the "How does Ninjatrader build Chart Bars?" section:

"Why can my chart look different after reloading historical data from the server? As ticks come into NinjaTrader in real-time, they are time stamped based on your local PC time if they do not already have an associated time stamp that is provided from the real-time data source. The majority of our supported brokerage feeds DO NOT time stamp ticks where most of our supported market data vendor feeds do provide time stamped ticks. NinjaTrader then builds bars based on the time stamp of the incoming tick and displays these bars in your chart in real-time.
"


My question is:

1.) Does barchart.com provide native timestamping?

2.) If so, then why does the O,H,L, C sometimes changes upon refresh. Shouldn't you get the same bars with the same OHLC upon refresh if the bars are natively timestamped by barchart.com?

[NinjaTrader_Jason]

1. Yes, BarChart provides native timestamping.

2. I am not sure. NinjaTrader acts like a radio receiver in that we display data as it is provided.

Could it be the real-time data you received from BarChart is changed once you reload historical data from BarChart?

[monti1a]

2. I am not sure. NinjaTrader acts like a radio receiver in that we display data as it is provided.

Could it be the real-time data you received from BarChart is changed once you reload historical data from BarChart?

Jason, do you mind asking somewhere there at ninjatrader if the real-time data I receive from BarChart is changed once I reload historical data from BarChart. Because if this is so, this means that it is IMPOSSIBLE to perform any BACKTEST on any HISTORICAL DATA through NINJATRADER because the historical data is UNRELIABLE. Ninjatrader markets backtesting as one of it's features. This is a serious problem.

If this issue is with Barchart.com, and the problem could easily be solved by switching to another data provider (ex., Esignal, DTN) please let me know, and I would be willing to switch if that solves the problem. As of now, it appears that historical data is unreliable in ninjatrader. Could you please escalate this issue to your superiors.

[NinjaTrader_Jason]

Hello monti1a,

I have notified my superiors regarding your case.

Can you please tell me what instrument is selected in the chart, so I can test it on my end to see if real-time data received by BarChart is changed once you reload historical data from BarChart.

[NinjaTrader_Dierk]

Please let me try to clarify (so I e.g. could raise an issue at barchart.com):
- you have unchecked Tools->Options->Data->Store realtime to make sure NT would not store any realtime data coming in
- you have unchecked Tools->Options->Data->Realtime tick filter to disable any realtime tick filter
- you are connected to barchart only (no additional connection)
- you open a 1 minute chart e.g. on MSFT
- you let this chart running say for 60 mins
- you take a screen shot of that chart = screenshot1
- now you disconnect, leave the chart open and reconnect
- as you then took a screen shot (=screenshot2) you would see that some bars in the past 60 mins are different than on screenshot2

Can you please confirm or make corrections if necessary?

Thanks

[monti1a]

Hello monti1a,

I have notified my superiors regarding your case.

Can you please tell me what instrument is selected in the chart, so I can test it on my end to see if real-time data received by BarChart is changed once you reload historical data from BarChart.

Currently, I am using stocks. You can try KLAC, and APC. However, I suspect any stock will work.

Load the stock, and then refresh in about 1-2 hours. "Some" of the bars will plot different OHLC values than before the refresh.

Note: my indicator plots changed for both KLAC and APC from the real-time plots when I refreshed their charts at approximately 10:56 am est.

[NinjaTrader_Dierk]

Thanks for your feedback. Could you please provide some info on my post #9? Thanks

[monti1a]

Please let me try to clarify (so I e.g. could raise an issue at barchart.com):
- you have unchecked Tools->Options->Data->Store realtime to make sure NT would not store any realtime data coming in
- you are connected to barchart only (no additional connection)
- you open a 1 minute chart e.g. on MSFT
- you let this chart running say for 60 mins
- you take a screen shot of that chart = screenshot1
- now you disconnect, leave the chart open and reconnect
- as you then took a screen shot (=screenshot2) you would see that some bars in the past 60 mins are different than on screenshot2

Can you please confirm or make corrections if necessary?

Thanks

1. I have "Tools->Options->Data->Store realtime" CHECKED. I was not aware that this box needs to be unchecked.

However, even if I have the box unchecked, won't we still encounter the same problem: that being, that the data coming in "real-time" will still be different than the "historical" data loaded on the chart?

If so, what's the purpose of loading historical data if it's incorrect (i.e., that's not what the chart looked like in "real-time")?

2. I have Mirus as my FIRST connection, and Barchart as my SECOND connection because mirus does not supply stock data.

[NinjaTrader_Dierk]

Thanks again for your feedback. Would you mind going through exact setup as per my post below in order to isolate the cause of trouble?

[monti1a]

Thanks again for your feedback. Would you mind going through exact setup as per my post below in order to isolate the cause of trouble?

I'm in the middle of trading. I'll have to deal with this later in the day.

[NinjaTrader_Dierk]

Excellent. Thanks again for your support.

[monti1a]

Thanks again for your feedback. Would you mind going through exact setup as per my post below in order to isolate the cause of trouble?

Dierk-

Before I waste my time going through the steps that you suggested - which involve me disconnecting from my data feeds - I need to know whether this is a "legitimate problem" or a ninjatrader "design" issue.

In other words, is the issue regarding data coming in "real-time" being different than the "historical" data loaded on the chart, part of the "normal" operation of ninjatrader (i.e., is this expected)?

The reason being your documentation states

"Let's say you have a tick (tick "A") with a time stamp of 10:31:00 AM which gets packaged into the 10:32:00 AM bar and happens to be the high of that bar. An hour later, you reload historical data from your historical data provider into NinjaTrader. This process will overwrite the existing data. The 10:32:00 AM bar now looks different since the high made by TICK "A" is now part of the prior bar, 10:31:00 AM. How is this possible?

Your PC clock could have been off so the time stamp is delayed
Your internet may have been lagging so the tick came in slightly delayed and therefore the time stamp is delayed
Due to standard latency, even 50ms delay (which is normal) could be the difference between a 10:30:59 and 10:31:00 time stamp
There is no way of knowing how the historical data provider packages their bars
The only way to ensure that data always looks the same is if every connectivity provider sent ticks with time stamps AND that all vendors synchronized on time stamps. Unfortunately, this is just not a reality nor plausible."


If you feel that this is a legitimate problem as opposed to an "EXPECTED design issue" than I would be willing to invest additional time in trying to resolve this.

[NinjaTrader_Dierk]

This is not a waste of time but to isolate the issue. Barchart provides native timestamps which is why the doc section below would not apply. The issue should not arise on a Barchart connection. Please proceed as per my post below to take next step. Thanks

[monti1a]

This is not a waste of time but to isolate the issue. Barchart provides native timestamps which is why the doc section below would not apply. The issue should not arise on a Barchart connection. Please proceed as per my post below to take next step. Thanks

Great. that is precisely what I wanted to know. I will work on it after 2 pm est.

[NinjaTrader_Dierk]

I forgot: Pleas make sure you have any tick filter disabled by unchecking Tools->Options->Data->Realtime tick filter (I amended instructions below accordingly)

[monti1a]

Please let me try to clarify (so I e.g. could raise an issue at barchart.com):
- you have unchecked Tools->Options->Data->Store realtime to make sure NT would not store any realtime data coming in
- you have unchecked Tools->Options->Data->Realtime tick filter to disable any realtime tick filter
- you are connected to barchart only (no additional connection)
- you open a 1 minute chart e.g. on MSFT
- you let this chart running say for 60 mins
- you take a screen shot of that chart = screenshot1
- now you disconnect, leave the chart open and reconnect
- as you then took a screen shot (=screenshot2) you would see that some bars in the past 60 mins are different than on screenshot2

Can you please confirm or make corrections if necessary?

Thanks

Dierk-

I followed your instructions and I have posted the two charts which shows that different charts are being produced:

-Barchart.com is the only connection.
-I placed a vertical blue line at 1515 EST as a marker showing when I connected to barchart.com
-I took a FIRST screenshot at 1550 ( HD 6_5_2009 (1 Min).1550.jpg)
-I DISCONNECTED, then RECONNECTED, and took a SECOND screenshot at 1551 ( HD 6_5_2009 (1 Min).1551.jpg)
-As you can see,.....bars 15:22, 15:23, 15:25, 15:26, 15:27 are different between the two charts. I stopped looking on bar 15:27 so there may very well be other bars that are different.

[NinjaTrader_Dierk]

Thanks for going through the setup. I will contact barchart.com with that information and ask for clarification.

[NinjaTrader_Dierk]

We got feedback from Barchart.com, that the Barchart.com API version used by NT6.5 could exhibit the behavior you described. Barchart have made some adjustments for Futures and Forex data, however it is possible for Stock minute bars to be off slightly during busy markets. The new version of the Barchart.com API used by NT7 does not exhibit these issues.

[monti1a]

We got feedback from Barchart.com, that the Barchart.com API version used by NT6.5 could exhibit the behavior you described. Barchart have made some adjustments for Futures and Forex data, however it is possible for Stock minute bars to be off slightly during busy markets. The new version of the Barchart.com API used by NT7 does not exhibit these issues.

Dierk-

Do you know whether this has been an issue with either Esignal or DTN?

[NinjaTrader_Dierk]

I'm not aware of an issue with DTN and/or eSignal. However, we always recommend doing your own research.