Bug: Daily data not added correctly via AddDataSeries()

[Harry]

This is not expected behavior, but a fat bug!

This is actually expected behavior. The daily bar for today is only built when connected to the provider (in this case Kinetick). When disconnected, the last daily bar is the previous day.

I know that today's bar is only built when connected. I am talking about the previous bar which is not being processed.

This is what I have observed:

When I open a chart today, today's AND yesterday's daily bars are not being processed (when disconnected).

This is expected for today's bar, but it is not expected for yesterday's bar, which should have been processed by OnBarUpdate, when yesterday's session closed.

Actually today (Thursday) the last daily bar processed by OnBarUpdate() was Tuesday's daily bar.

I have now added another chart showing the problem.

[Harry]

Hi PatrickH,


This thread seems to be bewitched.

Is there any hope that anybody will deal with the bug?

The problem is three and a half weeks old, easy to understand and can be reproduced every day.

[Harry]

I think that I have now understood what is happening.

Minute data

When connecting to a datafeed with an open minute chart, NinjaTrader downloads a fractional historical minute bar and adds real-time ticks. The fractional minute bar is saved in the historical data base. Once you disconnect from the data feed and reopen the minute chart, it will display the fractional minute bar as last bar.

As this fractional minute bar is not yet closed, indicators which are set to Calculate.OnBarClose do not perform calculations on the fractional bar. The indicator only plots for the second but the last bar, which has already closed.

Daily data

When connecting to a datafeed with an open daily chart, the current fractional daily bar is not downloaded to the daily data base. It is simply built from historical data and incoming real-time ticks.Once you disconnect from the data feed and reopen the daily chart, it does not show an current daily bar, but the last bar which is shown is yesterday's bar.

Now when not connected to a datafeed, we get the following situation

- today's bar is not plotted
- yesterday's bar is the last bar and is only processed in Calculate.OnEachTick / Calculate.OnPriceChange
- the last bar for which the indicator value is calculated on Calculate.OnBarClose is the bar closing two days ago

Basically, the concept "Calculate.OnBarClose + Calculate.OnEachTick" is not compatible with a situation, where the fractional bar has been removed. Although yesterday's daily bar is closed and sitting in the historical data base, it cannot be accessed by the indicator in mode "Calculate.OnBarClose".

Adding daily bars to an intraday chart

Here is where the real problem arises. When I disconnect from the datafeed and set my indicator to Calculate.OnBarClose

-> today's bar is not processed because there is nothing in the historical data base
(this is what I expected)
-> yesterday's bar is not processed, because the processing logic of daily bars stops with the bar 2 days ago, when the indicator is set to "Calculate.OnBarClose" (this was totally unexpected)

Strange design

This is a major design flaw, because it means that you cannot access the last historical daily bar via OnBarUpdate, when not connected to a data feed. Unfortunately, yesterday's daily bar is the most important for the purpose of calculating indicator values. It is now sitting in the historical data base and cannot be accessed.

Workaround

The design affects historical charts only, as the bar processing works as expected when NinjaTrader is connected to a data feed. There is the strange situation where yesterday's historical bar is sitting in the data base, but cannot be accessed by OnBarUpdate(). The only remaining option here is to use GetBar() which will collect yesterday's daily bar.

We already know that GetBar() does not work correctly for holidays with an early close and without settlement. In that case GetBar() needs to be looped back, until the last close (typically Friday's close, as most of the holidays are Mondays) is found.

Summary

The whole problem arises from applying a concept "Calculate,OnBarClose/Calculate.OnEachTick" to a case for which it had not been designed. The concept assumes that the last bar is a fractional bar and allows for differentiating between preliminary and final results.

When this concept is applied to a case where the fractional bar is simply removed and where the last bar is a complete bar, it simply introduces an unnecessary additional lag of 1 bar. As a consequence yesterday's daily bar cannot be processed, neither on a daily chart, neither on an intraday chart, when the indicator is set to Calculate.OnBarClose.

A complex workaround is available via GetBar(). The workaround itself has a problem as the method points to future daily bars. This becomes visible when looking at the NinjaTrader Pivots indicator, which partially uses future data instead of yesterday's data for calculating the pivots. The pivots are only correct in 97.5% of all cases.