Tick replay in strategy analyzer

[ToMer_K]

Hello ToMer_K,

Thank you for your patience.


Item # 1:

Tick Replay updates OnMarketData for each tick that occurs. OnBarUpdate() is based on the timestamp of the bar. If multiple bars come through with the exact same time stamp down to the millisecond then OnBarUpdate() will only see one event. In your specific case, you see more OnMarketData events as the OnBarUpdate would not be called for each tick that had the same timestamp down to the millisecond.


Item # 2:

Each event in Tick Replay will be a Last event for MarketDataType. This is expected.

(1) I still think it is a bug if a certain code expects OnBarUpdate on each incoming tick. For instance, if I want to calculate how many ticks (=trades) happened in succession until the price moved. I hope this is not the case when building the tick based bars.

Edit:
I just realized that every time someone creates a 1 tick chart and expect to see each tick and its volume is getting false graph in NT8. That is a serious bug! Did you implement the tick bars in the same way in NT7? If so I need to disqualify my tick research there too

(2) Tick replay, in the case you are using "Bid"/"Ask" data, should not provide false "Last" events. Or it should show the real tick type ("Bid"/"Ask") or it shouldn't show these events at all.

Think about it, a developer is making an indicator that calculates bid or ask data for plotting purposes, which has to rely on volume that appears in OnMarketData, and in real time it fills the values correctly (differing "Last", "Bid" and "Ask" events) and when calculating historically, all OnMarketData events are "Last" (Which includes "Bid" and "Ask" events among them).

This means that you are blocking any development that can utilize Tick Replay in case the developer is using data other than "Last" in his indicator/strategy.

Why not do the easy thing which is fixing the problem (=False event types in OnMarketData)?

I am not bothering you because I like it. I am a developer that has to stop his Tick Replay based development because of false event types.

I tried bypassing the bug with getting the "BarsInProgress" number from the OnBarUpdate and by knowing the number I can know the real event type that follows in OnMarketData. But after you answered me in (1) that there are ticks that you miss in OnBarUpdate because of same ms timestamp it eliminates this bypass method.

If you are not going to fix the bug anyway please help me understand how to bypass it when I am using 3 tick based data series:

		        AddDataSeries(base.Instrument.FullName, Data.BarsPeriodType.Tick, 1, Data.MarketDataType.Last);
		        AddDataSeries(base.Instrument.FullName, Data.BarsPeriodType.Tick, 1, Data.MarketDataType.Ask);
			AddDataSeries(base.Instrument.FullName, Data.BarsPeriodType.Tick, 1, Data.MarketDataType.Bid);

[NinjaTrader_Matthew]

1) To clarify, NT8 is not "missing" any ticks, and Patrick's response is not related to the way a Tick BarsType is built (since you are a developer, see BarsType\@TickBarsType.cs OnDataPoint method to see how bars are added by attaching to a debugger).

We're simply speaking about the logic that separates calls to a particular NinjaScripts OnBarUpdate() when dealing with multiple series on historical data. OnBarUpdate will always call for the tick that generated that bar - however, the exact sequence may not occur as you expected when multiple ticks occur at the same millisecond. This is discussed in further detail here: http://ninjatrader.com/support/helpG...meFrameObjects

2) Tick Replay simply does not deal with Bid/Ask events at all, nor is there any way to get bid/ask volume historically through OnMarketData(). The bid/ask data you are seeing under that "Last" event is the bid/ask price that was calculated when the "last" trade event went off. You cannot get any bid-ask volume information historically through tick replay or OnMarketData.

If you need historical bid/ask volume, you can use OnBarUpdate() by adding a MarketDataType.Ask/Bid, but you are now restricted to the millisecond limitations discussed above if you connect to any of our current supported historical data vendors

I cannot provide you with any methods to bypass this as it's just not a reality. The only possible option is to obtain historical bid/ask/last data that use microsecond timestamps which can be imported from a.txt format - although I do not have any vendors that I can suggest which might be using that granularity.

[ToMer_K]

1) To clarify, NT8 is not "missing" any ticks, and Patrick's response is not related to the way a Tick BarsType is built (since you are a developer, see BarsType\@TickBarsType.cs OnDataPoint method to see how bars are added by attaching to a debugger).

We're simply speaking about the logic that separates calls to a particular NinjaScripts OnBarUpdate() when dealing with multiple series on historical data. OnBarUpdate will always call for the tick that generated that bar - however, the exact sequence may not occur as you expected when multiple ticks occur at the same millisecond. This is discussed in further detail here: http://ninjatrader.com/support/helpG...meFrameObjects

2) Tick Replay simply does not deal with Bid/Ask events at all, nor is there any way to get bid/ask volume historically through OnMarketData(). The bid/ask data you are seeing under that "Last" event is the bid/ask price that was calculated when the "last" trade event went off. You cannot get any bid-ask volume information historically through tick replay or OnMarketData.

If you need historical bid/ask volume, you can use OnBarUpdate() by adding a MarketDataType.Ask/Bid, but you are now restricted to the millisecond limitations discussed above if you connect to any of our current supported historical data vendors

I cannot provide you with any methods to bypass this as it's just not a reality. The only possible option is to obtain historical bid/ask/last data that use microsecond timestamps which can be imported from a.txt format - although I do not have any vendors that I can suggest which might be using that granularity.

Hi Matthew, thanks for helping.
(1) I understand what is written in the link. What I saw is not related to the order of events but to the fact that there are missing ticks in OnBarUpdate that did appear in OnMarketData.

I guess I was wrong. The thing that confused me was that I got each event twice in OnMarketData because I used 2 "Last" based data series. I think you can improve memory consumption as well as runtime if you only call the Tick Replay once if it is from the same instrument and the based on "Last" data. For instance, if I have use MSFT1 as 5 minutes and MSFT2 as 100 ticks and they are both "Last" data then there is no need to call OnMarketData twice for exactly the same event.

(2) When I use Bid/Ask data as the main/secondary series with Tick Replay I do get only "Last" Events but the volume in the events is equal to the bid and ask volume (e.volume == Bid/Ask volume). That is a big issue because I cannot differ between real "Last" volume to "fake" volume generated from the tick replay of the Bid/Ask series.
Try it and you will see the volume is not correct for a last event (you will get high volumes that represent the best bid/offer on the DOM).
These events are redundant because they are marked as type=last with wrong volume.

[NinjaTrader_Matthew]

Thanks for your additional comments. I am running them by our engineers and I will let you know the outcome of that discussion.

[NinjaTrader_Matthew]

(2) When I use Bid/Ask data as the main/secondary series with Tick Replay I do get only "Last" Events but the volume in the events is equal to the bid and ask volume (e.volume == Bid/Ask volume). That is a big issue because I cannot differ between real "Last" volume to "fake" volume generated from the tick replay of the Bid/Ask series.
Try it and you will see the volume is not correct for a last event (you will get high volumes that represent the best bid/offer on the DOM).
These events are redundant because they are marked as type=last with wrong volume.

Tick replay was only ever designed to work with Last market data types. There is evidently some confusion with the bid/ask market data series when that comes into play.

As a result of these scenarios, we are making changes in the next incremental release which will prevent tick replay from working on a NinjaScript which uses any series which is not built from last price. This change will be reflected in our upcoming release notes under #10189

[ToMer_K]

Thanks for the update.
I don't think this is the best solution because it will limit the Tick Replay development.
The best solution is just to ignore these data series and allow OnMarketData events for Last.

Another thing that I believe is redundant is the fact that if I use a Minute based series and a Tick based series that both derive from the same instrument then all the OnMarketData events show up twice. Any reason for that?

[NinjaTrader_Matthew]

Thanks for the update.
I don't think this is the best solution because it will limit the Tick Replay development.
The best solution is just to ignore these data series and allow OnMarketData events for Last.

That was discussed but is just not a possibility at this time.

Another thing that I believe is redundant is the fact that if I use a Minute based series and a Tick based series that both derive from the same instrument then all the OnMarketData events show up twice. Any reason for that?

OnMarketData is subscribed for all bars series added to the script for real-time data. NT7 was the same way and we have not resigned that behavior.

As TickReplay is designed to mimic real-time OnMarketData, it is the same for historical values as well. Additionally, there is no concept of 'mixing' series where one is replayed and others are not. It is either all series added or no series (this is also why we cannot prevent updates to the bid/ask series)

You can use BarsInProgress filtering to remove redundant updates to your code.

I understand your points, but we simply are not in a state where we can accommodate changes of this level to the data engine. I appreciate your thoughts and I've recorded your feedback under SFT-1531 : Enhanced support for tick replay and bid/ask series

[ToMer_K]

It is a good thing you are telling me that in realtime I will also get the same last event twice if I use two data series of the same instrument. I haven't checked this but it is very important to know that for future development that counts on realtime data per instrument and not for each of the bars. Also for the bid and ask events will act like that in realtime?

[NinjaTrader_PatrickH]

It is a good thing you are telling me that in realtime I will also get the same last event twice if I use two data series of the same instrument. I haven't checked this but it is very important to know that for future development that counts on realtime data per instrument and not for each of the bars. Also for the bid and ask events will act like that in realtime?

That is correct.