duplicate trades

[greentrader]

Using 6.5.1000.7 and IQFeed, I've found, by comparing trades seen by OnMarketData callback with historical tick data from IQFeed, that the OnMarketData routine is being sent duplicate trades (about 1 duplicate in every 50 to 100 trades). These duplicates always happen immediately after the real trade, and occassionally have a different bid/ask associated with them (since the bid/ask may have drifted). This occurs with all of the index futures I've tested, and continues to occur if I disconnect and reconnect the IQFeed (so unlikely a problem with a particular server).

I notice that a bug like this has been reported in the past, but am not sure what its current status is. This is, obviously, a huge problem for anyone like myself who is tracking tick data. Anyone know anything more about this? Thanks.

[NinjaTrader_RJ]

Hello,

Thank you for your note.

I suggest to upgrade to the current version of NinjaTrader and test the issue:

Please download from this link: http://www.ninjatrader.com/webnew/do...g_software.htm

[greentrader]

Thanks. Have upgraded to .9 version and will look at data again for rest of day and overnight. (The problem wasn't mentioned in the update notes, so I avoided doing an upgrade to my otherwise stable system.) Will report back here when I've got enough data collected...

[greentrader]

Got an exact match with historical tick data from IQFeed between 3:30 and 3:40 PM ... will continue monitoring.

Were there some changes to related OnMarketData code that could account for this fix (so I can justify thinking the problem is truly gone)? Thanks.

[NinjaTrader_RJ]

Not that I know of, so please let me know your results.

Thank you!

[greentrader]

One duplicate trade found at 4:15:00.

May be that problem has always been there, but that its magnitude varies with some not-yet-defined set of circumstances. Will keep looking...

[greentrader]

The single duplicate trade noted in previous post might also simply be a tick error. As explained on the DTN/IQFeed forum...

You shouldn't get any duplicate trades. We will be making this more robust/easier to work with in the future as well as we will be adding the Tickid to the stream feed at some point in the future.

The historical tick data will likely not match the realtime feed as we do process exchange tick corrections. Thus, a bad tick the exchange sends through the feed may have been removed/modified in the historical data. This can happen "on the fly".

So IQFeed shouldn't be giving us duplicate trades, but is not immune to tick errors that are only (currently) corrected in their historical tick data.

[greentrader]

In overnight trading, I found:

2 ES duplicate trades at 7:32:41 PM, 3:47:50 AM

6 NQ duplicate trades at 7:29:25 PM, 1:49:05 AM, 2:04:22, 3:47:57, 4:38:49, 4:39:39

The proximity of the 2 ES trades with the corresponding 2 NQ trades is interesting, suggesting a common cause for these duplicates.

Well, this looks like an NT bug to me. With thousands of intervening trades showing no duplicates, its hard to imagine what I could be doing in my own code to cause the duplicates.

(BTW, in 6/8 of the above duplicates, the bid or ask had changed, implying that I received an intervening bid or ask price change message. Although in yesterday's trading, the fraction with bid/ask changes was smaller.)

(Another thing: The computer was unattended during this time, devoted only to capturing trade data. I guess another possibility is some sort of internet communication glitch, although the duplicates yesterday occurred during RTH with no internet disconnects.)

[NinjaTrader_RJ]

Hi Greentrader,

Can you provide me with the exact steps you took to reproduce this?

We can then test on our end.

Thank you!

[greentrader]

Originally Posted by NinjaTrader_RJStein

Hi Greentrader,

Can you provide me with the exact steps you took to reproduce this?

We can then test on our end.

Thank you!

Okay, I'll send you a PM with the details.

This is probably a case where you guys should develop your own test code so that you can keep an eye on this stuff. I'm just trying to get past this problem, and hope to not have to monitor it in the future, so my current code is not optimized for detecting such issues, but I can outline for you the steps needed to develop such code.

[greentrader]

I guess I'm going to put this down for awhile and just deal with it by updating my data files with historical data a bit more often.

For NQ today, there was one more duplicate at 5:44:48 AM ET, and then no others through 2:30 PM (last time I checked). Some observations:

- some days I see many dups, others hardly any
- rate of dups seems more connected to elapsed time than trade rate
- dups happen both within and outside of RTH
- not fixed by version update, restarting NT, or restarting strategies
- dups not dependent on user activity, or other running apps
- IQFeed claims they're not the culprit, but still possible (I guess I can check that by having SierraChart save tick data along with NT, and then seeing if they see the same dups - will let you know what I find)

[NinjaTrader_RJ]

You should not compare to other platforms.

What are your results when you use the export tick data feature for comparison?

[greentrader]

Originally Posted by NinjaTrader_RJStein

What are your results when you use the export tick data feature for comparison?

Good question! I had put that out of my mind since it didn't have the bid/ask info I needed...

Answer: The duplicate trades do not show up in your tick export, so the problem appears to be within NT's OnMarketData messaging.

Well that should be easy for you guys to see by just watching for when your messages get out of sync with your trade count. If you can't reproduce it, then I can try putting together a simpler strategy that illustrates the problem. Thanks again for all the feedback on this.

---------
Edit: The above assumes that the NT tick export is based on recorded realtime data, but if it's just being obtained from historical IQFeed, then we haven't learned anything new.

[NinjaTrader_RJ]

Hi,

If you wouldn't mind creating that strategy and posting it as well as what instrument and time frame, that would help a lot in troubleshooting.

Thank you!

[greentrader]

Originally Posted by NinjaTrader_RJStein

If you wouldn't mind creating that strategy and posting it as well as what instrument and time frame, that would help a lot in troubleshooting.

I'm going to first try comparing the DailyVolume message results with the trade message results to see if the two get out of sync due to duplicate trades. That will be easier to demonstrate in a simple strategy than a lot of file handling code. (I'm assuming that the DailyVolume is from the exchange, and not just a redundant sum of what NT thinks are realtime trades.)
--------
Edit: Not seeing any DailyVolumes reported, so will have to try something else.