observation on bad tick data/market replay/sim

[mjc4118]

The attached video contains a Market Replay Data issue that I would like to understand so I can determine how market replay and "reality" relate. It appears to me that there is a significant amount of errant tick date in market replay. As well as a significant comment/suggestion related to Simulator order fills ..that you can add to your ever growing list of suggestions/comments and observations.

A picture is worth a1000 words so I have included a video link below of a sample of some "odd" market replay fills that I have received and compared the charts on the video to the historical tick data. Your comments and insight would be appreciated.

http://www.emini-fools.com/images/observation_replay_data_sim_fills.wmv

As always the codec is at http://www.gotomeeting.com/codec.

And if I am driving you boys crazy you can always try to do what my wife does and tell me to go away ..it usually does not work for her either!

Thanks for the help.
Mark

[NinjaTrader_Austin]

Mark, thank you for your note. We will take a look at this starting on Monday.

[NinjaTrader_JoshP]

Mark,

Thank you for the video.

About the data you have in your Historical Data Manager. Is this data you have collected while having a real-time tick filter enabled? That could be the cause of the discrepancy as the replay data we provide does not use it.

For the fills, the fill algorithm is outlined here: http://www.ninjatrader-support.com/H...ulatorOverview

Unfortunately in terms of realism or not, there is no way we can simulate reality exactly in all situations, but we have found that this approach is much more accurate in providing an idea of possible results than just always arbitrarily choosing a price to fill you at, whether that price is optimistic or pessimistic.

[mjc4118]

The data is downloaded from market replay.
So it is not "saved" data from live sessions

I understand the logic in simulator. And I strongly suggest that you execute live trades on one computer and sim trades on another for 3 or 4 trading sessions and compare the real time results. Unless you do this you cannot compare the two. . . as I have been doing over the last 1 year. While I appreciate the complexity of the simulator. the fills should be realistic as I am sure you will agree.. And in many cases they are. But in thin markets you often skid in an unrealistic optimistic manner that simply does not occur in the actual market. I am not attemting to be negative in any way about the sim functionality. I am only trying to provide you information from someone who has traded strategies on live accounts and simulated data side by side and seen and compared the results.

Put one strategy on a chart and trade on a live account and put the same strategy on the same chart and point it to a simulator account and you will see the difference very quickly.

When trading the live market w/ Stop Limit orders in the CL I get my price fills over 93% of the time and 99.8% of the slippage is AGAINST my profit margin and usually only occurs on my stops simply because of where I place my orders and the pressure in the market.

When trading on Sim I get slippage over 20% of the time (often on my entry..that RARELY happens in real life) and in over 60% if the slippage is OPTIMISTIC and adds to my bottom line. When you compare the two there is a significant discrepancy and blows back testing all to crap.

[NinjaTrader_JoshP]

Mark,

Not sure what you mean by data being downloaded from Market Replay.
To get terms clear, there is a Market Replay data file and historical chart data. Historical chart data always comes from your data provider (Market Replay connection is not a data provider). Downloading Market Replay data files comes from NT recorded servers which would not necessarily match data from your data provider.

Just as you have seen situations where it would slide optimistically, there are also situations where it would slide pessimistically. The net result is that you would generally have more realistic fills than you would with just always choosing one or the other. Now this is not to say that it is a 1 to 1 for optimistic vs pessimistic. It really is dependent on market conditions and what the simulator thinks is a possible fill. Obviously what it thinks is a possible fill would not necessarily correspond with how the market actually works. It may simulate some markets better than others, but as such that is just a generic reality of the matter. Replays are simulations and simulations are never the same as reality and results should always be taken with a grain of salt.

[mjc4118]

Josh.

I think I was pretty clear in my post... download market replay ...as in File | Utilities | Download Replay Data.... I also clearly stated this data is NOT recorded from live sessions.

I will reply to your comments on the simulator tomorrow.

Thanks!
Mark

[NinjaTrader_JoshP]

Mark,

I am not talking about the data you download for Market Replay, I am talking about historical chart data. These are completely separate things as mentioned in my prior posts.

Historical chart data does not equal Market Replay data. Historical chart data comes from your data provider. Market Replay data comes from NT recorded servers. Different sources = discrepancies can arise.

[mjc4118]

Josh.

I am sure you do not mean to, . . but I feel like you are trying to make this difficult. Not all of us are unintelligent. Obviously they are different sources. The point remains that your "Replay Data" downloaded from "your servers" should match what actually happened in the market and should not contain "bad ticks". And The charts from the "replay data" show errant/bad ticks. I do not think that is correct in any case. If you look at the data on the chart created from "your replay data" downloaded from "your servers" they do not match the historical data for that instrument on that day according to CME group. Just thought you may want to know. Thus the reason for this post. The historical data you make available should be accurate.

And as far as the simulations and your suggestion that they be taken with a grain of salt... It pretty much defeats the entire purpose of back testing and simulation if you get optimistic results. I do not think you really want to update your NinjaTrader marketing materials to read:

NinjaTrader is a FREE application when used for advanced charting, market analytics, system development and trade simulation who's simulator results should always be taken with a grain of salt.

If you did not provide optimistic fills you would find that the results would generally be more reliable and could be taken VERY seriously. Any actual live money trader would agree. I even suggest you poll the paid/subscribing users and ask them in a fair and balanced poll I will gladly say I am wrong if they vote to allow for "optimistic fills" that can artificially inflate your P&L.

[NinjaTrader_JoshP]

Mark,

Ideally that would be the case, but unfortunately this is simply not so. Please note that the Market Replay download is a courtesy service provided for free. It is recorded in real-time with all ticks coming in on the recorder. If there is a bad tick, goes in as well. Should you want higher quality files I suggest looking into paid for services where it is their business to provide perfect data.

Backtesting, simulation, etc. should always be taken with a grain of salt no matter from NT or any other software or place you do your testing on. This is a fact of the matter for all testing and should always be adhered to any time you do anything in simulation anywhere. The reality of simulation is simply that it is never the same as real-life.

The fills algorithm is done by bid/ask volume and some simulating of queue positioning based on when you placed the order. If there is enough volume at the bid/ask prices you can get filled. NT will never give you a fill in simulator that would be impossible in real-life. If the volume is there to cover the position size you are asking for then it means it is possible to be filled at that price. Whether or not a real-life market would truly fill you at that price is something that cannot be accurately simulated for all markets with a catch-all algorithm. Ideally, if we were able to create an algorithm that could accurately simulate the markets filling of orders we could sell it to big banks and trading firms instead and probably make a much prettier penny doing that then our current business. Unfortunately that is simply not the case. The bottom line is that NT never inflates you by giving you impossible fills if the underlying volume is not present.

[mjc4118]

Josh

Thanks for the reply Josh. While I agree with your analysis that it does not inflate you if the volume is not present that is not the same thing as saying it does not inflate you. If you take my advice about eliminating or providing an option in the future for users to opt to to limit or prevent optimistic fills NT would find themselves in a position a lot closer to having the really awesome simulator you mention.

I have been involved in development of various mainframe, windows and web software applications for the last 20 years and have a unique perspective of understanding the design, development, and implementation cycle. Often I have seen really great software products fail to achieve their full potential because the designers are not necessarily aware of the practicalities and limitations imposed by their design while their design is perfect in theory. This is simply a case where you have the option of making the simulator a better tool or not. The choice ultimately lies with NT. Considering the few lines of code something like this would require it seems like a shame to at least not consider it. It would increase the effectiveness of the product as a whole because live trading would or should always provide better results than the simulator. So that if you were profitable on the simulator your probabilities for being profitable in the market would be greater. Now that is really worth paying extra for and it is what a good tool or software package should do. Push the user and help the user improve their trading style, techniques and strategies. No tool should return mediocre results that provide additional allowances that leave room for or induce failure when they could be better.

Personally I never liked mediocrity.
Thanks for your insight.
Mark

[NinjaTrader_JoshP]

Mark,

Thank you for your post and comments on this subject. I have added your suggestion to our feedback list for future considerations.

[mjc4118]

Thank You Josh.
That is all I ask

Mark