Market Replay

[NinjaTrader_Josh]

I was wondering where the recorded replay data was stored. Thanks, having difficulty finding this info.

[NinjaTrader_Dierk]

<my documents>/NinjaTrader6/db/data. However, you should not touch nor move/rename these files/directories.

[NinjaTrader_Josh]

Alright thanks. I have a few additional questions about Market Replay.

1) If I start recording data a few hours after market open, can I get historical data to fill in the few hours I missed for plotting purposes? (I think this might be my data feed's issue because opentick doesn't have historical data till the end of the day or something)

2) If I leave the recorder on 24/7 will it know a new day has started and create a new fiile accordingly? Since the market opens at 6:30AM my time I don't wake up in time a lot of the times, but would like to record the openings. If I had my computer running overnight in preparation of recording the next morning's opening, will I end up having to fastforward through 8 hours of blank tape? Alternatively, will it know if the market is open/closed and start/stop recording accordingly?

3) Can I start a recording, run for a few hours, stop, start a new recording a few hours later and stop? How will Market Replay deal with the gap? Will it create two separate files corresponding to each recording?

3) Is there any other way to stop a recording besides disconnecting from my datafeed?

4) Am I suppose to uncheck "Run market replay recorder" before I watch my replays?

5) When I go to the "Available Instruments" from the playback controls window what is L2 Begin/End and how do I use it?

Heh thats actually a lot of additional questions. Thanks for all the help.

[NinjaTrader_Dierk]

1) no
2) no, your need to restart every day
3) yes, gap will not be filled. One file per day
3) no
4) no
5) L2 is recorded as well in case you turn on a level 2 steram (e.g. via DOM)

Also: please consult the docs on how to use market replay

[NinjaTrader_Josh]

Would modifying indicators on a chart "mess up" a recording by causing a reloading of historical data?

[NinjaTrader_Dierk]

No it would not.

[grd974]

Hello,

I have recorded the 20 previous ES trading days and when I have finished playing June 20 and want to go to June 19, is not there any other way to do so than : Control Center > File > Disconnect > Market Replay Connection then
File > Connect > Market Replay Connection > 06/19/07 ?

Thanks for any reply,
Gérard

[NinjaTrader_Dierk]

No, the file is available with the date as you started recording. That's why you should restart recording once a day.

[grd974]

I mean I have 20 different files under the path NinjaTrader 6\db\data\ starting from 00070529 to 20070623 because I actually start recording a new file everyday.
So, I would like to be able to switch, for instance, from June 23 to Mai 29 without having to disconnect from Market Replay Connection then to Connect again to Market Replay Connection.
Is not there a less tedious way to play back those 20 files each containing one trading day ?

[NinjaTrader_Josh]

I would also like to suggest that Market Replay automatically know it is replaying so we don't have to manually uncheck "Run market replay recorder" every time.

[NinjaTrader_Dierk]

I mean I have 20 different files under the path NinjaTrader 6\db\data\ starting from 00070529 to 20070623 because I actually start recording a new file everyday.
So, I would like to be able to switch, for instance, from June 23 to Mai 29 without having to disconnect from Market Replay Connection then to Connect again to Market Replay Connection.
Is not there a less tedious way to play back those 20 files each containing one trading day ?
Not possible.

[NinjaTrader_Dierk]

I would also like to suggest that Market Replay automatically know it is replaying so we don't have to manually uncheck "Run market replay recorder" every time.
Not surr I follow. You don't have to uncheck "run market replay recorder" as you are replaying.

[NinjaTrader_Josh]

Oh really now. You replied yes to my #4 question up above so I assumed that was a necessary step, but I guess not. Sweet.

[NinjaTrader_Dierk]

My apologies. I edited my reply below.

[NinjaTrader_Josh]

Is there anyway to crop out the blank tape I get if I turn on the recorder at midnight and let it run til market close? I'm getting about 5 or so hours of no ticks till the pre market activity starts getting recorded. It would be nice if I could some how chop those out so I don't have to use the seekbar everytime I open the replay.

[NinjaTrader_Ray]

Unfortunately this is not possible.

[daven]

Is there any thought or possibility of establishing a central data source of market replay data available to developers as an additional testing data source? Because of some problems with a third-party custom indicator I needed to uninstall and re-install my NinjaTrader installation. As per the instructions, I deleted all related subdirectories after the uninstall process. Unfortunately I forgot to save my recorded data, so now I have nothing to run tests on until I record more data. My bad, but it is a pain in the a$$ to have to wait for all that data to be recorded. If you just had a download site for say a few weeks of data on a few instruments, (may I suggest the current e-mini Russell contract to start), I suspect it would be applauded by many novice developers such as me.

[daven]

In the last post you noted not to touch or move the market replay data. It appears that it is possible to copy one or more replay files and place them on another computer with NT. Is there a problem doing this? Are there cautions or things that I should be aware of when doing it?

<my documents>/NinjaTrader6/db/data. However, you should not touch nor move/rename these files/directories.

[NinjaTrader_Ray]

It is fine to move copy them to another computer.

Regarding the other suggestion, we will be working on a data back up utility that should provide some relief from accidentally not saving critical data. I will add your suggestion to our list.

[NinjaTrader_Josh]

Is it just me or does the VOL indicator show messed up numbers in replay mode? From all the replays I have recorded whenever I play them it will say the volume for AAPL is in the hundreds of millions within the first 15 minutes. When I look at the volume row in my T&S window it only says 3mil. Whatever is causing this makes it literally impossible to gauge if there are spikes in the volume from previous days because the 100+ mil dwarfs all previous volumes to being about 1 pixel tall.

[NinjaTrader_Dierk]

Please verify, that observed behavior is reproducible with a simple chart with just VOL on it.

If reproducible then please send me via PM (dierk AT ninjatrader DOT com):
- screenshot of your chart (including window caption)
- your replay files for that day (located at <my documents>/NinjaTrader6/db/data)

Thanks

[NinjaTrader_Josh]

I have sent you an email with all the related files.

[NinjaTrader_Josh]

Thank you Dierk for helping pinpoint the issue with the volume. I will contact OpenTick and see whats up on their end.

Another problem I am having with replays is this though. I have a Market Analyzer loaded with the list of instruments I want to record. Every day at least 1 or 2 of those instruments just doesn't want to record. They end up as 1kb files compared to the 100+kb files of my other instruments. When I play back those 1kb sized instruments there is no data on the file so nothing shows up. Before I thought it was just a datafeed error, but now I can confirm that I am getting real-time data for those unrecorded instruments. For instance, today it doesn't want to record SHLD yet I can see the last/bid/ask price constantly changing in the Market Analyzer. The instrument that doesn't record is seemingly random. Yesterday BIDU didn't record for me, but it is today. I'm not quite sure what is going on here. Any ideas?

[NinjaTrader_Dierk]

Do you have any funny messages in the logs?

[NinjaTrader_Josh]

Ahh yes. The logs. Why didn't I think of those?

It would appear there are funny messages.

MarketData (Ask) for 'SHLD' at 6/29/2007 5:00:00 AM exceeds max price gap (192.25/471)
MarketData (Ask) for 'VSEA' at 6/29/2007 4:30:01 AM exceeds max price gap (2000/44)Those are the messages showing on my log menu after this error

Market data error for instrument 'VSEA Default': No data (1003)I don't think this error is a problem because it is showing for all of my instruments and it corresponds with the time I turn on NT to start recording which is hours before market opens.

The first error I posted comes after the "No data" error and I only get two of them, one for each of the instruments not recording. Any idea what the deal with those are? A bad spike in the data I presume? NinjaTrader can't just pass over the spike and continue recording normally?

[NinjaTrader_Dierk]

The replay recording is canceled as you see these message in the logs. I'll add your change request to the list.

[NinjaTrader_Josh]

Sounds good. Thanks.

[NinjaTrader_Dierk]

On the VOL issue. I got feedback from OpenTick support: we need to change NT since right now volume for realtime data on stocks is multiplied by 100. This fix will be available with next update. Sorry for the confusion.

[NinjaTrader_Josh]

Any chance for a custom build addressing this? It would suck to continue recording erroneous replay files for another month or more. As always thanks for the quick response in handling my concerns.

[NinjaTrader_Dierk]

It will be incorporated in next custom build. No ETA yet.

[NinjaTrader_Josh]

It would be nice if my charts could revert back to the last time series I had on my charts before watching a replay. It is tedious work to constantly change the chart settings to reflect the current day every time I switch between replays and live.

[NinjaTrader_Ray]

This has been resolved for 6.5 due out in fall.

[NinjaTrader_Josh]

I think it would be beneficial to have NinjaTrader automatically know not to download any historical data when it is recording. On numerous occasions I have lost connection to my datafeed and when it automatically reconnects it wants to download historical data and that destroys the recording. Breaks in the datafeeds should just produce gaps in the charts instead of having a whole day's worth of recording wasted.

[NinjaTrader_Dierk]

Downloading historical data for charts will not affect the data recorded for market replay, since this is 2 different pools.

[NinjaTrader_Josh]

Hmm. If losing connection isn't what is breaking my replays than I'm not sure what is. I start up my NinjaTrader to record data around 12AM or 1AM and just leave it running till market close. My "good" files all end up having start times around 4AM (premarket open PST) and go till 1PM (market close PST). My "bad" replays all seem to start at 5PM the previous day and end at seemingly random times. When I scroll through the replay all the data is jumbled up and I can't make sense of it. Any ideas on why these replays are turning sour? My success rate on replays is about 33% right now.:confused:

[NinjaTrader_Dierk]

Recording replay data stops are there are bad ticks coming in which are reported in the logs.

[NinjaTrader_Josh]

Would these be considered bad ticks? Even if they stopped the recording for the few select stocks in question it doesn't explain why all recorded instruments are messed up.

7/11/2007 1:50:23 AM|1|2|OpenTick: Primary connection=Connecting, Price feed=Connecting
7/11/2007 1:50:26 AM|1|2|OpenTick: Primary connection=Connected, Price feed=Connected
7/11/2007 1:50:27 AM|3|4|Market data error for instrument 'AAPL Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument '^COMP Nasdaq': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'AKAM Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'BIDU Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'BIIB Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'CROX Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'CTSH Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'DRIV Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'ESRX Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'FMCN Nasdaq': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'GOOG Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'JOYG Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'KLAC Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'LRCX Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'NIHD Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'NTRI Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'PCAR Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'QCOM Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'QQQQ Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'RIMM Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'SHLD Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'SMH Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'UAUA Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'VSEA Default': No data (1003)
7/11/2007 1:50:29 AM|3|4|Market data error for instrument 'WYNN Default': No data (1003)
7/11/2007 4:01:03 AM|3|4|MarketData (Bid) for 'GOOG' at 7/11/2007 4:01:01 AM exceeds max price gap (529.9/55.55)
7/11/2007 4:31:33 AM|3|4|MarketData (Ask) for 'ESRX' at 7/11/2007 4:31:31 AM exceeds max price gap (54.93/2000)
7/11/2007 4:31:33 AM|3|4|MarketData (Ask) for 'ESRX' at 7/11/2007 4:31:31 AM exceeds max price gap (54.93/2000)
7/11/2007 9:17:45 AM|1|4|Verifying license at primary server...
7/11/2007 9:17:46 AM|2|4|Session Break (Version 6.0.1000.3)
7/11/2007 9:17:46 AM|1|4|Current license: Type=SimulationOnly State=Verified StartDate=12/31/2004 EndDate=11/30/2099 Collective2 Dtn ESignal External OpenTick Replay Simulator Yahoo Gain FFastFill Hosted InteractiveBrokers MBTrading Patsystems TrackData TradingTechnologies AdvancedStrategyManagement AutomatedTrading BasicEntry Charting SuperDom
7/11/2007 9:18:08 AM|1|4|Automated trading disabled
7/11/2007 9:18:12 AM|1|2|OpenTick: Primary connection=Connecting, Price feed=Connecting
7/11/2007 9:18:14 AM|1|2|OpenTick: Primary connection=Connected, Price feed=Connected
7/11/2007 9:18:19 AM|3|4|MarketData (Ask) for 'ESRX' at 7/11/2007 9:18:18 AM exceeds max price gap (52.51/2000)
7/11/2007 9:18:25 AM|3|4|MarketData (Bid) for 'GOOG' at 7/11/2007 9:18:22 AM exceeds max price gap (542.61/55.55)
7/11/2007 9:18:25 AM|3|4|MarketData (Bid) for 'GOOG' at 7/11/2007 9:18:22 AM exceeds max price gap (542.61/55.55)
7/11/2007 9:18:25 AM|3|4|MarketData (Bid) for 'GOOG' at 7/11/2007 9:18:22 AM exceeds max price gap (542.61/55.55)Notice how those events happen after a disconnect and the subsequent reconnect though.

I would also like to note that the image in my previous post is suppose to be a market replay of 7/11 not 7/10 as stated in the date. The recording was started as stated in the logs here at 7/11 1:50AM.

[NinjaTrader_Dierk]

Right.

These bad ticks ultimately will stop the recording for the affected instrument:
7/11/2007 9:18:25 AM|3|4|MarketData (Bid) for 'GOOG' at 7/11/2007 9:18:22 AM exceeds max price gap (542.61/55.55)

[NinjaTrader_Josh]

Yes, but what I am having problems with is what is causing my 7/11 replay data to want to tell me the start time on ALL my instruments are 7/10 5PM instead of the usual 7/11 4AM.

[NinjaTrader_Dierk]

Not sure why there is confusion:
- apparently you connected to your provider at 5PM (7/10?) and left it running
- you experienced temporary losses of connection, but you did not (!) manually disconnect from your provider -> replay recording keeps going
- after the connection comes back, there were some bad ticks which halted the recording for the affected instruments

[NinjaTrader_Josh]

Ok I think we are misunderstanding each other. Here are the steps I take each day.

1AM turn on NinjaTrader -> connect
4AM Pre-market data starts getting recorded
6:30AM Market opens; recording continues
1PM Market closes
2PM turn off NinjaTrader
rest of day I turn NinjaTrader on and off to watch replays/develop etc
repeat for every new day

Basically I start up my recording session for the new day at 1-2AM on the new day and not 5PM of the previous day. It should also be noted that all my messed up Market Replay data decided to have a start time of exactly 5:00:00PM of the previous day. Too precious for it to be my manual doing.

[NinjaTrader_Dierk]

Sorry, I don't know what would cause that. The recording starts as soon as you connect (1AM) and not 4AM (provided you have the recording option set).
I suggest monitoring and see if you can detect a pattern.

Also: make sure you don't change timezone settings on your PC back and forth. I would suspect this could throw off stuff.

[NinjaTrader_Josh]

From my experience even though I connect at 1AM there are no ticks till 4AM and the instrument start time comes in at the first tick so thats why I say it "starts" at 4AM.

Do you think there are any activities I could possibly be doing in NinjaTrader during my "rest of day I turn NinjaTrader on and off to watch replays/develop etc" that could be damaging the replay data? I was thinking maybe watching replays was throwing it off, but it was already clarified earlier that I did not have to uncheck the "Run market replay recorder" button before watching replays.

I don't change my PCs timezones ever so that shouldn't be an issue. Hmm. I will try my best to pinpoint an exact activity for us as this seems to be a bug.

[NinjaTrader_Dierk]

Sorry, I'm out of ideas.

[NinjaTrader_Josh]

Ok Dierk. I believe I have found out the source of this problem.

I did as I always do. Start recording at 1AM. A few hours after market close I closed my data connection to stop the recording. I went into Market Replay Connection to test if my data recorded properly and it did (note: no disconnects from datafeed today). 4AM start times and end times are set at whenever the latest tick for the instrument was in after-market hours. So far so good so I saved out my replay data folder to preserve the good data. Then I proceeded into reconnecting to my datafeed. My "Run market replay recorder" is still checked and also be aware that after-market is still in operation right now. My charts all wanted to reload historical data. It might be worth noting that all the data for the current day now no longer shows volume in the millions as it does from the real-time volume bug we sorted out earlier. The volume seems accurate now after the charts reloaded. I now disconnected from my datafeed without doing anything else and reopened the Market Replay Connection. Lo' and behold the replay now says 7/11/2007 5:00:00PM start time for all my instruments instead of the prior 7/12/2007 4:00:00AM.

Only thing I can think of is loading historical data is not as independent of market recording as we previously thought. Perhaps it has something to do with replacing the erroneous data volume that NinjaTrader remembers. Could it also have something to do with reopening the recording feed during after-market hours and the subsequent recording of ticks during historical data loading that is messing things up?

[NinjaTrader_Dierk]

Please PM me for a follow up at "dierk AT ninjatrader DOT com"

[NinjaTrader_Dierk]

This is a bug which will be fixed with next update. Thanks for reporting this.

[NinjaTrader_Josh]

Great Dierk. I have not had problems with replays since installing the custom build. Everything seems to be in order. Thanks.

[NinjaTrader_Dierk]

Thanks for that feedback.

[NinjaTrader_Josh]

I was wondering about how Market Replay increases replay speeds. Does it skip frames and just use a keyframe to draw charts from or does it still fill in all the ticks inbetween? I am wondering this because I want to know if I can forward test my strategy at 50x to get quick results.

[NinjaTrader_Dierk]

Replay always replays all ticks available -> ticks are never skipped.

How ever you might experience different behavior, due to race conditions on when actual (simulated) fills might come in.

[NinjaTrader_Josh]

Generally speaking I will only be affected by these different race conditions if my computer doesn't have enough raw computing power right?

[NinjaTrader_Dierk]

Nope, sim fills trigger in after a configurable delay (in milliseconds). This is not synced with the replay speed.

[NinjaTrader_Josh]

Ahh. May I suggest this to be synced then in the next release. You can't really test strategies at anything but 1x without fills matching the replay speed.

[NinjaTrader_Dierk]

We'll add it to the list.

[NinjaTrader_Josh]

I did a simple test with Bars.PercentComplete in Market Replay and this is the behavior I am experiencing.
At 1x speed I can get this code to output perfectly. It spits out the percent of the bar from 99% till the bar becomes 100%.

if(Bars.PercentComplete*100 >= 99)
{
Print("Time: "+Time[0]+" "+Bars.PercentComplete*100);
}

The problem is when Market Replay is sped up to 50x. It will stop printing these outputs and act like it completely skipped them. I suspect because of the way Market Replay increases speed (it seems to jump on steady intervals of 50secs at 50x) it causes Bars.PercentComplete to probably go from 90% straight to 100% thus never satisfying the inbetween condition at 99%. Any workaround ideas for this?

[NinjaTrader_Dierk]

>> it seems to jump on steady intervals of 50secs at 50x
That is correct.

>> Any workaround ideas for this?
Sorry, none of the top of my head.

[NinjaTrader_Josh]

In my replays I think the Bars.FirstBarOfSession is not working properly. I have my chart set to session begin at 6:30AM. I confirmed that the Bars was getting the proper session begin time via Print(Bars.SessionBegin);. When I do a simple print on Bars.FirstBarOfSession though it returns false even if it is the first bar of the session.

I did Print(Now+" "+Bars.FirstBarOfSession); in OnBarUpdate() and this is what it returns to me "7/26/2007 6:30:02 AM False". 6:30:02AM is definitely in the first bar and yet the bool is still returning false.

[NinjaTrader_Dierk]

Is this reproducible? If so, then please PM your replay file and a simple sample strategy. Thanks

[NinjaTrader_Dierk]

This is a bug which will be fixed with next release. Thanks for bringing this up and for proving the reproducible scenario.

[fragalles]

This is a bug which will be fixed with next release. Thanks for bringing this up and for proving the reproducible scenario.

Is there no way to implement the feature that the replay feature can be used for any historical intraday data (which can be optained for example from opentick)?
It would be simply amazing to actually trade 1 day that I have missed in the past, for example the 31st of dezember last year...

Another perfect example is LEND last friday (10th august 2007) 45,27%+ but where was the entry point with the lowest risk, would I have been disciplined enough to set my trailing stop to the right values?

All this I could test with the intraday replay but for a stock for which I already know how it moved, its almost no sense (its like watching a soccer match where I already know when and how many goal have been scored...)

If someone has some replay data of some of the last days biggest gainers stocks I would be more than thankful if you could send me a zip file of NinjaTrader 6\db folder.

[NinjaTrader_Josh]

If you have the data down to every single tick technically you should be able to construct a replay from the timestamps of these ticks. I don't know how feasible it would be for NinjaTrader to implement such a thing, but it would be pretty amazing not needing to have your computer sit there recording every day.

[NinjaTrader_Dierk]

>> Is there no way to implement the feature that the replay feature can be used for any historical intraday data (which can be optained for example from opentick)?
Thanks for your suggestion. We'll add it to the list

[RRMessiah]

Hi Everyone, I'm new to NT. I've been looking for a free simulator for awhile now and NT seems to fit the bill, but I lack any stock replay data. I noticed everyone archiving futures data but unfortunately I don't trade futures. Can someone post some stock data to http://ninja.korthrun.net/ ? Thank you!

[NinjaTrader_Ben]

Hello,

NinjaTrader does not supply or distribute data, but maybe a fellow trader will post some to share with you.

[paulchow2k]

Sure but I've only got australian stocks although a few of them are listed in UK & US exchange. Stocks such as BHP in London stock exchange & Newmount Mining in NYSE Exchange.

regards

Paul

Hi Everyone, I'm new to NT. I've been looking for a free simulator for awhile now and NT seems to fit the bill, but I lack any stock replay data. I noticed everyone archiving futures data but unfortunately I don't trade futures. Can someone post some stock data to http://ninja.korthrun.net/ ? Thank you!

[NinjaTrader_Bertrand]

Thanks for your help Paul.

[paulchow2k]

Thanks for your help Paul.

No worries Bertrand. 1 question, do you know if the next version of NT comes with the ability to change the Tick size for securities? It's currently defaulting to 0.01 for equities but we can't change it.

Is NT7 still on track to be released after winter?

Paul

[NinjaTrader_Bertrand]

Hi Paul - yes this will be possible in NinjaTrader 7, it's expected arrival will be Q2 2009.