BidAsk Historical...

[funk101]

Does NT7 remember bid ask historical data now?

[NinjaTrader_Dierk]

If by "remember" you mean that bid/ask data would be stored as you record historical chart data, then yes.

[funk101]

actually, what I meant was, realtime bid and ask volume on some indicators gets wiped when you refresh the chart, for instance, the volume profile, or VolumeUpDown. Is this still the case?

[NinjaTrader_Dierk]

Yes, VolumeProfile and VolumeUpDown start from scratch as you throw them on chart.

[gomifromparis]

Hi,

Is there a way to access historical bid/ask data ? It doesn't seem possible in either indicators or bartypes ??

[NinjaTrader_Dierk]

Sure, there e.g. is signature below which you should see in intellisense:

void Add(string instrumentName, Data.PeriodType periodType, int period, Data.MarketDataType marketDataType)

[gomifromparis]

Ok that works, but then there is no way to link the bid/ask values to the price values. They are just other charts. So in that case I suppose there would be no way to compute the historical delta volume....

[NinjaTrader_Dierk]

They would be "independent" series.

[darthtrader]

If by "remember" you mean that bid/ask data would be stored as you record historical chart data, then yes.

Dirk, if bid/ask data is stored as chart data is recorded..
does this mean that VolumeProfile is wiped clean because of the way it was written and not upgraded for 7 or that we simply can not backfill stuff made using onmarketdata() in 7 like we could not in 6.5?

[NinjaTrader_Dierk]

>> that we simply can not backfill stuff made using onmarketdata() in 7 like we could not in 6.5
Your understanding is correct.

[gomifromparis]

I tried to reconstruct historical data using on chart by adding tick data bars.

protected override void Initialize()
{
Add(new Plot(Color.FromKnownColor(KnownColor.Orange), PlotStyle.Line, "P1"));
Add(new Plot(Color.FromKnownColor(KnownColor.Red), PlotStyle.Line, "P2"));
Add(new Plot(Color.FromKnownColor(KnownColor.Orange), PlotStyle.Line, "P3"));

Add(Instrument.FullName,PeriodType.Tick,1,MarketDa taType.Last);
Add(Instrument.FullName,PeriodType.Tick,1,MarketDa taType.Ask);
Add(Instrument.FullName,PeriodType.Tick,1,MarketDa taType.Bid);


}

protected override void OnBarUpdate()
{

P1.Set(Closes[0][0]);
P2.Set(Closes[1][0]);
P3.Set(Closes[2][0]);
}
But the fact is that it uses a lot of time and memory, because the bid/ask bar array is very large, even for a 5 days history. So I don't think it will be easily usable , except maybe for intraday memory.

[funk101]

What about writing/reading from a filestream? Would that be a viable solution?

[gomifromparis]

What would be cool would be to publish the spec of the ntd files.... Then we could read them by custom programming...

[darthtrader]

Ok that works, but then there is no way to link the bid/ask values to the price values. They are just other charts. So in that case I suppose there would be no way to compute the historical delta volume....

I don't have the beta to mess around with yet but doesn't this depend on how these are updating:
Add(Instrument.FullName,PeriodType.Tick,1,MarketDa taType.Ask);
Add(Instrument.FullName,PeriodType.Tick,1,MarketDa taType.Bid);

I take it since you mentioned that the bid/ask bar array is very large, these update on each change of the bid/ask and are not just a snapshot of what the bid/ask was when last goes off...
Wouldn't a way to get historic delta volume then be to compare the timestamp of last with the timestamp of the bid and ask and see if its at/above/below/between?

I soppose I was expecting 7 to have a historic version of onmarketdata(), but with storing bid/ask this seems even more thorough, more a question of filtering out what we don't need.

[NinjaTrader_Josh]

darthtrader,

It is possible to have many ticks with the same time stamp.

[rt-trader]

I imagine there will be a considerable number of 6.5 users waiting for this Historical Bid/Ask capability to build historically what is now possible live via OnMarketData.

From these early comments it seems this may not be straightforward. Is it possible for some sample code on how this might be accomplished be posted?

Or better yet - an enhancement to (or derivative of) OnMarketData which will allow this to be more easily?

Thanks

PS I dont have NT7 yet so can not experiment myself.

[NinjaTrader_Ray]

I imagine there will be a considerable number of 6.5 users waiting for this Historical Bid/Ask capability to build historically what is now possible live via OnMarketData.

From these early comments it seems this may not be straightforward. Is it possible for some sample code on how this might be accomplished be posted?

Or better yet - an enhancement to (or derivative of) OnMarketData which will allow this to be more easily?

Thanks

PS I dont have NT7 yet so can not experiment myself.

Unfortunately there is no reference sample forthcoming anytime soon. This is strictly based on bandwidth and priority of other items however, this is one of those things on the list.

[rt-trader]

OK - thankyou for the reply Ray.

[funk101]

Ok, how can one accomplish "retrieving historical bid ask"? I asked before, is stream reading/writing from a serialized file the best way to achieve this? Is this a painfully slow approach? When a user refreshes the chart, the data get's loaded back in? What are the options for trying to implement this?

[darthtrader]

darthtrader,

It is possible to have many ticks with the same time stamp.

true, i soppose i was thinking about a thinner market earlier...
I was just kind of throwing that out there to hopefully get the ball rolling.
I can totally understand the reason for not messing with onmarketdata()...
breaking alot of other uses of it wouldn't make sense just to backfill this kind of stuff...

It just seems to me we do have the tools now to do this, just need to figure out how...I mean don't we basically have something like this if you were to visualize the code gomi posted as far as the arrays:
A - ask
B - bid
L - last

a a a aaa aa a a a aa a aaaa a
L L L L L L L L L L
bb b b bbb b bbbb b b bbbbb b

The problem is the index of the arrays will not line up but we do have the data we need to figure this out.
I could see how streaming from a file could work with onmarketdata but i would think we should be able to do this with just on bar update.

[NinjaTrader_Dierk]

Guys,

Let me outline the opportunities and limitations of the new historical bid/ask feature.
- you could have multiple bid/ask/last series on your indicator/strategy
- the historical bid/ask series would hold ALL bid/ask events sent out by the exchange. Thus, it would NOT be the bid/ask at the time a trade went off
- on the historical part they would be replayed by OnBarUpdate
- OnMarketData only would be triggered realtime
- on historical replay the series are synced by their timestamp
- timestamps in NT are down to 1 second granularity
- on realtime obviously the OnBarUpdate events on multiple series come in at the sequence as sent out by the exchange

So:
- the EXACT sequence of a mix of bid/ask/last events will NOT be maintained while processing the historical data from multiple series
- the EXACT sequence of the bid/ask/last events by themselves WILL be maintained while processing the historical data from multiple series

However, on events with DIFFERENT timestamps the sequence always IS maintained.

Hope this sheds some light.

[funk101]

Hmm, ok, but what about plotting based on price, vertically as it were? Same does not apply it would seem.

[NinjaTrader_Dierk]

Sorry I don't follow.

[dnoff]

Why is NT data only timestamped down to the second? Clearly in any type of normal instrument, the bid ask data synchronized to this level has zero chance of being useful if is not able to be sequenced? Is this a data provider limitation or a NT limitation?

If it is an NT limitation, this design decision does not make sense to as this implementation is not all going to help those who are looking to historically test or even reload indicators on charts that use onMarketData(). As a result I do not see the benefit of adding this to NT 7 in this manner???

(I may be misunderstanding your comments above Dierk, but I thought you mentioned that bid ask data on historical data is only synchronized by timestamp?)

[NinjaTrader_Josh]

Not all of the data providers NT supports offers millisecond timestamping.

[dnoff]

Not all of the data providers NT supports offers millisecond timestamping.

Thanks for the reply and this makes sense. However I would have thought as a design solution, could you have taken the opposite approach here. For example, I would include a timestamp that is down to the millisecond, and when a data provider does not support millisecond timestamps, I would add ".000" to the timestamp when it is received.

This way you are enabling a solution for those those data providers and traders wanting this. Today it seems that instead it is degraded to the lowest common denominator, you can support those data feeds that do. Then as user of the platform I can chose to go to those data vendors that support if it fits my trading needs??

[NinjaTrader_Dierk]

Guys, here is what you needed to understand: no whatever fine granularity would ensure that bid/ask/last could be replayed in the right sequence. There always could be bid/ask/last events having the exact same timestamp (as sent by the exchange) and then your trading system relying on the correct sequence would just fail.

-> asking for finer granularity on the timestamp is the wrong approach. The only approach which really would resolve the issue would be sequence counter. We held off on implementing this with NT7, since the ressource investment would have delayed NT7 much further than the majority of our users would be willing to accept.

[trend]

Hi,
how far back the bid/ask data in the NT servers go?
Any guidance for the brokers on their bid/ask historical data
availability?

thanks,

[NinjaTrader_Dierk]

- On our data server I believe you should see a couple of months worth of data.
- please contact your broker to clarify how far their data would go.

[darthtrader]

-> asking for finer granularity on the timestamp is the wrong approach. The only approach which really would resolve the issue would be sequence counter. We held off on implementing this with NT7, since the ressource investment would have delayed NT7 much further than the majority of our users would be willing to accept.

Dierk, no offensive but I can't disagree more. What you posted makes sense from a software engineering standpoint of getting things EXACTLY right. From a trading standpoint though, the exact sequence would only matter to someone trying to develope a real high frequency strategy system in Ninja...
Anyone with any real knowledge of high frequency strategies is not going to bother with Ninja anyway.
With this design decision though you guys are throwing out the baby with the bathwater..
For alot of people, I imagine what seperates Ninja from other software is onmarketdata()...It strikes me as obvious that a less than optimal historic onmarketdata solution would have vastly higher utility than no solution at all..
A great example is Volume profile..while I simply love your software..its hard to "pimp" to my market profile type trading friends when you have to caveat how great volume profile is with that if anything happens with your computer, you lose all your information...or if you wake up at 11am to get ready for an afternoon session, you should have set your alarm earlier to start collecting data pre cash open...its absurd from a trading standpoint.

[NinjaTrader_Dierk]

Most of the requests we got in that area or from people how wanted to maintain the EXACT sequence.

Not sure why you feel unable to restore your volume profiles on restarting your custom indicators.

[dnoff]

Dierk, no offensive but I can't disagree more. What you posted makes sense from a software engineering standpoint of getting things EXACTLY right. From a trading standpoint though, the exact sequence would only matter to someone trying to develope a real high frequency strategy system in Ninja...
Anyone with any real knowledge of high frequency strategies is not going to bother with Ninja anyway.
With this design decision though you guys are throwing out the baby with the bathwater..
For alot of people, I imagine what seperates Ninja from other software is onmarketdata()...It strikes me as obvious that a less than optimal historic onmarketdata solution would have vastly higher utility than no solution at all..
A great example is Volume profile..while I simply love your software..its hard to "pimp" to my market profile type trading friends when you have to caveat how great volume profile is with that if anything happens with your computer, you lose all your information...or if you wake up at 11am to get ready for an afternoon session, you should have set your alarm earlier to start collecting data pre cash open...its absurd from a trading standpoint.

Darthtrader - thanks for writing this. This is my thoughts also. I also use something similar to market profile (a custom developed variant but a similar concept in analyzing volume and price trader relative the market bid/ask that the exact point in time of the sale).

For me, I also find it hard that once I have loaded the charts my indicator in (amongst some others) that if there are any changes, I lose all of the indicator values for the session). I also am not able to backtest and validate anything, except for using market replay.

So I what I think would be helpful for me, like the volume profile, is an historical onmarketdata or a "lite" version of the bid/ask data that is synchronized with each tick

[rt-trader]

A question to Ray...

Back in post 17 of this thread you mentioned that reference sample which deals with handling historical tick data may be forthcoming when bandwidth allows.

I have spent many hours attempting (unsuccessfully) to properly use this capability (because I havent been able to find an accurate way to sync Series0 data start with Series1 data).

My problem (to this point at least) has been that there appears to be no current way to accurately fix the start point of the opening bar of the chart when loading and processing the dataseries up to the point of !Historical.

So Ray - would it be possible to for you to outline what the reference sample(s) or any planned methods might facilitate when they become available.

I believe many others would be interested to know this as well.

Thankyou...

[NinjaTrader_Dierk]

Unfortunately other priority issues are still on our plate (as Ray pointed out). We will get back to that item as soon as possible.

[warrior]

Dierk and others: I couldn't agree more with feedback given by darthtrader, dnoff, funk101, rt-trader, gomi etc. etc. It seems that this major feature request that all along has been promised with NT7 where historical bid/ask volume/price data fix so that indicators that rely on it such as volume profile, gomcd would work without having to run these indicators 24x7 collecting data themselves in files, was dropped. I even remember somewhere in the feature list that such data would be collected on NT servers for past couple of weeks so that indicators could get to them. Despite all the feedback and frustration voiced on this thread, looks like it is not a priority for you. I understand you have a major version uplift with NT7, but dropping this imp fix is a major disappointment leaving NT7 still a major step behind competition. Hoping you/RayD reconsider this for NT7.

[NinjaTrader_Dierk]

Thanks for your feedback.

[NickA]

Hi NT guys,

I'd like to add my 2 cents in support of this request to accurately sequence historical bid/ask/last data. You only have to look at Gomi's thread and indicators to see there is massive demand for this feature and these sorts of indicators.

The only reason I am not using NT6.5 (and it looks like I will not be using NT7.0 until this is addressed) is that it is imperative for me (and a whole load of approaches and indicators) to sequence this data properly so that it can be processed from the database in the same order as it was received in real time.

These ticks are received in certain sequence (and must be stored in the database ina certain sequence), scripts need to be able to retrieve them in that same sequence. Whether that is achieved by time stamping or sequence number or some other method is not so important.

I am not sure why NT chose to separate the 3 types of data? Globex sends them as a single data stream with a flag to signify that a tick is a bid or ask change or an actual print. The data stream is inherently sequenced so it does not matter if you get 5000 ticks with the same time stamp, they are recieved in order. Of course a data providers might mess with this but I am unaware of it if they do. (Other charting platforms can accurately sequence with a variety of data venfors so I would be surprised if this was the case).

Anyway a heartfelt plea to re consider the priority of this problem. :)

[NinjaTrader_Ray]

Thanks for sharing your thoughts. I do realize that there is demand for this and this is on our list for the future. Unfortunately, this is not something we can do for NT7 at this time since technically, there is a world of impllications in doing so and we are too far along to introduce anything that would put as back at square one in terms of a beta process.

[NickA]

Thanks for sharing your thoughts. I do realize that there is demand for this and this is on our list for the future. Unfortunately, this is not something we can do for NT7 at this time since technically, there is a world of impllications in doing so and we are too far along to introduce anything that would put as back at square one in terms of a beta process.

Thanks for your reply Ray. I appreciate that you must have lots of pressure to deliver NT7.0 and that this would need some architectural change to the database. I hope it is something that your developers can bear in mind so that it does not become even more difficult to implement at a later stage.

There might be interim solutions that would rely on additions to classes this would have less far reaching implications than changes. I do realise that it would still have some implications.

An example would be to add a couple of flags to each entry in the trade database indicating that a trade occurred at the best bid or the best ask. These could be returned through new methods or extensions to existing methods. Forgive me if I have my terminology wrong I am not an OO programmer.

Anyway I do appreciate the constraints that you guys must be operating under as open beta and release looms ever closer. (Having said that there is an argument for change sooner rather than later as it is likely to be harder to make these sorts of changes after release).:)

Cheers.

[paulg]

The only reason I am not using NT6.5 (and it looks like I will not be using NT7.0 until this is addressed) is that it is imperative for me (and a whole load of approaches and indicators) to sequence this data properly so that it can be processed from the database in the same order as it was received in real time.

It was one of the few exciting promises months ago.... now we're back to step... 6.5... making also any form of backtesting that relies on those information substantially unreliable. :rolleyes:

[NinjaTrader_Dierk]

Not sure what promises you are referring to which would not be met and why you feel like falling back to NT7.

[Ricam]

I am also disappointed that the historical bid ask data will not be available as a core functionality.

For now, would it be possible for Ninjatrader HQ to collect the GomRecorder ticker data 24 hours per day and make it available for download, so that those of us relying upon this data would not need to run our computers 24/7?

[NickA]

I am also disappointed that the historical bid ask data will not be available as a core functionality.

For now, would it be possible for Ninjatrader HQ to collect the GomRecorder ticker data 24 hours per day and make it available for download, so that those of us relying upon this data would not need to run our computers 24/7?

I can't help wondering if there is some sort of ingenious interim solution that would have low impact on existing architecture. I feel there must be, there nearly always is.

I wonder how the data is held on NT's servers? Is still a financial data series there?

I don't think anyone would disagree that this is a fundamental flaw in architecture. There will never be a god time to deal with it, but it seems to me in beta is better than after NT7 is live. My concern is that this won't see the light of day until NT 8.0.

[NinjaTrader_Dierk]

I am also disappointed that the historical bid ask data will not be available as a core functionality.

For now, would it be possible for Ninjatrader HQ to collect the GomRecorder ticker data 24 hours per day and make it available for download, so that those of us relying upon this data would not need to run our computers 24/7?
To clarify: Please read my posts #21 and #27 to understand what is available and what not. Thanks

[dnoff]

To clarify: Please read my posts #21 and #27 to understand what is available and what not. Thanks

Dierk,

Thanks for the reply - I know you guys have been flat out stablising the Beta and getting this release advanced. A while ago this was discussion of having samples of how to use the historical bid ask capability - has that been developed yet??

Dnoff

[NinjaTrader_Josh]

dnoff,

Unfortunately additional educational reference samples are not available yet, but to move things along it really is fairly simple.

All you have to do is think of it as a multi-series strategy. You use Add() to add the Bid/Ask bar objects and then you can access it just like how you would access different bars in a multi-time frame strategy.

Add("AAPL", PeriodType.Minute, 1, MarketDataType.Ask);
Add("AAPL", PeriodType.Minute, 1, MarketDataType.Bid);

[dnoff]

dnoff,

Unfortunately additional educational reference samples are not available yet, but to move things along it really is fairly simple.

All you have to do is think of it as a multi-series strategy. You use Add() to add the Bid/Ask bar objects and then you can access it just like how you would access different bars in a multi-time frame strategy.

Add("AAPL", PeriodType.Minute, 1, MarketDataType.Ask);
Add("AAPL", PeriodType.Minute, 1, MarketDataType.Bid);

Thanks for that - I guess my question is what happens with these data streams though when they are all at the second interval...consider the normal occurrence in real time over a 1 second timeframe:


bids: b1 b2b3 b4 b5
asks: a1 a2a3 a4 a5
trades: t1 t2 t3


so when using OnBarUpdate, the historical data is now all with the same 1 second timestamp, so do I get the same sequence of events individually played back at the intervals here, or is it that I get them all at the start of the second interval?? (I suspect that later, but I just want to clarify).

I need to know that t1 had b1 and a1, and t2 had b4 and a3 and t3 had a4 and b5 etc...

What I try to do in some indicators is to track bid and ask price at the time a trade is executed and perform certain calculations etc (like a market profile) so knowing what the bid/ask is at the time a trade turns up in OnBarUpdate is the "sauce" that makes that run.

Dnoff

[NinjaTrader_Josh]

Dnoff,

From what you have outlined on what you are trying to do you can take a step back and completely drop this historical bid/ask deal. You can use OnMarketData() or OnMarketDepth() to achieve what you want. The bid/ask events received before the last event = the current bid/ask prices as of that last trade.

When using a 1 sec interval though, CalculateOnBarClose determines when you will receive the events. Provided you set it to true, then it will update sequentially in the order you add the Bar objects. If primary = last, then you will receive lasts before bid/ask which is not what you want. You want to put last as the tertiary series to achieve the sequence you want.

[dnoff]

Dnoff,

From what you have outlined on what you are trying to do you can take a step back and completely drop this historical bid/ask deal. You can use OnMarketData() or OnMarketDepth() to achieve what you want. The bid/ask events received before the last event = the current bid/ask prices as of that last trade.

When using a 1 sec interval though, CalculateOnBarClose determines when you will receive the events. Provided you set it to true, then it will update sequentially in the order you add the Bar objects. If primary = last, then you will receive lasts before bid/ask which is not what you want. You want to put last as the tertiary series to achieve the sequence you want.


Thanks Josh,

I use OnMarketData today - this is how my indicators work. The issue that everyone commenting on here faces, is that the indicators only work going forward or with captured reply data. Thus we are stuck with the situation whereby to test/validate any trading strategies we need to create our own back-testing stats/systems using replay data.

I also cannot refresh any indicators or charts during the day without losing all previous calculations - accidently hit "f5" or have to reload any chart data and our calcs are gone and lost, which seems very archaic to most here.

To have systems developed that are not recoverable during the day is a tenuous position at best, which is why there are so many comments about the feature/capability we thought we were going to get in NT7 and really are not.

[dam5h]

i agree, this is a serious bummer!

i may be getting a refund (still in first 30 days) on my lifetime license [NinjaTrader_Dierk: removed reference to competitive product]. you only need to know if last was best bid or best offer. comon guys: its just one more bit (1 or 0).

for that matter i'd settle for a volume profile that uses historical ticks without bid/ask distinction. just fill in the profile back as far as i like with out starting fresh everytime i add a new chart. like the grey bars on the gom software, thats better than nothing!

[NinjaTrader_Dierk]

I removed the reference to a competitive product. Please contact sales AT ninjatrader DOT com if you wanted to get a refund while on our 30 days money back period. Thanks

[dam5h]

sorry about that competitive product thing, i was simply speaking my mind--it certainly wasn't designed to compromise any NT loyalty here on the forum.

any comment on the possibility of a profile without bid/ask distinction? the gross magnitude of volume at price is most important on the larger time frame, not so much whether it was at the bid or ask.

personally, i'd rather not ditch your product as i do love the dom and forthcoming nt7 looks like a nice product, but i'd appreciate if you'd clarify whether or not NT7 will have a more useful profile without relying on user's machine running uninterrupted 24/7.

thanks in advance

[NinjaTrader_Dierk]

Not sure I follow. In post #27 and #21 of this thread I provided background on our business decision.

Furthermore: with NT7 you can download replay data from our server. Thus I'm not sure why you feel you needed to have your PC running 24/7.

[dam5h]

this replay thing sounds interesting, how quickly does this happen? you're suggesting that every time you refresh the chart you would need to download something separate than what happens when you create a new chart and the prices populate? any documentation on this? is there intelligent logic that appends old files to the new updated data since last download.

i studied the thread (including your previous posts 21 and 27) and it seems there may be a disconnect in what some folks are referring to and what you are talking about. i am confused as to how those posts answer my previous questions, i could be missing something entirely though.

i am not interested it every bid/ask event from the exchange, that would be a ton of data. i'm only interested in whether or not the "last" price was at the bid or the ask. i'm mainly interested in the volume profile, and it seems that my fellow recent complainers are on the same page. i would love to see something like GOMVolumeProfile available natively using data from the NT server, without having to keep an extra file on hand (be it a replay download or self recorded).

thanks again.

[NickA]

Not sure I follow. In post #27 and #21 of this thread I provided background on our business decision.

Furthermore: with NT7 you can download replay data from our server. Thus I'm not sure why you feel you needed to have your PC running 24/7.

That sounds like it might be a life saver. Does replay data have correctly sequenced (the same sequenced as occured in real time) bid/ask/last data?

[dam5h]

in previous posts he described it would be time stamp dependent with 1 sec granularity.

[GoldStandard]

This is the one single issue that still has me, and a large number of other people I know, looking at other platforms. There are so many things that depend on this information to work correctly... indicators, backtests, optimization. Without it all the other great features of Ninja are cast into doubt because you're not sure if you can rely on them. Without reliable data, nothing of reliably high quality can be built on top of it. It doesnt matter how many great features or indicators the platform has if you can't trust the data powering it.

I've been trying to interest an experienced trading systems programmer I know in working with Ninjatrader and when he found out that the bid/ask data he would have to use for backtesting is not in order he lost interest pretty quickly.

[timmyb]

i agree, this is a serious bummer!

i may be getting a refund (still in first 30 days) on my lifetime license [NinjaTrader_Dierk: removed reference to competitive product]. you only need to know if last was best bid or best offer. comon guys: its just one more bit (1 or 0).

for that matter i'd settle for a volume profile that uses historical ticks without bid/ask distinction. just fill in the profile back as far as i like with out starting fresh everytime i add a new chart. like the grey bars on the gom software, thats better than nothing!


I am on the 3 month paying plan and this will be my final 3 months also. We requested this feature over a year ago and Ninja chose to ignore its customers in this behalf. I suggest anyone else also disgusted with current bid/ask data errors and storage to state they intend to move also.

[timmyb]

Not sure I follow. In post #27 and #21 of this thread I provided background on our business decision.

Furthermore: with NT7 you can download replay data from our server. Thus I'm not sure why you feel you needed to have your PC running 24/7.

The reason they are runnign 24-7 is that we trade futures. We need 24 hour bid/ask data. Since ninja doesnt allow for proper tick data storage we must record it ourselves with GOMCD. Without the 24 hours data what most of us do with the bid/ask data is useless. Just try saving 3 days of data yourself then go load a new indicator and see it all dissapear. OH JOY

[NinjaTrader_Dierk]

>> also disgusted with current bid/ask data errors a
Not sure what "bid/ask errors" you are referring to. Please clarify. Thanks

[tulanch]

Perhaps this is what he is talking about...when I use the replay data I do not automatically get 24 hours of data, I only get the data from when I logged on till I logged off...thus only the data Ninja "touches" during the time I was logged in (by design of course). Thus if I leave myself logged in for 24 hours, my market replay data contains all the data for the 24 hour period becuase Ninja "touched" and thus stored all the incomming data in market replay database. But...your connection needs to be rock solid and the datafeed needs to remain in tack..sometimes during the daily futures market shutdown I find that the system does not recover gracefully for what ever reason.

I can use Market Replay data to "refreash" my bid/ask indicators (only during market replay of course) but a true windows "refresh" event can not. I belive I read that bid/ask data is not stored the same in both approaches.

I'm wondering if there is a way to save the dataseries data I use in my indicator and have it used on the chart update after a windows refresh? I'm certain I could write it to disk and pull it back in...will give that a try...I think thats what the one GOMCD indicator does...but numerous issues arrise related to the chart time frame, it would need to remain the same for this apparoch or the raw market data would need to be saved and processed accordingly.

[NinjaTrader_Josh]

tulanch,

I do not believe this is an error. This is just the limitations of the historical bid/ask streams vs replay as discussed.

[timmyb]

Perhaps this is what he is talking about...when I use the replay data I do not automatically get 24 hours of data, I only get the data from when I logged on till I logged off...thus only the data Ninja "touches" during the time I was logged in (by design of course). Thus if I leave myself logged in for 24 hours, my market replay data contains all the data for the 24 hour period becuase Ninja "touched" and thus stored all the incomming data in market replay database. But...your connection needs to be rock solid and the datafeed needs to remain in tack..sometimes during the daily futures market shutdown I find that the system does not recover gracefully for what ever reason.

I can use Market Replay data to "refreash" my bid/ask indicators (only during market replay of course) but a true windows "refresh" event can not. I belive I read that bid/ask data is not stored the same in both approaches.

I'm wondering if there is a way to save the dataseries data I use in my indicator and have it used on the chart update after a windows refresh? I'm certain I could write it to disk and pull it back in...will give that a try...I think thats what the one GOMCD indicator does...but numerous issues arrise related to the chart time frame, it would need to remain the same for this apparoch or the raw market data would need to be saved and processed accordingly.

The GOMCD indicator will record it easily for you. Unfortunately All CD usuers are having serious issues with data dropping using ninja now and it is no longer recomended for this type of trading :( HOpefully the issues can be fixed soon.

[NickA]

tulanch,

I do not believe this is an error. This is just the limitations of the historical bid/ask streams vs replay as discussed.

Whether you call it an "error" "oversight" "design limitation" does not matter unless the bid ask and last historical data can be sequenced in the same order as it occurred live it is not really useful.

I hope that I do not appear overly critical that is not my intention. The bigger concern is that I don't think the guys that originally architected things like this fully understood how historical bid ask data might be used. My ongoing concern is that perhaps the issues are still not fully understood now. It is simply my intention to ensure they are (understood) and that is the reason for my posts:). It is imperative that historical bid ask and last data can be processed in the same sequence that it was received in real time. That is all really.

Can bid ask and last data be replayed preserving the same sequencing of all data streams in relation to each other as occured in realtime? If they can then there is may well be a work around.

[NinjaTrader_Dierk]

We understood your request very well (since it's reasonable). However, I hope you understand our business decision to not support what you are looking for at this time (which I outlined in earlier posts) as well. Thanks

Also: see #27, #21 in this thread.

[NickA]

We understood your request very well (since it's reasonable). However, I hope you understand our business decision to not support what you are looking for at this time (which I outlined in earlier posts) as well. Thanks

Also: see #27, #21 in this thread.

I certainly respect the decision as I have mentioned previously too. :) I appreciate that tough decisions need to be taken to get the product out of the door. I still can't help feeling that it might not be the best decision in the long run. As Raymond pointed out changing things has implications, but, not changing things certainly does too :). It must have been a tough call!

[GoldStandard]

Any chance of getting bid/ask sequencing in NT 7.1 or 7.2?

We understood your request very well (since it's reasonable). However, I hope you understand our business decision to not support what you are looking for at this time (which I outlined in earlier posts) as well. Thanks

Also: see #27, #21 in this thread.

[NinjaTrader_Dierk]

At this time we have no plans nor commitments to disclose regarding this issue.

[superarrow]

Hi NT,
I have been following the discussions in this thread with great interest, as well as in some trading forums. I strongly subscribe to the opinion for inclusion of the realtime sequence of bid/ask/last into the historical framework for future releases of NT. My trading depends on it to a great deal. So naturally this is important to me but I also understand the business decision NT has taken to date.

NT has come a long way and I am already happy with this great product. Most of the competing products took alot longer to get to where NT is today. I still wouldn't trade with them for their products lack in areas that NT shines.

However, as every business is dependent upon having an edge and providing its clients with the right tools, could I kindly request NT put the needs prescribed in this thread as a high priority for the next generation of NT if not sooner.

Cheers,
SA

[NinjaTrader_Dierk]

Thanks for your feedback.

[KBJ]

...as every business is dependent upon having an edge and providing its clients with the right tools, could I kindly request NT put the needs prescribed in this thread as a high priority for the next generation of NT if not sooner.

Generally, I try not to just add "me too", when it's already been said (and quite well) by others, but in this case I feel that staying out of the discussion is a tacit agreement that the product is acceptable the way it is.

And this is not the case: it was expected that NinjaTrader V7 would have this feature, and its lack is a tremendous disappointment, and in some ways a "deal breaker" for my group.

Also, I know that the NinjaTrader team has their hands quite full at the moment (as I also have done software field testing and many releases over the years and have a fairly good idea of what you are going through) and this is why I just bite my tongue most of the the time instead of making suggestions that would be nice for only few people and not a majority of the NinjaTrader community.

That being said, sooner would be far better on this issue.

[NinjaTrader_Ray]

Hi All,

We do realize that there is demand for historical bid/ask data associated at a per tick level. We also realize that this is important for some of our users. Our focus now is to get NT7 stable and into production. Once we have hit that milestone, we will review this requirement to see what can be done. Thanks for your understanding.

[laredo]

this must be in error

http://vimeo.com/8943177

[NickA]

this must be in error

http://vimeo.com/8943177

Without stating why you think it is 'an error' it will be hard for anyone to help your understanding. If you read the last dozen or so posts in this thread carefully then it should be clear what the issue is.

[laredo]

Not with you Nick,but Im not concerned w/it.

[paulg]

I've been trying to interest an experienced trading systems programmer I know in working with Ninjatrader and when he found out that the bid/ask data he would have to use for backtesting is not in order he lost interest pretty quickly.

Clearly the real market for Ninja is the 3rd party software that can produce good-looking indicators that seem to work wonderfully in realtime (maybe) but they are worthless when you read the backtesting results.
This remind me the so-called bouncing ticks issue in mid-90' when trading systems developed with TradeStation showed overvalued P&L (obviously positive) results to be published on the web or on the magazines.
Just my 2 cents.

[timmyb]

What is teh bid/ask there in the historical data manager for if we arent able to download it. I am confused even more now



Hi All,

We do realize that there is demand for historical bid/ask data associated at a per tick level. We also realize that this is important for some of our users. Our focus now is to get NT7 stable and into production. Once we have hit that milestone, we will review this requirement to see what can be done. Thanks for your understanding.

[NinjaTrader_Josh]

timmyb,

There is a clear line drawing the difference between what is being discussed in this thread and what is in the Historical Data Manager.

What is being asked for here is an exact tick by tick storing of the sequence all bid, ask, last, etc. come in on. What is available is just a stream of bid, stream of ask, and stream of last prices. Which one comes first is unknown when they share the same timestamp.

[Ricam]

What everybody wants to to know is what the current bid and ask prices were for each trade. This information is easy to get in real time. This suggestion about how to make it available for historical data DOES NOT require any changes to the time stamping. However, changes would have to be made to the internal workings of Ninjatrader, which appear to be DLL's that read and write the historical and replay data to the binary data files.

In each series (bid, ask, last) there is a VOLUME field. Add a decimal point to the volume. After that add two digits. These are to be ignored (truncated) when a chart object fetches volume data from the historical data store. However, these digits allow the current bid and ask prices to be recorded as offsets from the Close price.

In the Last file, the first added digit signifies how many TickSizes separate the Close price from the current Bid price, and the second digit signifies how many TickSizes separate the Close price from the current Ask price. One value in each series can be reserved for "out of range/unknown". (So let's say 0 = out of range, 1 = 4 ticks below, 2 = 3 ticks below.... 5 = no difference, 6 = 1 tick above.. etc)

In the Bid file, the first added digit signifies how many TickSizes separate the Bid price from the current Close price, and the second digit signifies how many TickSizes separate the Bid price from the current Ask price.

etc.. You get the idea.

Some variation of this should work without rocking the boat too much. Or have I been dreaming......

[tulanch]

What seems strange to me is Historic data (back testing) does not seem to have this information readily available, but when I use Market Replay of previous days...wala it's there.

[NinjaTrader_Josh]

tulanch,

Market Replay has it available because it is literally recording the market exactly how it played in the exact order things occurred. Backtesting does not have this. It is only running on one stream of data, usually the last prices. You can add additional data streams for bid and ask, but you do not have the exact sequence things occurred at. You can only run off of timestamps.

[laredo]

Does the 'market re-play' store this data on my PC?
In this case the hist. Bid Ask

Tks

[tulanch]

as far as I know it does, but I've not dug into it, as it's tucked away in the DB within the Ninja folder structure I belive.

I had to turn the collection aspect on in my options. This action causes Ninja to store the entire session data while I am logged in. I only focus on the YM.

I have turned off all updates, time outs, power saving features, etc... and let my machine run each day for the entire session.

I then just use the connect -> market replay feature and my bid/ask indicators work great.


I have months of this data segmented by day. I "practice" or test my indicators this way. (I would think someone could build a business around suppling data in this format) Another interesting twist is if/when you hit re-freash button (wish I could turn that off) you can stop the replay, rewind it, restart it, and wala you get the bid/ask indicator display back...it works perfectly if I fast forward the replay (even 500 times faster).


enjoy

[laredo]

I wonder why Big Mike trading is publishing that if comes off the NT servers .

[laredo]

In the same fashion I assume that ensign software supplies back up hist.data via DTN.

[gregid]

I wonder why Big Mike trading is publishing that if comes off the NT servers .

The market replay data Big Mike is providing on his forum is not the same as NT server market replay. He and other users of the forum upload market replay that they recorded themselves using NT6.5.

The NT market replay server data is only available for NT7

[mrlogik]

Hey Guys,

I wrote up a test indicator to get historical tick data via OnBarUpdate, and also OnMarketData.

The Tick data series I added to the indicator is via

Add(Instrument.FullName,PeriodType.Tick,1,MarketDa taType.Bid);


In the OnBarUpdate, I filter on printing only when the Bid data stream is updated with BarsInProgress.
In the OnMarketData, I filter on printing only when the Bid data stream is updated with MarketDataType.Bid.

When I run this on my chart, I notice for every one OnBarUpdate print I get three OnMarketData calls.

OnMarketData = 1.5386
OnMarketData = 1.5386
OnMarketData = 1.5386
******************
OnBarUpdate = 1.5386

Is this expected? I would assume I should get one from each function call per incoming tick.

Indicator attached to reproduce.

NT7.14B, MBTrading data feed.

Thanks

[NinjaTrader_Ray]

Thanks for the sample code, we will investigate this.

[mrlogik]

You got it Ray. Please let me know if you need me to investigate further.

thanks

[NinjaTrader_Ray]

What instrument did you run this on?

[mrlogik]

This was on GBPUSD.

[NinjaTrader_Ray]

What was the primary series? Did you run the indicator in a chart or something else?

[mrlogik]

Hi Ray,

Sorry for my scant description.

I loaded a 1 minute chart of GBPUSD only. The indicator was loaded on the chart with the default inputs.

[NinjaTrader_Dierk]

1 minute on Last, correct?

[mrlogik]

Yes.

I am filtering the OnBarUpdate on the BarInProgress == 1 attribute, which I believe corresponds to

Add(Instrument.FullName, PeriodType.Tick, 1, MarketDataType.Bid);

[NinjaTrader_Dierk]

I always see a sequence of 2 OnMarketData events (identical) and 1 OnBarUpdate event which is expected since both series subscribe to the same instrument. There is no filter on "bid" or "ask" or "last", only on instrument.

Note: BarsInProgress always is 0, since NT finds the first series matching an instrument (which is BIP=0).

Note: ignore the "*****" lines since they rather confuse the image.

[mrlogik]

Hey Dierk,

Yes, I see two now as well.

I think I see what you're saying.

So now, for example, lets say I'm running on a 1 minute chart with CalculateOnbarClose = false. In the Initialize() function I do the following

Add(Instrument.FullName,PeriodType.Tick,1,MarketDa taType.Ask);
Add(Instrument.FullName,PeriodType.Tick,1,MarketDa taType.Bid);

In OnBarUpdate(), to differentiate between the primary data series, Ask, and Bid is the following true?

Primary data Series BarsInProgress = 0, ref Price = Closes[0][0]
Ask data Series BarsInProgress = 1, ref Price = Closes[1][0]
Bid data Series BarsInProgress = 2, ref Price = Closes[2][0]

And, how could I filter on a particular instrument then in the OnMarketData function, to differentiate between instruments?

Thanks again.

[NinjaTrader_Dierk]

>> And, how could I filter on a particular instrument then in the OnMarketData function, to differentiate between instruments?
Unfortuantely I would not know of a way to do that since all 3 series are on the same instrument.

[Cheater5]

Hello all:

I'm new to NT (using NT7) and am currently migrating from TradeStation. Forgive me if I'm off on this but don't at least some data services provide historical volume profile for each bar (ie, up trade volume/down trade volume and upticks/downticks)? I know that in TS I get up/down volume and up/dn ticks for each bar (equities, options and futes). Since Reuters provides this data, isn't the fix for this as easy as NT7 supporting Reuters QuoteCenter?

[NinjaTrader_Bertrand]

Cheater5, welcome to NinjaTrader and thanks for the suggestion to add QuoteCenter support - making the volume profiles persisting through the needed historical data is already on our list to look into for the future.