yesterday's data all adjusted downwards!

[adamus]

Yesterday I ran strategies via IB and of course had to close down at 22:15 GMT+0 because of the usual IB TWS daily forced shutdown. Stupidly I didn't have a chart open for $GBPUSD and it was only because my strategies were giving me crazy log statements that I checked.

Please see the attached image.

What could cause this?

Did it collect data badly all day or did it just adjust the data for some reason?

I have the Market Analyzer open with $GBPUSD plus indicators in order to capture tick data from IB.

I just looked in my HDM now and there is no tick data nor minute data for yesterday.

And now I just looked at the chart and all of today's data has dropped down to the same level as yesterday's!

I'm using NT7 beta 18 with IB TWS 906 as recommended.

Log files are attached.

[adamus]

So what can I do to get $GBPUSD back?

[NinjaTrader_JoshP]

adamus,

Are you reloading your data? What steps did you take from between looking at today's data to when today's data appeared to be down adjusted? What does it show in the HDM edit tab?

[adamus]

This token handling for expired sessions caused me to retype my message for the third time now! Is there no way in this forum to set it not to expire?

Anyway, re: the issue at hand - your colleague Chris logged into my machine via logmein123.com to solve another issue right after I posted, so he fixed up the data for me with 'repair'.

I don't know what caused it the second time, my guess is that the Market Analyzer or the Strategy Analyzer requested data from IB which is currently causing problems with the recommended build for NT7beta18 - 906 won't connect for me. I have downgraded to 900.

Next time it happens, I'll note down literally every action I take for the record.

[NinjaTrader_JoshP]

Thank you for your efforts. We appreciate it.

[adamus]

I have to notify you that this problem hasn't gone away.

There are generally about 2 or 3 occurences of data being messed up every week.

By "messed up" I don't mean always as badly as shown above, rather a 'flat spot' where 10 minutes or an hour of data contains the same price for OHLC and for every bar in the flat spot, so you see just a horizontal line, or massively bad ticks.

Normally it is not isolated to one instrument, but can affect all instruments in the Market Analyzer.

I assume that it is the Market Analyzer which collects or creates the problem, or it is something in the interface with IB TWS API.

A normal reload of the historical data in the chart fixes the issue.

However when I don't notice it, I believe it affects my strategies running.

I had a massively bad tick from midnight 00:00 9th Sept in $EURCHF (and 3 other instruments). I think this caused the strategy to run badly and it attempted to submit orders with nonsensical prices, which triggered it to shutdown automatically.

$USDCHF    Sell    Market    100000    0    0    Filled    100000    1.02321    0        DU58298    IB Simulated    ThreeDucks002    10/09/2010 09:49
$USDCHF    Sell    Limit    100000    67.5310    0    Cancelled    0    0    100000        DU58298    IB Simulated    ThreeDucks002    10/09/2010 09:49
$USDCHF    Sell    Stop    100000    0    -4.7911    Rejected: Error validating request:-'xb' : cause - Negative stop price is not supported (321)    0    0    100000        DU58298    IB Simulated    ThreeDucks002    10/09/2010 09:49
$USDCHF    Buy    Stop    100000    0    1.0230'5    Filled    100000    1.0231    0        DU58298    IB Simulated    ThreeDucks002    10/09/2010 09:49

[NinjaTrader_JoshP]

adamus,

Ultimately NT just acts like a radio receiver for information the data provider pushes through their API. What I suggest is to turn on the real-time tick filter in Tools>Options on your end and filter away these bad ticks.

[adamus]

That would sort out the bad ticks.

The flat spots though are a different story.

And the original problem which I outlined - that I can't reproduce to date - was most likely due to NinjaTrader corrupting the data, rather than IB. What I will do is try to run an NT chart side by side with an IB chart to see whether it occurs on both. I suspect that it is disconnecting and reconnecting to IB that causes the issue.

Secondly the strategy behaviour - I just saw a repeat of the issue of bad order prices with no sign of bad ticks or flat spots from the data, so I guess that is the strategy coding which I need to check.

[NinjaTrader_JoshP]

adamus,

NT does not corrupt/change or do anything with data. It comes in at whatever price and that is what NT will show. Please note that IB's TWS is not equivalent to their TWS API. Their charts on their TWS software might look fine, but as they push data through their API it simply can be different. For example, volume coming through TWS API has been known to be very different in real-time through their API than their historical volume, yet the issue is completely nonexistent on their TWS software. This is outside the control of NT as we cannot control what and how the data is pushed down through their API.

[adamus]

OK I see you've put your defensive shields up. I'll try to get some help on the TWS API list I am a subscriber to. I'll obviously be back if I have anything to stick on NinjaTrader.

[adamus]

adamus,

NT does not corrupt/change or do anything with data. It comes in at whatever price and that is what NT will show. Please note that IB's TWS is not equivalent to their TWS API. Their charts on their TWS software might look fine, but as they push data through their API it simply can be different. For example, volume coming through TWS API has been known to be very different in real-time through their API than their historical volume, yet the issue is completely nonexistent on their TWS software. This is outside the control of NT as we cannot control what and how the data is pushed down through their API.

I can't find anybody who suffers the same degree of data disruption as me purely through the TWS API.

On Friday close of business I shutdown NT with my charts and data in more or less perfect condition.

Tonight, at 21:00 GMT, I turned NT back on without reconnecting to IB. I cheched the data and found what you can see in the images attached.

The charts show exactly what I was talking about above re 'flat spots' in the data. That data was fine when everything was running on Friday night. NinjaTrader though either in shutdown at that point or just now at start-up has wiped the data for 18:30 to 22:00.

[NinjaTrader_JoshP]

adamus,

-Do you have "Save chart data as historical" checked?
-Do you have "Get data from server" checked?
-What session template is your charts on?
-What timezone is your PC in?
-Were the charts left open the entire time or are these newly created charts?

[adamus]

-Do you have "Save chart data as historical" checked?

Yes

-Do you have "Get data from server" checked?

Yes

-What session template is your charts on?

<Use session template> i.e. forex.

-What timezone is your PC in?

GMT+1

-Were the charts left open the entire time or are these newly created charts?

This was one chart, saved in the workspace. I closed NT on Friday evening and restarted it Sunday, and NT opened the chart automatically. I only had the one chart window. I looked through all the instruments in my default list and 8 out of 10 were affected - I uploaded images of some.

[NinjaTrader_JoshP]

And you only see this happen on weekends? And not only did you not reconnect to IB, but you literally make zero connections at all as you started up NT and the issue was already seen?

[adamus]

It's not just a weekend phenomenon.

Your assumption is correct - I don't connect NT to anything automatically. I just opened up NT and the graph was sitting there in front of me with the flat line at the end.

Unfortunately it's not a one-off situation, this has happened several times previously and sometimes the data corruption takes other forms.