yesterday's data all adjusted downwards!

[adamus]

Yesterday I ran strategies via IB and of course had to close down at 22:15 GMT+0 because of the usual IB TWS daily forced shutdown. Stupidly I didn't have a chart open for $GBPUSD and it was only because my strategies were giving me crazy log statements that I checked.

Please see the attached image.

What could cause this?

Did it collect data badly all day or did it just adjust the data for some reason?

I have the Market Analyzer open with $GBPUSD plus indicators in order to capture tick data from IB.

I just looked in my HDM now and there is no tick data nor minute data for yesterday.

And now I just looked at the chart and all of today's data has dropped down to the same level as yesterday's!

I'm using NT7 beta 18 with IB TWS 906 as recommended.

Log files are attached.

[adamus]

So what can I do to get $GBPUSD back?

[NinjaTrader_Josh]

adamus,

Are you reloading your data? What steps did you take from between looking at today's data to when today's data appeared to be down adjusted? What does it show in the HDM edit tab?

[adamus]

This token handling for expired sessions caused me to retype my message for the third time now! Is there no way in this forum to set it not to expire?

Anyway, re: the issue at hand - your colleague Chris logged into my machine via logmein123.com to solve another issue right after I posted, so he fixed up the data for me with 'repair'.

I don't know what caused it the second time, my guess is that the Market Analyzer or the Strategy Analyzer requested data from IB which is currently causing problems with the recommended build for NT7beta18 - 906 won't connect for me. I have downgraded to 900.

Next time it happens, I'll note down literally every action I take for the record.

[NinjaTrader_Josh]

Thank you for your efforts. We appreciate it.

[adamus]

I have to notify you that this problem hasn't gone away.

There are generally about 2 or 3 occurences of data being messed up every week.

By "messed up" I don't mean always as badly as shown above, rather a 'flat spot' where 10 minutes or an hour of data contains the same price for OHLC and for every bar in the flat spot, so you see just a horizontal line, or massively bad ticks.

Normally it is not isolated to one instrument, but can affect all instruments in the Market Analyzer.

I assume that it is the Market Analyzer which collects or creates the problem, or it is something in the interface with IB TWS API.

A normal reload of the historical data in the chart fixes the issue.

However when I don't notice it, I believe it affects my strategies running.

I had a massively bad tick from midnight 00:00 9th Sept in $EURCHF (and 3 other instruments). I think this caused the strategy to run badly and it attempted to submit orders with nonsensical prices, which triggered it to shutdown automatically.

$USDCHF Sell Market 100000 0 0 Filled 100000 1.02321 0 DU58298 IB Simulated ThreeDucks002 10/09/2010 09:49
$USDCHF Sell Limit 100000 67.5310 0 Cancelled 0 0 100000 DU58298 IB Simulated ThreeDucks002 10/09/2010 09:49
$USDCHF Sell Stop 100000 0 -4.7911 Rejected: Error validating request:-'xb' : cause - Negative stop price is not supported (321) 0 0 100000 DU58298 IB Simulated ThreeDucks002 10/09/2010 09:49
$USDCHF Buy Stop 100000 0 1.0230'5 Filled 100000 1.0231 0 DU58298 IB Simulated ThreeDucks002 10/09/2010 09:49

[NinjaTrader_Josh]

adamus,

Ultimately NT just acts like a radio receiver for information the data provider pushes through their API. What I suggest is to turn on the real-time tick filter in Tools>Options on your end and filter away these bad ticks.

[adamus]

That would sort out the bad ticks.

The flat spots though are a different story.

And the original problem which I outlined - that I can't reproduce to date - was most likely due to NinjaTrader corrupting the data, rather than IB. What I will do is try to run an NT chart side by side with an IB chart to see whether it occurs on both. I suspect that it is disconnecting and reconnecting to IB that causes the issue.

Secondly the strategy behaviour - I just saw a repeat of the issue of bad order prices with no sign of bad ticks or flat spots from the data, so I guess that is the strategy coding which I need to check.

[NinjaTrader_Josh]

adamus,

NT does not corrupt/change or do anything with data. It comes in at whatever price and that is what NT will show. Please note that IB's TWS is not equivalent to their TWS API. Their charts on their TWS software might look fine, but as they push data through their API it simply can be different. For example, volume coming through TWS API has been known to be very different in real-time through their API than their historical volume, yet the issue is completely nonexistent on their TWS software. This is outside the control of NT as we cannot control what and how the data is pushed down through their API.

[adamus]

OK I see you've put your defensive shields up. I'll try to get some help on the TWS API list I am a subscriber to. I'll obviously be back if I have anything to stick on NinjaTrader.

[adamus]

adamus,

NT does not corrupt/change or do anything with data. It comes in at whatever price and that is what NT will show. Please note that IB's TWS is not equivalent to their TWS API. Their charts on their TWS software might look fine, but as they push data through their API it simply can be different. For example, volume coming through TWS API has been known to be very different in real-time through their API than their historical volume, yet the issue is completely nonexistent on their TWS software. This is outside the control of NT as we cannot control what and how the data is pushed down through their API.

I can't find anybody who suffers the same degree of data disruption as me purely through the TWS API.

On Friday close of business I shutdown NT with my charts and data in more or less perfect condition.

Tonight, at 21:00 GMT, I turned NT back on without reconnecting to IB. I cheched the data and found what you can see in the images attached.

The charts show exactly what I was talking about above re 'flat spots' in the data. That data was fine when everything was running on Friday night. NinjaTrader though either in shutdown at that point or just now at start-up has wiped the data for 18:30 to 22:00.

[NinjaTrader_Josh]

adamus,

-Do you have "Save chart data as historical" checked?
-Do you have "Get data from server" checked?
-What session template is your charts on?
-What timezone is your PC in?
-Were the charts left open the entire time or are these newly created charts?

[adamus]

-Do you have "Save chart data as historical" checked?

Yes

-Do you have "Get data from server" checked?

Yes

-What session template is your charts on?

<Use session template> i.e. forex.

-What timezone is your PC in?

GMT+1

-Were the charts left open the entire time or are these newly created charts?

This was one chart, saved in the workspace. I closed NT on Friday evening and restarted it Sunday, and NT opened the chart automatically. I only had the one chart window. I looked through all the instruments in my default list and 8 out of 10 were affected - I uploaded images of some.

[NinjaTrader_Josh]

And you only see this happen on weekends? And not only did you not reconnect to IB, but you literally make zero connections at all as you started up NT and the issue was already seen?

[adamus]

It's not just a weekend phenomenon.

Your assumption is correct - I don't connect NT to anything automatically. I just opened up NT and the graph was sitting there in front of me with the flat line at the end.

Unfortunately it's not a one-off situation, this has happened several times previously and sometimes the data corruption takes other forms.

[NinjaTrader_Josh]

And to be sure we cover all bases, you are on B21 now correct?

[adamus]

I upgraded Sunday from beta 20 just before I became aware of this occurence.

[NinjaTrader_Josh]

I will throw this on a chart on my computer here and run it for a day and see what I get tomorrow.

[adamus]

You might have to repeat it a few times. My estimate is that this happens approximately 1 in every 5 re-starts.

[adamus]

But then again it might occur straight away, as has happened here.

IB TWS closed itself down - something I normally prevent but things were a bit hectic and I missed my alarm.

On opening it, I have a different type of error - a bad bar. Exactly the same process as before. This time I then reconnected to IB but the bad bar remains - presumably until I redownload the data.

I'll have a look in the historical data to see what it is. Here are some images. It affected 7 out of 11 instruments in my default list.

Looking in the HDM, I can't see any bad ticks but I have a bad 1 minute bar for today 07:00 GMT+1 with the Open and the Low at 1.2871 on the $EURUSD for instance. The previous bar close was 1.2994 and the High is 1.2996'5 and Close is 1.2993.

The bad bar has the same timestamp on each chart where it appears.

Hope that helps,

Adam

[adamus]

I discovered that the bad bars in all the forex data originates from IQFeed.

I subscribe to IQFeed for FXCM data as a substitute for the slow, restricted IB historical data.

I can reproduce the bad bars.

However I cannot reproduce the other bad data and while investigating this issue, just came across another event. See images. Gaps in AUDUSD, NZDUSD and USDCAD.

The gaps appeared after I disconnected and restarted NT. My usual graph showed gaps on these three instruments - all 8 others were fine.

On reconnecting to IB with historical data disabled, the gaps remained. I then connected to FXCM and the gaps were filled but the bad bars appeared.

I downloaded the IB historical data to get rid of the bad bars. The gaps do not appear in the IB data either.

Hope that helps

[NinjaTrader_Josh]

adamus,

Please go back to the step where you connected to FXCM. After getting the bad bars, please open up your Historical Data Manager and find that bar in the Edit tab. If the data is listed "wrong" in there then unfortunately that is simply the data being pushed down the pipe from your data provider. When you run into such a scenario you have the options to edit that bar in the Historical Data Manager. This is why this feature was added in NT7.

Also, you mention now that you connect to IQFeed as well. How does this play in for your previous reports? Did you connect to IQFeed at all? I recommend simplifying the scenario and just stick to one only for the flatline issue. (Note: My test yesterday was unsuccessful in reproducing.)

[adamus]

I discovered that the bad bars in all the forex data originates from IQFeed.

I subscribe to IQFeed for FXCM data as a substitute for the slow, restricted IB historical data.

I can reproduce the bad bars.

However I cannot reproduce the other bad data and while investigating this issue, just came across another event. See images. Gaps in AUDUSD, NZDUSD and USDCAD.

The gaps appeared after I disconnected and restarted NT. My usual graph showed gaps on these three instruments - all 8 others were fine.

On reconnecting to IB with historical data disabled, the gaps remained. I then connected to FXCM and the gaps were filled but the bad bars appeared.

I downloaded the IB historical data to get rid of the bad bars. The gaps do not appear in the IB data either.

Hope that helps

As I said above, the FXCM data was bad. That is not an issue here.

I cannot stop using IQFeed (as I said above). IB historical data is slow to download and restricted. Plus there are connectivity issues between NT and IB. If I stopped using IQFeed, I would stop using NT.

Anyway, it is probably part of the problem. I have my IB connection set up not to download historical data, so NT must get the history from IQFeed.

The gaps I mentioned above with the chart images were probably down to this.

[NinjaTrader_Josh]

adamus,

Please then outline the complete exact interaction between both data feeds. This is very important. We will need to know exactly when which feed is connected first, connected second, etc. We need to know exactly which feed is disconnected first and second. We need to know if one feed was left on while the other was shut off.

Please provide exact listed steps. For example:

1. Connect IB
2. Connect IQFeed
3. Open up new 15min chart of $EURGBP on Forex session template
4. Leave running all day
...etc...

The relationship and order in which things are done is very important. Thank you for the information.

[adamus]

When it happens again, I'll note exactly what I did. I'm not 100% sure my recollection of the last event is correct.

[adamus]

Didn't see any anomalies in the data this weekend.

I'll keep checking.

I noted the way I shutdown and restart.

When shutting down on Friday night, I don't disconnect the live feeds manually. I just disable the strategies that are running and then exit NT.

On Sunday evening when I restart, I start two instances of IB - one live, one simulated account - and then I start NT.

Before connecting NT to IB at all, I look through the data from the last week to see if it's OK.

I hook up the live IB account first and then the simulated account. Both account connections are configured not to download historical data.

I then connect the IQFeed.

So that's the way I do it, for the record.

[NinjaTrader_Josh]

Thanks for the info. I did not see anything out of whack on my end this weekend too.

[adamus]

I experienced an unusual occurrence with the chart updates yesterday.

I had run NT for roughly 18 hours already, and at 18:00 last night, for an hour or more I had not looked at the chart and I switched to it. It was a 5 min chart and when the next bar started as I watched, the price gapped upwards. I thought that was quite bizarre for mid-session and considering the relatively low volatility, and then I saw the times of the bars jumped 50 mins and the chart had just skipped about 10 bars. This was the same across all forex pairs in my default lists, which I cycled through to check.

I looked in the logs but couldn't see anything. There were no connectivity problems, no errors or warnings, nothing suspicious.

Here's the log: TracesAndLogs.zip (http://www.ninjatrader-support2.com/upload/20100927_77be18f991d14430881b4087d2e55e1b_TracesAn dLogs.zip)

Normally I have the market analyzer running to collect the incoming tick data, but this week I forgot to start it or it got closed. Perhaps that would have prevented the issue.

Also suspicious is the secondary data supplier, IQFeed. I have IB connected first but with no historical data allowed, and then IQFeed. I sometimes see that the chart takes the IQFeed live stream, and sometimes IB. This time though I didn't check which was being used by the chart since it's difficult to tell (how could I tell, assuming I didn't know which provider was loaded first?).

[NinjaTrader_Kyle]

adamus,

Which instruments were effected, specifically?

Did reloading historical data clear the gap, or is it still in effect?

[adamus]

Hi Kyle,

reloading the data filled the gap.

It was AUDUSD, EURCHF, EURGBP, EURJPY, EURUSD, GBPJPY, GBPUSD, NZDUSD, USDCAD, USDCHF and USDJPY.

[NinjaTrader_Kyle]

adamus,

Thanks for the clarification.

1.) To confirm, you are using TWS 908.6, correct? (Help > About Trader Workstation)

2.) Please provide me with a copy of your workspace.
You will find workspaces in the Documents > NinjaTrader 7 > Workspaces folder.

[NinjaTrader_Brett]

Hello,

You lost connection in the trace at

2010-09-27 05:27:10:779 (IB) Cbi.Connection.ProcessEventArgs.ConnectionStatusEv entArgs Connected Connected NoError Market data farm connection is OK:cashfarm(2104)
2010-09-27 05:27:10:780 (IB) Cbi.ConnectionStatusEventArgs.Process: Connected Connected
2010-09-27 05:27:16:111 (IB) IB.Adapter.HandleMessage DataFarmConnectionOk


This is the reconnect time however this could explain the missing bars.

[adamus]

That's the wrong time. It was between 18:00 and 19:00.

But anyway then there should be a corresponding disconnected warning both in the bottom right corner of IB TWS and in the logs in NT, which there wasn't. (I'm searching on 'disconnect')

[adamus]

Last night as I was bringing up my strategies for the week and I ran into another data problem.

I lost the 29th September from my history for all symbols in my default list.

I'll try to detail the situation as it arose last night. Due to fighting other fires to get the strategies going, the whole process took a long time and I ended up too tired to post the message at that point. Hopefully I'll remember everything now.

I started up the PC, and installed NT beta22. I started up NT and connected to IB with the connection that allows historical data requests.

I decided to use IQFeed only in urgent situations or when IB is down, so I didn't connect to it then. So the only connection was the one connection to IB TWS live account.

I let the Market Analyzer window download its required historical data and get up and running on the live feed, which didn't take so long. It seemed that there was no disruption of the IB data feed detailed in another thread.

I switched to the chart window, which was also already opened by the NT startup. It was set to 30min bars on USDJPY with no indicators, only the Chart Trader active. I went through my default list of instruments (11 forex pairs), slowly waiting while each one loaded up historical data from the lame IB historical data server.

I didn't find any problems and I was about to post a msg here saying so, when I noticed suspicious slowness on the internet and I decided to reboot the broadband connection to stabilise it before the week began, hopefully. Since IB and NT should reconnect on their own, I left them running and went to the lan gateway and rebooted.

As the internet connection came back online, NT announced that the connection was lost, followed 5 seconds later by the reconnect.

I switched back to the trading workstation and both IB and NT were connected. However the chart showed that 29th Sept data was gone. On the 30 min bar chart, it went from 2010-09-29 00:00:00 straight to 2010-09-29 22:30:00 without a bar in between.

This was the same on all instruments in my default list. I had to reload all historical data for each one in turn.

Once that was finished, I was able to restart my strategies.

Here's a link to the traces & logs I uploaded: TracesAndLogs.zip (http://www.ninjatrader-support2.com/upload/20101004_a8dacc9343c84df09725c15f6a0a9bbb_TracesAn dLogs.zip)

[NinjaTrader_Kyle]

adamus,

Thanks for this - I'll see if I'm able to replicate based on what you've provided here.

[adamus]

Got another one for you.

Here's the TracesAndLogs.zip (http://www.ninjatrader-support2.com/upload/20101004_9ad39e015bf14866a2fa6713583fc0a2_TracesAn dLogs.zip)

and here's the log from IB, which is probably worth looking at: Attachment.zip (http://www.ninjatrader-support2.com/upload/20101004_9ad39e015bf14866a2fa6713583fc0a2_Attachme nt.zip)

and also here's what happened - see image:

Everything has been running 24 hours now almost.

During the day I added several symbols to my default list, including the one that isn't loading now $EURAUD. I loaded these extra symbols into the Market Analyzer. At that point, NT was able to obtain IB historical data, if slowly.

However just 10 minutes ago I changed the instrument on the chart to $EURAUD and NT decided to pull some historical data from IB, but it wouldn't download and repeated attempts failed and I even somehow ran into the "IB pacing violation" despite being unable to do a download.

The question is: is this affecting my strategies? They don't require historical data since they are running live off real-time data, correct? I am monitoring them for any reaction to bad data but so far I haven't detected any problem.

[NinjaTrader_Kyle]

adamus,

Clarify for me what type of strategies you're asking about - ATM or NinjaScript?

[adamus]

I'm asking about ninjascript strategies running in the strategies tab.

I have no idea what's under the hood as far as the data cache is concerned and whether this issue means that the whole data cache for all parts of NT has been foobar'd.

In my experience, if I try to call up any other instruments on this chart, the same bars will be affected and have bad data. Does this mean that the strategies running in the strategies tab will also be looking at bad data?

[adamus]

It might be relevant to your investigations to know that it sorts itself out after 23:00 GMT+1. Up until then, especially in the previous 2 or 3 hours, it's impossible to fetch any historical data.

[adamus]

I have another data problem. This time it's about 5 30min bars all adjusted downwards and flat-lining.

I think it was set off by a disconnect with IB. I was trying to reload the historical data on a chart. Since it was after the close tonight I thought I'd reboot. I didn't check any of the charts before rebooting so it might have occurred before that. After rebooting, I saw this problem on about 9 out of 11 of the markets in my default list.

By the way, this reloading historical data means that I can't collect the live data. There is a measurable difference between InteractiveBrokers' live data and the historical data they provide. I've checked it out and tested this.

[NinjaTrader_Josh]

At what time were you disconnected from IB? You should be able to see the time in the logs for when you were disconnected which would allow you to try and correlate with the chart.

[adamus]

These are the logs:

http://www.ninjatrader-support2.com/upload/20101006_0c2650aa90154ab8b7b8d7e8a42a7bac_TracesAn dLogs.zip

Yesterday was a bad day for disconnects.

I'm not sure why you ask what time I was disconnected. As I said, I believe the charts were messed up by that disconnection while I tried to load the chart. I think it was $EURAUD which I had just selected and I saw what I thought was a gap so I did a history reload to check.

Then it disconnected and then I had an unmistakeable error on the chart, so I just restarted.

[NinjaTrader_Kyle]

adamus,

The reason we ask is to check against what is in the diagnostic files, and to determine if this was an expected or unexpected disconnection.

Please update us on the time as you're able.

[adamus]

If I am reading the log correctly, it was 22:50. On the 5th.

[NinjaTrader_Kyle]

Hello adamus,

The disconnection here looks to be unexpected.

In working on my end, I'm not able to replicate the behavior when simply dropping my internet connection to initiate the disconnection.

Would this suffice to replicate the issue on your end?

So that I may further test, please provide me with your workspaces and chart templates.
Workspaces will be found in the Documents > NinjaTrader 7 > Workspaces folder.
Templates will be found in the Documents > NinjaTrader 7 > Templates > Chart folder.

[adamus]

Kyle,

I did have the issue on a couple of occasions when the internet connection when down. However that may be entirely co-incidental.

I have connectivity problems with IB and they are still unresolved and the result is that they just kill the IB connection to the IB historical market data servers at frequent random intervals, especially in the US afternoon session.

However I think you could try to replicate this quite simply by try to download so much data from IB that you run into the "pacing violation" that IB slaps down when it detects too many requests from you. This lasts five minutes after appearing so in that time you could then try to reload historical data on a chart.

Actually just a thought, that might not replicate the error since it will never get any data rather than getting some and then being interrupted.

[adamus]

Kyle,

I've sent it in the files via the app's embedded email function.

[NinjaTrader_Kyle]

Hi adamus,

Can you please send me a PM with your email address? I can't seem to find your message.

[adamus]

I just want to point out that this error is still occurring.

Please see the attached image. The whole 24 hours of data for 2010-11-24 has disappeared from the chart. This occurred to all instruments in my default lists apart from 1. So that is 20 'reload historical data' operations.