Bug? Problem with import data from textfile

[FireFly]

If I export EOD data to a textfile, I see in the resulting file that there are only dates, not times. So the daily bar closing time (usually 9pm for the US markets in my timezone) is not in the file.

When I then import the same file back into NT7, the bar has a different closing time. The funny thing is that regardless of the timezone that I select when importing the data, the bar time is ALWAYS 5:00am. However, the DATE of the bar DOES change at some point depending on the timezone that I select.

So it looks like NT makes an assumption on the closing time of the EOD bars and then attempts to convert that to the correct timezone (therefore I see the date changing). But somehow the time remains fixed at 5:00am

[NinjaTrader_Brett]

Hello,

So if you use the time zone that your PC is set too? You get incorrect date?

The time stamp on the daily data is a virtual time stamp and can be ignored. Mainly I just want to make sure the date is correct.

Let me know if I can be of further assistance.

[FireFly]

Hello,

So if you use the time zone that your PC is set too? You get incorrect date?

The time stamp on the daily data is a virtual time stamp and can be ignored. Mainly I just want to make sure the date is correct.

Yes! If I use the same timezone as the PC is set to (in my case GMT) I get an incorrect date. This is how I discovered the problem. If I export a symbol and then import it again by a different name (by changing the filename) the two charts are not aligned.

Btw: even if the date were correct then there is still an alignment problem because the time is different. This can be seen when I put one chart above the other, bring up the databox and move with the crosshair around in the chart. The databox will sometimes show different dates for both instruments.

[NinjaTrader_Brett]

Hello,

Checking into this with development.

Thanks for your patience. May not be until tomarrow where you get a response from me.

[NinjaTrader_Josh]

FireFly,

Let us compare apples to apples first. Please look at the Historical Data Manager's Edit tab and write down a few dates with their respective prices/volume. Export your data, delete the old data, reimport. Then check in the Edit tab again (not charts). What do you see as far as the imported data? Remember to select the same timezone your PC is in.

[FireFly]

In the Edit tab I can see no difference between the original and the imported data. However, if I print the date and time in the output window using this statement:

Print(Time[0]+" Close[0]= "+Close[0]);

Then I get this:

Original:
12/21/2010 9:00:00 PM Close[0]= 14.89
12/22/2010 9:00:00 PM Close[0]= 15.14
12/23/2010 9:00:00 PM Close[0]= 15.34
12/27/2010 9:00:00 PM Close[0]= 15.23
12/28/2010 9:00:00 PM Close[0]= 15.25
12/29/2010 9:00:00 PM Close[0]= 15.13
12/30/2010 9:00:00 PM Close[0]= 15.21
12/31/2010 9:00:00 PM Close[0]= 15.39

Imported:
12/21/2010 5:00:00 AM Close[0]= 14.77
12/22/2010 5:00:00 AM Close[0]= 14.89
12/23/2010 5:00:00 AM Close[0]= 15.14
12/24/2010 5:00:00 AM Close[0]= 15.34
12/28/2010 5:00:00 AM Close[0]= 15.23
12/29/2010 5:00:00 AM Close[0]= 15.25
12/30/2010 5:00:00 AM Close[0]= 15.13
12/31/2010 5:00:00 AM Close[0]= 15.21

I have also attached a picture of what I get when I put both charts together. You can see the shift between both series.

[FireFly]

And to add to the previous post, here is what happens if I don't use my own timezone (GMT) when importing but use HongKong time (+8) instead:

12/21/2010 5:00:00 AM Close[0]= 14.89
12/22/2010 5:00:00 AM Close[0]= 15.14
12/23/2010 5:00:00 AM Close[0]= 15.34
12/27/2010 5:00:00 AM Close[0]= 15.23
12/28/2010 5:00:00 AM Close[0]= 15.25
12/29/2010 5:00:00 AM Close[0]= 15.13
12/30/2010 5:00:00 AM Close[0]= 15.21
12/31/2010 5:00:00 AM Close[0]= 15.39

As you can see, the date shift is then not happening but the time is stuck to 5:00 AM.

[FireFly]

Is this a confirmed bug now?

[NinjaTrader_Bertrand]

FireFly, sorry for the non update - I'll check with the techs who worked last on this one and get back to you.

Thanks,

[NinjaTrader_Brett]

Hello,

The times in the HDM are correct. However the times displayed in the chart will be adjusted based on your session template. What time zone is your PC in?

I look forward to assisting you further.

[FireFly]

I'm in the GMT timezone.

But as you can see from my previous posts, times in the chart are NOT properly adjusted based on the session template after importing data. Regardless of the timezone I always get as time 5:00am but sometimes the date switches.

[NinjaTrader_Brett]

Hello,

Ok heres the reason this is occuring.

When you export daily bars, they are exported without a time stamp.

When you import them back in they are given a generic time stamp of 12:00.

Therefor when you take the 12:00 Time stamp then apply your session time stamp too it then gives you the time you see.

Therefor this will always occur wih exported data.

The time stamp from your provider is not 12:00, it is most likely something else.This is why the data is mismatch.

I guess the main issue here is understanding, why your exporting the data and then importing it directly back in as this is not designed to function this way. Export of data is used to import into another application for example.

Let me know if I can be of further assistance.

[FireFly]

So what you are saying is that it doesn't matter which timezone I select when I import EOD data? This is clearly NOT the case, since the selected timezone can make the date jump one day (see previous posts).

So either:

1) NT assigns a generic timestamp of 12:00 when importing, regardless of the timezone. Then I should always see 5:00am, but then I should NEVER see the date jump based on the selected timezone (but I DO see the date jump!).

Or:

2) NT assigns a timestamp based on the selected timezone when importing. Then I should see different dates and times depending on the timezone setting at import.

It has to be one of these two. Otherwise there has to be a bug!

[NinjaTrader_Brett]

Hello,

For me to confirm personnaly with any case and to double check this information, Can you please send me your exported file to support @ ninjatrader dot com and reference this forum post and put ATTN: Brett in the subject line.

I look forward to assisting you further.

[NinjaTrader_Brett]

Hello,

Its

2) NT assigns a timestamp based on the selected timezone when importing. Then I should see different dates and times depending on the timezone setting at import.


It will by default set a time stamp of 24:00. However is I for example add the offset based on the timezone selected. Based on EST.

Therefor, if I set MTN time zone which is two hours - EST. I get a time stamp of 22:00.

Therefor you need to adjust your Import time zone to match the time of your other data.

Let me know if I can be of further assistance.

[FireFly]

Hello,

Its

2) NT assigns a timestamp based on the selected timezone when importing. Then I should see different dates and times depending on the timezone setting at import.


It will by default set a time stamp of 24:00. However is I for example add the offset based on the timezone selected. Based on EST.

Therefor, if I set MTN time zone which is two hours - EST. I get a time stamp of 22:00.

Therefor you need to adjust your Import time zone to match the time of your other data.

Let me know if I can be of further assistance.

Interesting, because I don't see that happening! The time never changes regardless of the timezone that I select for import. However the DATE will change at some point. How can the date change but not the time?

Just to be clear: you actually DO see a different time when you change the timezone while IMPORTING data? (as opposed to changing the timezone of your PC).

[NinjaTrader_Brett]

Hello,

In my testing a had selected the same time zone as my PC. This duplicated these results.

Its seems it actually adds at 12:00 Time stamp that is offset by your PC clock. Not your selected timezone of the improted data. This is specific to daily data only.

Therefor you would need to change your local PC time stamp. Restart NinjaTrader and then import this data in.

Let me know if I can be of further assistance.

[FireFly]

What you mention here is a WORKAROUND, but I still consider this a bug. If NT is ignoring the timezone setting of the Historical Data Manager, it should ignore it COMPLETELY and NEVER make a shift in the DATE! Now it is actually doing a conversion at import (and it should), but is is doing it incorrectly.

Here is how it is supposed to work:
NT doesn't find a time in the time in the file so it will set it at 00:00. But when I tell the Historical datamanager that the data originates from a +7 hrs timezone, it should adjust this timestamp to 17:00.

The current bug makes it impossible to EVER align two EOD series properly without doing workarounds like messing with the Windows timezone settings. You should never have to do that. And NT has a built in capability to take care of that (in the historical data manager). I can see that because the historical datamanager will sometimes change the DATE (but never the time). So it IS actually trying to do a timestamp correction at import.

In other words, the functionality is there but a bug prevents it from working properly!

I suggest you discuss this with one of your collegues to check if he agrees that this is a bug of not. In my opinion it most definately is!

Thank you for your support.

[NinjaTrader_Brett]

Hello,

I understand your concearn, However the question I asked earlier is why are you exporting the data and then importing it back in?

Thank You.

[FireFly]

Hello,

I understand your concearn, However the question I asked earlier is why are you exporting the data and then importing it back in?

Thank You.

To test if the fact that imported data is not aligned is indeed a bug or somehow caused by the input data. By exporting the same data and then importing it back in using different timezone settings in the Historic Data Manager I was able to confirm that it is indeed a bug.

If things work properly in NT, I should always be able to find a setting of the timezone that makes the data align properly. This is unfortunately not the case.

[NinjaTrader_Brett]

Hello,

Ok, I will check into this to see if there is something that can be done.

You will not get an update until Tuesday on this as this is when development will be back in.

Let me know if I can be of further assistance.