Newbe questions: Historical data and formats

[r44chopper]

I would like to get historical EOD data for all the "common" Futures for as far back in time as I can (the beginning).

I think I understand (?) that I can subscribe to one of many services, who for $60/month for CBOT, another $60 fro CME, etc etc etc, will let me see their data.

1) If I do, can I save the data they supply, and unsubscribe after a month (I don't need monthly/weekly/daily updates to that database)

2) Do they supply the data in the form of "continuous contracts"

3) Whose data would be the best, most reliable, most compatible, etc.

4) These folks (www.premiumdata.net) want me to ask if Ninja
" if they would consider supporting the de-facto industry standard MetaStock format."

Since then I can buy their database, which looks a) huge and b) just what I want, and c) it's only $65
It just that their data format isn't NT compatible (I think).


and finally
5) Someone else out there has solved this problem. What else am I missing? Is there some better way to crack this nut?

Thanks for all/any answers to the above queries,

R44chopper

[NinjaTrader_Ray]

1) Yes
2) You would need to ask the data vendor
3) We support many options for real-time/historical data such as eSignal, IQFeed, OpenTick. As for historical data only vendors, there are quite a few and I personally do not have much experience with them. Google Maddock Data, they have some and is also a customer ours and I believe can provide data formatted for NinjaTrader already.
4) I will add this to our list as a potential future enhancement

[r44chopper]

Thanks Ray,

FYI, if you send a query to techsupport@maddockdata.com
you will get a reply from premiumdata.net, the folks I was referring to.
Which means that Maddock does not know about Ninja, and their data is not compatible.

I am surprised that Ninja isn't offering the data as part of the package. We all need it.

Is there someone else there (anywhere) who does have experience with historical data in a format compatible with Ninja, at a reasonable price?
Thanks again for the help,
R44chopper

[NinjaTrader_Ray]

I will alert the owner of Maddock of this thread.

[r44chopper]

Hi Ray,
I have more info.

I sent this to them:
My original email was to techsupport@maddockdata.com, and your reply was from support@premiumdata.net
I'm confused about the connection between these two sites.

I got this back:

Maddock Data is a reseller of our products – sorry I didn’t pick this up earlier. There is no functional difference between the products that Maddock sell and we sell, other than you’re dealing with a reseller rather than the source.


Thanks again,
R44chopper

[Moog]

Hello. Ray asked me to make a post.

I do resell data from Premium data. I do also use Ninja trader and do trade. If you email tech support it will go straight to premium data tech support, you can email me at sales@maddockdata.com. Before you ask, this is a throwback to say 3-4 years ago when I was teaching high net worth individuals how to trade and backtest their strategies. Long story but I did give data to my students and it sort of continued on from there. The chaps at premium data are very nice people and can answer technical issues. To be honest I never really had any questions about the technical issues as the software and updates tends to work. Never had a complaint either come to think of it. Anyhow Im waffling, as to the question of data converter I did write one in C# a while back when I was looking into Ninja script. it just converts the data for you from ASCII.

Cheers

[r44chopper]

For those of you considering using IQFEED for historical data for Futures:
Don't bother.

Despite what they say as you are wasting hours getting set up for a 7 day trial, they only have historical data for a year or two.

I would advise looking elsewhere.

R44

[zoltran]

For those of you considering using IQFEED for historical data for Futures:
Don't bother.

Despite what they say as you are wasting hours getting set up for a 7 day trial, they only have historical data for a year or two.

I would advise looking elsewhere.

R44
Sorry about your luck. I was up and running very quickly with iqfeed and find it more than adequate.
Maybe not as refined as esignal is, but certainly the best value out there by far.

Re Historical data .. They only advertise 8 months intraday for stocks, and 18months for some eminis... and 8 calendar days for tick data.
My experience is that they provide this and more. (although they are currently short on tick history due to the exceptional volatility out there..)

What other providers have you found that have clean historical data for longer than this?
Esignal is about the same with 240 trading days for intra-day 'minute data' and only a few days for tick history ...

As for the forms ... again.. pls advise us of the provider that does not make you go thru a number of exchange forms for a Real Time feed? I've yet to see one.

[r44chopper]

ON this page:
http://www.iqfeed.net/index.cfm?displayaction=data&section=services

You will find:
"Daily, Weekly and Monthly Historical data (10+ years)"

Which is the part I was taking issue with. They do not.

[zoltran]

Sure they do ... I just picked up an ibm chart back to 1996. And a monthly ES chart back to 1997

Of course, you have to use the continuous contract format for futures ..
You can't get a long history for the current contract, as it's only exists for 3 months . :rolleyes:
So either your chart program or provider needs to splice all the individual contracts together to make a continuous contract.
NT does not do this. IQfeed, Esig, Opentick do.
Search the forum for some good discussion on this topic.

Try adding an instrument to NT instrument manager using ##.## for the expiry date.

Works like a charm.

[r44chopper]

Maybe you could try corn or oats and let me know what you get?

I did call them, and they said that they did not have data going back further than 1-2 years in most futures.

PM me for further discussion, as I suspect this thread is dead.....

[zoltran]

Yes, Corn/Oats back to 2001, bonds back to 2002 .. so some are better than others.

[svuhmed]

I was recently told by qcollector (www.mechtrading.com) that they are in the process of producing a version for esignal (and for iqfeed in months) that will export data in the ninjatrader format. I'm awaiting a trial. Great if it works.

[rcsingleton]

Hello.

I want to import historical daily Forex data into NT. I have the option of purchasing the data either in Metastock or ASCII format. Does NT support the Metastock format, or only text ?

Regards,
R. C.

[NinjaTrader_Dierk]

Unfortunately NT only supports the text format documented here: http://www.ninjatrader-support.com/HelpGuideV6/Import.html

[rcsingleton]

Hello.

I've taken a look the document describing the format required for importing daily data into NT. I've got a question about the first line of the file. I've got a file, 'AUDCAD.csv' which contains historical daily data. But the first line of the file contains the header, 'Date,Open,High,Low,Close'.

Does that line need to have its commas replaced with semi-colons to look like this ==> 'Date;Open;High;Low:Close' ?

Or does it matter ?

Regards,
R. C.

[NinjaTrader_Dierk]

Only the format documented is supported -> no header lines, no commas

[8DTK8]

Update 5 min later.
Sorry. It was just slow loading up.
Your advice worked great.
Thanks. :D


---

I want a daily chart as far back as it will go. I just signed up with IQFeed and after a few errors got it to connect. I'm having trouble getting the continuous contract. I tried using the ##-## expiry date. It just hangs and my Windows Task Manager and IQFeed tell me that there's nothing going on.

Thanks in advance.




Sure they do ... I just picked up an ibm chart back to 1996. And a monthly ES chart back to 1997

Of course, you have to use the continuous contract format for futures ..
You can't get a long history for the current contract, as it's only exists for 3 months . :rolleyes:
So either your chart program or provider needs to splice all the individual contracts together to make a continuous contract.
NT does not do this. IQfeed, Esig, Opentick do.
Search the forum for some good discussion on this topic.

Try adding an instrument to NT instrument manager using ##.## for the expiry date.

Works like a charm.

[8DTK8]

Although it functionally works, it's taking a long time. After my success with loading 6 months of daily data (which took a decently long time 5-10 minutes), I got brave and thought that I would try the entire ES history from 1997. It's been 3 hours (I've been able to walk to the gym, work out, come back, cook dinner, eat dinner and do the dishes) and it's still going.

Task manager says that everything is still running and nothing has crashed and IQ Connection Manager still registers as being on (even though it's between .00 - .04 KB/sec). This is what it looked like when I loaded the 6 months of data so I don't know whether to let it run or force it to quit.

Is this because I'm still on the 7 day trial or is this what I can expect once I start paying too?

Thanks.





Update 5 min later.
Sorry. It was just slow loading up.
Your advice worked great.
Thanks. :D


---

I want a daily chart as far back as it will go. I just signed up with IQFeed and after a few errors got it to connect. I'm having trouble getting the continuous contract. I tried using the ##-## expiry date. It just hangs and my Windows Task Manager and IQFeed tell me that there's nothing going on.

Thanks in advance.

[billitin]

IQFeed works fine for me. Pulled up ES daily chart for the last 10 years in 10 seconds, and intraday chart for the last 3 years in 2 minutes.

[NinjaTrader_Dierk]

@stefij (http://www.ninjatrader-support2.com/vb/member.php?u=8213): I deleted your post since it referred to competitive products. Also: please do no submit posts redundantly in different threads. Thanks

[everington_f]

Hello. Ray asked me to make a post.

I do resell data from Premium data. I do also use Ninja trader and do trade. If you email tech support it will go straight to premium data tech support, you can email me at sales@maddockdata.com. Before you ask, this is a throwback to say 3-4 years ago when I was teaching high net worth individuals how to trade and backtest their strategies. Long story but I did give data to my students and it sort of continued on from there. The chaps at premium data are very nice people and can answer technical issues. To be honest I never really had any questions about the technical issues as the software and updates tends to work. Never had a complaint either come to think of it. Anyhow Im waffling, as to the question of data converter I did write one in C# a while back when I was looking into Ninja script. it just converts the data for you from ASCII.

Cheers


Do you still have that data converter for ASCII to a NT friendly format?

Regards

[saltminer]

On this topic, if I purchase and import 20 years of stocks data, and then connect to say Kinetick with a 5 year chart open, what will happen ?

A) Any data not in the cache will be requested from Kinetick to create a complete chart.(e.g. the last 3 days)

B) Data for the 5 years the chart wants will be requested from Kinetick and overwrite the previously imported data

C) Something else

Thanks,
saltminer

[abhi701]

Hello everyone ,

I thought that it would be better to observe sthg doubtful before posting to the forum, so i observed the data displayed on NT charts for 1 week and saw following anomalies:
Although data was available to ninja trader for upto 20th Jan only , on the chart it was showing the day as 21st jan
Also It was delaying the historical data for 1 day , like if i start downloading data say after 6PM IST , even then i would not get the latest data. So in effect historical data is delayed for 1 day.
I am using yahoo data feed for nse stocks. While writing this post one thing just clicked me that whether all this could be some time zone error ?

Posting the snaps for the above errors..

Thanks,
Abhishek

[NinjaTrader_Chip]

Hello abhi701

Welcome to the NinjaTrader Support Forums!

What time zone is your PC clock currently set to?

What is one of the instruments you are using when you experience this issue?

[abhi701]

Hello abhi701

Welcome to the NinjaTrader Support Forums!

What time zone is your PC clock currently set to?

What is one of the instruments you are using when you experience this issue?

Hi Chip my PC is been set to (UTC+05:30) Chennai, Kolkata, Mumbai, New Delhi.
It happens on stocks for Indian markets in NSE , like HDIL ,DLF.

Thanks,
Abhishek

[NinjaTrader_Brett]

Hello,

This is the reason this occurs.

It also occurs due to the fact that your using a Default 24/7 session template most likely.

Please try using a Regular Trading Hours or Extended trading hours session template for the chart under right click->data series menu.

What occurs is the time stamp of the daily bar is being midified for EST. Since 12:00 AM bar stamp from your broker is being modified for EST time it is move the time for enough that it is displaying the next day.

Let me know if I can be of further assistance.

[murthymsr]

Hello everyone ,
.............
Although data was available to ninja trader for upto 20th Jan only , on the chart it was showing the day as 21st jan
.........
Abhishek

Hi abhi,

The link below discussed this problem and presented the solution.

http://www.ninjatrader.com/support/forum/showthread.php?t=31877&highlight=Daily+data+displacement:

[abhi701]

Hello,

This is the reason this occurs.

It also occurs due to the fact that your using a Default 24/7 session template most likely.

Please try using a Regular Trading Hours or Extended trading hours session template for the chart under right click->data series menu.

What occurs is the time stamp of the daily bar is being midified for EST. Since 12:00 AM bar stamp from your broker is being modified for EST time it is move the time for enough that it is displaying the next day.

Let me know if I can be of further assistance.


Hi Brett & (http://www.ninjatrader.com/support/forum/member.php?u=16392)murthymsr , thanks for your help and i did as told by you , but was successful only to an extent :( , To clear the doubts i am attaching screenshots for the same. (http://www.ninjatrader.com/support/forum/member.php?u=20759)
Data is till 21st only: http://www.ninjatrader.com/support/forum/attachment.php?attachmentid=16413&d=1295931668


25th data doesnt exists in history but is displayed: http://www.ninjatrader.com/support/forum/attachment.php?attachmentid=16414&d=1295931668

Indicators are drawn upto 2nd last value only: http://www.ninjatrader.com/support/forum/attachment.php?attachmentid=16415&d=1295931686

Data in Data Manager: http://www.ninjatrader.com/support/forum/attachment.php?attachmentid=16416&d=1295931686

Even on 25th i am not getting data for 21st: http://www.ninjatrader.com/support/forum/attachment.php?attachmentid=16417&d=1295931699

I will be greatly obliged if u cud further help me in this problem..
Thanks.

[NinjaTrader_Brett]

Hello,

In this case the session template you are using is still applying too much of an offset. Please go into the session manager by control center->tools->Session manager.

Then edit the session template you will be using, change the this time zone is at the bottom their to your own time zone or a time zone that if farther to the east. Then try the chart again.

As far as the indicators. If you want them to display to the current bar you will need to right click->indicators on the chart and then turn Calculate On Bar Close = False.

Let me know if I can be of further assistance.

[abhi701]

Thanks Brett , i am really thankful to u , what seemed to me like major problem of missing last bar indicator u provided a very simple solution :) , the session thing u mentioned i did it as mentioned by murthimsr , but it didnt seemed to work. I think it may take some more time to get adapted to ninjatrader and then will bother u with my queries.
But i must admit NinjaTrader is one of the best although i have ever seen and really thank Ninja developers for developing such a wonderful tool. Although only issue is that of datafeed options are not very reliable.Hats of to u guys.

Thanks,
Abhishek , India

[NinjaTrader_Brett]

Thank You!

Also, please check out www.kinetick.com (http://www.kinetick.com) for a data feed option that is very reliable with the adapter build directly into NinjaTrader.

Let me know if I can be of further assistance.

[abhi701]

Brett my 1 more problem got solved :) ,in historical data manager when i deleted the data and downloaded it again with new session template , then chart was displaying correct dates.
Secondly does kinetick offers datafeed for NSE/India.I checked out there options but NSE wasnt there. And is it paid or free. I only require EOD data.And are there any other datafeeds available for NSE except yahoo and google.
Thanks..

[NinjaTrader_Brett]

Excellent.

Also, Kinetick does not support NSE unfortunately and I cannot think of any others that are free and support NSE Unfortunately.

Let me know if I can be of further assistance.

[abhi701]

Hi Brett i am back at u again :)

Firstly, As shown in the image , the LHS drop down menu on charts shows only the default instrument list , can this behavior be changed to show other instruments list as required. Just as an example suppose i trade in 2 types of products stocks and commodities , i add stocks in default and commodities in other instrument list , so if i am analyising commodities then i would like to see all commodities in dropdown.

Secondly how can Ninjatrader be used to trade in indian stocks using "Chart Trader", is NinjaTrader also a broker? Do i need to open a paid demat account with Ninja for this purpose..

Thanks,
Abhishek, India

[NinjaTrader_Brett]

Hello,

No problem.

For your first question only the default list can be used here Unfortunaley.

Also, For your second question NinjaTrader is the trading application only and not a broker.

I believe the only broker that we have that may support this is Interative Brokers. Please check them out on their website and then use the below connection guide to connect to them with NinjaTrader.

http://www.ninjatrader.com/ConnectionGuides/Interactive-Brokers-Connection-Guide.php

[abhi701]

Hi Brett , how can i backtest a particular security for certain period of time , like in NT i cud go only back upto 1 year using daily charts but cud get back further through weekly charts , but can i analyse past say 3 years back using daily charts.

Thanks,
Abhishek

[NinjaTrader_Brett]

Hello,

Thanks for the note.

Please increase the number of days to load in your daily chart.

Right click on the chart and select Format Data Series.

Then change the setting for Days To Load.

You will only be able to download what daily data is provided from your data feed provider.

Also, Weekly charts are formed with Daily data. Therefor you should be able to pull up daily charts with the same amount of data going back that you can in a weekly chart.

[abhi701]

Hello,

Thanks for the note.

Please increase the number of days to load in your daily chart.

Right click on the chart and select Format Data Series.

Then change the setting for Days To Load.

You will only be able to download what daily data is provided from your data feed provider.

Also, Weekly charts are formed with Daily data. Therefor you should be able to pull up daily charts with the same amount of data going back that you can in a weekly chart.


Simply awesome , thanks brett once again. Since i am quite new to tech analysis so i like to observe past data to see how indicators worked in past.

Thanks,
Abhishek

[milos.adamec]

Hello,
i read thru this thread but havent find an answer.

I need to load historical data for say MAR2005 corn contract (from first to a last trading day - not a merged data)
I need to do the same with more comodities and months.
My way is to define an unmerged instrument linked to a particular contract month.
Unfortunatelly , with a instrument editor i can only define contract months back to beginning of 2010. (i cannot use ##-## as it does the merge)

Can somebody advise how to do that? How to define contract month for example 03-2005

Thanks
Milos

[NinjaTrader_Kyle]

Hello Milos,

In order to access past expiries, you'll need to set your PC's date back to when that contract would have been valid. In your case, change the year back to 2005, then restart NinjaTrader and it will be possible to add this older expiry.

[obama6493]

Thank You!

Also, please check out for a data feed option that is very reliable with the adapter build directly into NinjaTrader.

Let me know if I can be of further assistance.
Thanks Brett , i am really thankful to u , what seemed to me like major problem of missing last bar indicator u provided a very simple solution http://www.ninjatrader.com/support/forum/images/smilies/smile.png , the session thing u mentioned i did it as mentioned by murthimsr , but it didnt seemed to work. I think it may take some more time to get adapted to ninjatrader and then will bother u with my queries.
But i must admit NinjaTrader is one of the best although i have ever seen and really thank Ninja developers for developing such a wonderful tool. Although only issue is that of datafeed options are not very reliable.Hats of to u guys.

_____________
Watch TV Online (http://www.tvonlinewatch.com/)

[NinjaTrader_Brett]

Thanks Obama6493!

Hope you had a good weekend!

[abhi701]

Hi Brett , how can i get the yahoo data adjusted for splits? Every 2nd stock has gone through splits in the past one year so i m not able to backtest with indicators properly. Plz provide a solution brett..

Thanks,
Abhi

[NinjaTrader_Brett]

Hello,

Please goto Control Center->Tools->Instrument manager then edit the stock and then goto the Misc tab and then click the update Splits and Dividends button while you are connected to Yahoo.

Let me know if I can be of further assistance.

[abhi701]

Hello,

Please goto Control Center->Tools->Instrument manager then edit the stock and then goto the Misc tab and then click the update Splits and Dividends button while you are connected to Yahoo.

Let me know if I can be of further assistance.

Thanks Brett , whenever i do an update as told by u , its just hangs up. I tried it on atleast 7 stocks and waited for around 15 minutes on 3 of them but with no success , or is that it requires longer than this for splits details to get updated..

Thanks,
Abhi

[NinjaTrader_Brett]

Hello,

Please insure your master symbol is the correct symbol for yahoo as this it was it does a split and dividant download. If the symbol is incorrect you may experience this.

Also the update of splits and dividends should only take less then 30 seconds.

Let me know if I can be of further assistance.

[abhi701]

Hi Brett, i ensured that everything was right and now i have upgraded to RC4 and was able to solve my problem.So i think it must have been a bug with RC2.
But now there is one more thing, In RC4 as we scroll through the chart with "Global Crosshair" only weekly dates are shown unlike earlier version which showed every bar date. So even if i am on say 6-03-2011 it will continue to show 4-03-2011. Do we have a workaround to change this setting?

Thanks,
Abhi

[NinjaTrader_Brett]

Hello,

Glad to hear your up and running.

On your secondary question.

Can you give me an example of both time frame charts you have selected so that I can test this on my side.


I look forward to assisting you further.

[abhi701]

Hello,

Glad to hear your up and running.

On your secondary question.

Can you give me an example of both time frame charts you have selected so that I can test this on my side.


I look forward to assisting you further.

Hi Brett plz find the attached images:

As u can see even 4 bars later "global crosshair " is showing me the same weekly date it displayed me before.

Thanks,
Abhi

[NinjaTrader_Brett]

Hello,

Can you please send a note to support at ninjatrader dot com and reference this forum post and put ATTN: Brett in subject line then include the following attachment.

My Documents->NinjaTrader 7->Workspaces->Your Workspace Name This occurs on.xml

This way I can test with your exact settings on my side as I would not expect this.

I look forward to assisting you further.

[abhi701]

Hello,

Can you please send a note to support at ninjatrader dot com and reference this forum post and put ATTN: Brett in subject line then include the following attachment.

My Documents->NinjaTrader 7->Workspaces->Your Workspace Name This occurs on.xml

This way I can test with your exact settings on my side as I would not expect this.

I look forward to assisting you further.

Well thanks Brett. And once again i was able to find out my own stupid mistake and in the process managed to trouble u too. But i think the greater mistakes one makes , more quickly does he learn things :) . I am sorry for troubling u again and again with those trivial issues.
Now what i did wrong was add a session template named NSE , with start day and end day both in the same row instead of making 5 different rows for each day, and hence it was giving me only 1 date per week.

Thanks,
Abhi

[abhi701]

Hi Brett, i have around 45 securities added to my default instrument. So if say sometimes after 5 days i want to analyse all of them then i have to download historical data for each one of them. The reloadall button provided in reload is not working for me as it is designed to be. Am i missing sthg. Please help me with this.

Thanks

[NinjaTrader_Brett]

Hello,

Hope all is well.

Try left clicking on the market analyzer to select it, then press the F5 key on your keyboard. This should reload all historical data and recalculate all the indicators for all the stocks listed in the market analyzer.

Let me know if I can be of further assistance.

[abhi701]

Hello,

Hope all is well.

Try left clicking on the market analyzer to select it, then press the F5 key on your keyboard. This should reload all historical data and recalculate all the indicators for all the stocks listed in the market analyzer.

Let me know if I can be of further assistance.

Hey Brett Thank you very much , i was down with typhoid so was not active on my laptop and hence such a delayed thanks. I tried it out and it worked like wonders for me.And i just thought how come i not care to ask you earlier. Thanks once again Brett.

[abhi701]

Hi Brett , how r u. Well i dont know if after 4-5 months of experience with Ninja Trader platform am i still a newbie and shud i be posting here :) . But i feel i have always a got perfect reply from u whenever i had some difficulty so i still feel this is the best thread for me to post my doubts.
Brett is there a way i can import EOD data for say 50 stocks at once from a text file in which i will manually edit the entries like open ,high , low,close. Right now if i want the latest data either i have to manually edit each symbol data in "HIstorical manager" or 1st i have to export the symbol text file , then put data into it and then import it back.

Thanks,
Abhishek

[NinjaTrader_Brett]

Hello,

Thanks for the note.

Unfortunately this is not possible to do all at once. Would require one text file for each stock and you would have to do it manually for each stock at a time.

http://www.ninjatrader.com/support/helpGuides/nt7/importing.htm

Let me know if I can be of further assistance.

[abhi701]

Hello,

Thanks for the note.

Unfortunately this is not possible to do all at once. Would require one text file for each stock and you would have to do it manually for each stock at a time.

http://www.ninjatrader.com/support/helpGuides/nt7/importing.htm

Let me know if I can be of further assistance.

Thanks Brett. But i wud like to request u developers to think abt including this functionality in ur next versions, bcoz this will ease up the process to a large extent.

[NinjaTrader_Kyle]

Hi abhi701,

Thanks for the suggestion - I will forward this on to development for further consideration.