No historical daily data from forex.com?

[henry.omd]

Hi, I doubt this is an indicator program issue...
I tried to run backtest against the free forex.com connection you guys provide, but I don't seem to be getting any data hence no chart. Strangely, the same indicator works just fine when I use intra-day bars.

Is this a limitation of the free forex.com account (i.e. something fixable when I get a real account at forex.com) or the interface issue with forex.com (something that can't be fixed even if I get a real account)?

Please advise.

[NinjaTrader_Josh]

Right. forex.com does not have historical daily data. http://www.ninjatrader-support.com/H...ricalData.html

[henry.omd]

Thanks for the clarification Josh. I see from the documentation (Historical Data --> Overview) that forex.com does not provide daily historical data, but that NT can store RT data into its own historical data as long as I continue to have market analyzer up. This sounds promising, but I have some concerns:
* What is the granularity of the historical data? I don't need tick-by-tick historical data. I guess ideally, it should only be as fine as the minimum resolution required by the columns of the Market Analyzer, which for me is daily. If I switch over from daily to 30 minute granularity at some point (or vice versa) do you do anything to the stored historical data?
* my home PC is subject to outtage (especially during the summer). Will I have gaps?
* How do I monitor the size growth of the historical database and back up this historical data? Can it be on a separate drive (maybe even a USB drive)?

Thanks much for your help!

[NinjaTrader_Josh]

- The historical data will be recorded tick-by-tick. Your higher time framed charts will be populated accordingly with the tick-by-tick data.
- If you have an outage then you will most likely have gaps too.
- You can see the size of your database by going into My Documents\NinjaTrader 6\db and looking at the NinjaTrader.mdb file. If you want to backup your database, backup that file.

[henry.omd]

Thanks Josh, I just need 1 more clarification:
Let's say for the purpose of the example that I am only interested in 1 day bar for some currency pairs. The way to use NT to capture the historical data from forex.com then is to:

1. Dedicate a machine that will ALWAYS run Market Analyzer for those instruments.
2. Periodically, while the NT program is running, I would need to copy My Documents\NinjaTrader 6\db\NinjaTrader.mdb to another drive to back it up
3. I can then overwrite this file to another NT machine to look at the historical forex.com data.

Does this sound about right? I am actually concerned about 2 things:

Problem 1: NT is going to be writing to NinjaTrader.mdb while I try to copy the file. Isn't there some kind of concurrency issue?

Problem 2: Since I can't tell NT to flush the file in day granularity and NT is writing to the DB in a tick-by-tick manner, this NinjaTrader.mdb is going to quickly grow to unmanageable size. If I wanted to watch all currency pairs, I would expect it to grow to > 1 GB in a matter of few months, and > TB in a matter of few years. How am I going to keep copying such a big file, even if I had a big enough HD--which I don't. Since NT doesn't even give me an option to specify the location of the database, it's currently writing to a 20 GB disk, where Windows is installed...

Am I in the right ballpark?

[NinjaTrader_Dierk]

I suggest going with NT6.5 and ONLY copy data by using the new backup/restore concept (since this takes care of consistency issues).

[henry.omd]

Thank you for the suggestion Dierk,
When I try to back up (File --> Utilities --> Backup) in NT 6.5, a modal dialog pops up: "You need to close any account connection in order to run a backup."

While I understand why this *may* be necessary, this will cause me to miss some ticks. As I was asking you earlier, the use case is to have a PC that runs the MA ALL THE TIME to capture every tick from forex.com, so the current NT behavior does not fit the use case.

All these problems because forex.com doesn't support daily historical data, sigh... Do you think I am missing something?

[NinjaTrader_Dierk]

Your concept will not work for that reason: DB could be inconsistent while connected. Sorry..

[henry.omd]

Hi Dierk, thanks for the reply. Supposing that I use NT to manage historical tick capture, how can I purge old ticks (say more than 1 year old) from the DB? Since NinjaTrader.mdb is a binary file, I can't really edit it on my own... Historical Export function seems promising, and it even has an option to set the bar type. If I export say EURUSD from 1/1/2007 to 12/31/2007 using day bar, does that mean that I can create a file that has only a day bar for the EURUSD? I would be grateful if you can confirm.

-Henry

[NinjaTrader_Josh]

Yes. Export the file then delete NinjaTrader.mdb and reimport the file you just exported. You will be left with only the data you reimported.

[henry.omd]

Thanks Josh. I didn't know you would take it that far, because I wasn't sure whether it is safe to delete the NinjaTrader.mdb file, whether NT is running or not (well, especially if it's running). So are you saying that it's safe to delete the NinjaTrader.mdb file in both cases (NT running or not)?

-Henry

[NinjaTrader_Josh]

Deleting NinjaTrader.mdb is not officially supported. When you do so you will lose a lot of things besides your data. If you want to dab around with this please backup your files first. You will probably want to do this when NinjaTrader is closed also.

Also, curious as to why you want to do this operation? Couldn't you just tell the chart not to go that far back?

[NinjaTrader_Ray]

Next NT 6.5 release will include an option for deleting historical data per instrument from the database.

[henry.omd]

Hi Josh, I just tried exporting the DB to create daily historical bar from the gain demo feed which does not have the daily granularity. Unfortunately, Export Historical Data window pops up "No historical data found" when I attempt to export the data series with a bar type Day. The minute bar type works fine, but this only defeats the original purpose of the export: to get daily historical bar. It would appear then that there is no way to get historical daily bar in NT unless one signs up for the eSignal feed, which I am not yet willing to do.

Do I have it just about right?

[NinjaTrader_Josh]

Right. Unfortunately if you do not have daily bar data then there is simply nothing the export function can do.