Historical Data for Backtesting

[gunsnmoney]

Do not use the Minute data from disktrading.com. Instead there is a "build minute bars' option when importing the split tick files into Ninja. Minute bars are the smallest increment NT can display on charts so this will display minute bars. This worked for me. I don't know why this isn't the default.

G&M

[qewcool]

Do not use the Minute data from disktrading.com. Instead there is a "build minute bars' option when importing the split tick files into Ninja. Minute bars are the smallest increment NT can display on charts so this will display minute bars. This worked for me. I don't know why this isn't the default.

G&M

thats pretty much the only way to get volume as well since their minute data dont include volume info
Im looking at the samples they provide and volume is all equal to 0 in the minute data.

For tick data, anybody knows why there are some entries with 0 volume ? 0 volume would not be considered a tick right or am i missing something ?

[qewcool]

I have a different issue with the disktrading historical tick data. I was able to upload the split files (10 of them) and they imported all successfully. But I can't seem to generate a tick chart with the data at all.

I resolved the minute data issue by just importing the minute data instead of trying to build from the tick data.

Is there some special trick to be able to use the tick historical data?

How big is the size of each file ?
Where did you get the 10 split files ? Is it how they provided it to you ?

or did you split them yourself ? if done yourself, can you tell me how you did it (which software) ?


They tell me it is only 1 continuous file which scares me a bit on how I am going to be able to use it since looking at their sample for ES for example, one week of tick data is about 60Mb.

That comes out to 2.8Gb per year and 28Gb for 10 years ? 28 Gb in one file ?

How the hell do you manipulate (open, copy/paste etc) a file of 28Gb ??
I can use Ultraedit but even with that software, I doubt you can easily open a file of 28Gb ?

Now if they split the tick data into 10 files it would be relatively easier to manipulate although files of 2.8Gb is not that easy to open either.

Anyone can share their experience before I buy data from them ? Thanks in advance.

[gunsnmoney]

disktrading.com provides a splitter program for historical tick data. Run the splitter and it splits the tick file into bite sized pieces. Inside it is split by date. Then you place one of the split files in the NT Import folder and rename as the current contract name in the NT Instrument Manager (like NQ 03-10). Then Import tick data as minute bars as NT charts will not display charts from tick data, they have to build bars first. Then delete that file from the Import folder and place the next split file in the Import folder and again rename as above - use the same name as used before. The data is from a different date set so NT will just add it to the existing data.

You can open the files with wordpad but I have not tried this with tick data. Have had to do it with their Day data because there were some errors.

For the price I've found this supplier to be worth it. Iavor has been very helpful in helping me sort this out having never dealt with historical data before.

I too am concerned about it being a continuous contract as now my back tests are being run across the rollover periods. The only other alternative would be to add each contract back to the beginning of the tick data, in NQs case 2000, which would be about 40 contracts (10yrs x 4/contracts per yr) and then opening and editing, saving each tick file to have only data from that contract. Big project and someday I may do that if my system gets less than acceptable results. disktrading does provide an excel file detailing the rollover dates.

One other tip: If you import tick data for a certain contract and then that contract rolls to the new one as NT will let you know. NT will tell you to delete the old contract from the Instrument Manager and add the newer contract. If you do that your historical data will not be associated with the newer contract as it was added to NQ 03-10, not NQ 06-10. So I just left the old contract there (well added it back actually) and run my backtests using it.

[qewcool]

the only splitter i see for ninjatrader is one that split per number of lines ( it says 15 million lines).

Iavor told me he is working on a PerYear splitter for ninjatrader which should split the ninjatrader tick file into yearly files. It should be available soon.

[qewcool]

Anybody got the rollover dates file from disktrading.com
I am trying to find it but cant find it anywhere in the folder they gave me.

[mtthwbrnd]

beware of disktrading.com tick data...

I bought the FOREX tick data from disktrading.com. The timestamps only have a resolution of minutes.

To be clear, I'm not saying not to buy data from disktrading.com. They are very fast to respond to questions and have good customer service and I will be happy to purchase more stuff from them.

As far as I know their minute and other data is okay. I have only looked at some of their FOREX data. Just writing a heads up that their tick data, as they confirmed in email (below), has all ticks in the same minute bucket marked with the same timestamp i.e. the resolution of the time stamps is minutes... that makes the data pretty much useless for most backtesting. I find this pretty extraordinary, and do not know the rationale/reason for that. None was offerred.

to "disktrading.com" <[email protected]>
date 17 April 2010 11:01

Hi,

The timestamps on the "tick data" is only to a resolution of 1 minute. I.e. all ticks during the same minute appear to have the same timestamp, quoted to the minute. In that format it is not useful for anything faster than 1 minute.

Do you have any tick data with the correct timestamps, to the second?

Thanks,
Matthew.



from "disktrading.com" <[email protected]>
date 17 April 2010 11:07

Hi Matthew,

We don't have seconds in the timestamps, but these are still ticks.

Thanks!
Iavor




to "disktrading.com" <[email protected]>
date 17 April 2010 11:16

Yes, they are ticks - but without the timestamps they are not useful for testing sub minute strategies, only for know the sub minute statistical distrubution of the ticks. The temporal ordering has been lost.

I'm curious, how is it that you have the tick data but not the timestamps?

Are you looking into getting the tick data with the time stamps? If you do, then email me, I would be interested in purchasing.


Thanks,
Matthew.



from "disktrading.com" <[email protected]>
date 18 April 2010 11:10

Hi Matthew,

Not sure if we will be adding seconds any time soon, will let you know in case we do.

Thanks!
Iavor

[adamus]

Matt,
why do you want the seconds? The ticks will be in consecutive order, and that's good enough for me. I have never even thought about trying to access the seconds for the ticks.

OK if you are an HFT outfit then I can see you might want the tick seconds for backtesting intra-minute trades, but if you are using NinjaTrader, it's unlikely you're doing HFT.

[mtthwbrnd]

Matt,
why do you want the seconds? The ticks will be in consecutive order, and that's good enough for me. I have never even thought about trying to access the seconds for the ticks.

OK if you are an HFT outfit then I can see you might want the tick seconds for backtesting intra-minute trades, but if you are using NinjaTrader, it's unlikely you're doing HFT.

How do you know the ticks are in the correct order?

What possible rationale is there for selling the data for each tick, but not marking them with the correct time stamps? How can that situation even arise, where one does not have the time stamps but does have the ticks?

For me it is an indication that there might be other issues with the data. It is a red flag.

[giacomo]

How do you know the ticks are in the correct order?

What possible rationale is there for selling the data for each tick, but not marking them with the correct time stamps? How can that situation even arise, where one does not have the time stamps but does have the ticks?

For me it is an indication that there might be other issues with the data. It is a red flag.

Just walk forward.

Many tick vendors will use the same time stamp for ticks within the same minute. So the 0945 time stamp will contain all ticks from 094401 to 094500. It is not a big deal.

The way you work with this data is by walking forward just like in real time trading. In real trading, you never know the future tick.

[mtthwbrnd]

Just walk forward.

Many tick vendors will use the same time stamp for ticks within the same minute. So the 0945 time stamp will contain all ticks from 094401 to 094500. It is not a big deal.

The way you work with this data is by walking forward just like in real time trading. In real trading, you never know the future tick.

I am not familiar with many tick vendors. Which ones are you referring to who supply data in that way?

I have got lots of data from GAIN and they do mark their data to the second. I note though that in the data files the ticks are not actually in temporal order, there are many glitches. So if the timestamps were not there, it would be incorrect to naively "walk forward" and assume that the ticks were in order.

Do you know why or how a vendor comes about data that does not have the correct timestamps? Anyhow, this is not really an active issue for me anymore because I am not using the data supplied by diskTrading.com.

[NinjaTrader_Chip]

Hello kocmodpom,

As with the import of other historical data into NinjaTrader, proper formatting is essential. Please see the following excerpt from our Help Guide regarding the import of Tick Data:

Tick Format
Each tick must be on its own line and fields must be separated by semicolon.

The format is:
yyyyMMdd HHmmss;price;volume

Sample data:
20061107 000431;1383.00;1
20061107 000456;1383.25;25
20061107 000456;1383.25;36
20061107 000537;1383.25;14

Please let me know if you are still unable to get this going.

[adamus]

It also works with the timestamp format

yyyyMMdd HHmm;price;volume

which is disktrading.com's chosen format.

Regarding GAIN and their ticks being out of sequence, that is pretty bizarre. Either it means that they lost the ordering at some point which is unlikely, since it would have been much more random and noticeable, or the timestamps were created on another server and the delay between the moment of creation and the point at which GAIN captured and stored the tick is just greater for those ticks. Or does GAIN have another explanation?

Assuming that I'm correct, and that GAIN received the ticks in the order presented and not in the order of timestamps, then the timestamps aren't going to be any use to you, since you would never see those ticks at the time of those timestamps, rather you would see them after a delay the same as GAIN does.

PS I realise it's academic for you but it might not be for me - I am using disktrading.com

[mtthwbrnd]

It also works with the timestamp format

yyyyMMdd HHmm;price;volume

which is disktrading.com's chosen format.

Regarding GAIN and their ticks being out of sequence, that is pretty bizarre. Either it means that they lost the ordering at some point which is unlikely, since it would have been much more random and noticeable, or the timestamps were created on another server and the delay between the moment of creation and the point at which GAIN captured and stored the tick is just greater for those ticks. Or does GAIN have another explanation?

Assuming that I'm correct, and that GAIN received the ticks in the order presented and not in the order of timestamps, then the timestamps aren't going to be any use to you, since you would never see those ticks at the time of those timestamps, rather you would see them after a delay the same as GAIN does.

PS I realise it's academic for you but it might not be for me - I am using disktrading.com

I do not think there is anything wrong with that gain data having some ticks out of order because they are all uniquely stamped. Most likely explanation is that the files were dumped from a database without sorting.

Are you using the disktrading.com (shall we call it DTC?) data for a sub minute system? I don't have anything against DTC, as I said before I am very happy with the other data I got and would not discourage people from buying from them. It would just be nice if they would mention the timestamp situation before the data is purchased. I hope that the ticks they supply are in the correct order, but I won't trust it until I know what the rationale is for not having the timestamps.

[adamus]

I have tried it in backtesting. Not in real trading. I found that using 150 tick bars in NT is sometimes quite useful.

Re: dumping from a database. Most databases, e.g. Oracle, mySQL, MS SQL Server, all have a natural ordering, and that is the insertion order. It could have been ordered differently deliberately but you would notice straight away. I think what you are seeing is the order that GAIN stored the ticks into their database.

Re: DTC, have you just asked them straight out? I guess you didn't and you now have no reason to.