NT7 memory consumption [Archive] - NinjaTrader Support Forum

NinjaTrader_Dierk

06-04-2009, 05:47 AM

Guys,

As you probably know we applied several performance and memory improvements to the NT7 code base. Our internal tests are more than promising. I would not want to provide actual numbers at that time, but the improvements are significant so far.

However, internal testing always will be limited. We wanted to run some reality checks by verifying against your "real world" scenarios (we are targeting around 10 different scenarios).

Could you please mail me to dierk [dot] droth [at] ninjatrader [dot] com your "typical" scenario where you experience memory issues with NT6.5?
Your mail should include:
- a reference to this thread
- a zip archive holding these folders:
* <my documents>/NinjaTrader6.5/workspaces
* <my documents>/NinjaTrader6.5/templates
- step by step instructions on how "to hit the wall"
It's crucial that you would provide the folder/files mentioned above in your mail.

Important: Please make sure your setup
- only would use NT standard indicators
- only would use NT standard strategies or simple as possible custom strategies (please include those as well)
- your setup rather would take minutes than hours to experience the issue in question
- would run on the main monitor and not require additional screens
This is to ensure we really focus on the subject.

I'm unsure on how much feedback we'll get. Please don't be disappointed if I would reject your input in case we would get way more than the targeted 10 setups.

Thanks in advance for your continued support
----------------------------------------------------------------------------------------------------------------------------------------
Results:
Setup1, many tick/volume/second based charts on high volume futures, NT startup and chart recovery:
NT6.5: blew up after >5min with OutOfMemory exception and 1.2GB RAM peak usage
NT7: finished after 75 secs, RAM usage always was below 100MB

Setup2, backtesting of a multi-series strategy on a basket of instruments (I created a similar setup, since user could not disclose all details):
NT6.5: RAM usage >1.3GB, blew up with OutOfMemory exception after >6mins
NT7: RAM usage < 430MB, done after 80 secs

Setup3, backtest of SampleMaCrossOver on 50 ticks ES series, 2009-03-15 through 2009-06-08:
NT6.5: RAM peak usage 1.2GB, 550MB after backtest complete, took > 4mins
NT7: RAM usage always < 110MB, backtest completed after 40secs

Setup4, export 1 year of ES continuous contract tick data
NT6.5: User reported was not possible and only could be done in chunks of 1-3 months
NT7: RAM usage < 500MB, took ~5mins

Setup5, backtest of SampleMaCrossOver on 50 ticks ES series, 2008-06-01 through 2009-06-09:
NT6.5: User reported was not possible due to OutOfMemory exception
NT7: RAM usage < 140MB, took < 3mins

designer

06-06-2009, 03:05 PM

When is NJ7 being released now?

NinjaTrader_Kyle

06-06-2009, 03:42 PM

Hello designer,

Thank you for your post.

NinjaTrader 7 is scheduled to release in beta by the end of the month.

ctrlbrk

06-06-2009, 03:53 PM

Dierk,

As you already know from our previous exchanges, I have hit this limitation very often. I have not "applied" to help by providing my strategies because 1) proprietary DLL's are involved, and 2) well, they are "my" strategies and I don't trust anyone and 3) [most importantly] the largeaddress patch has made them far less frequent and annoying.

But, if you don't get the desired response and really need some help, even though it would be some work for me to re-program some of the strategies, I will do it in the interest of helping the community (and myself) improve NT7. Let me know if it is required.

BTW, I applaud your post asking for help from the community. I hope to see more posts like this in the future, it would signal a change for the better, in my opinion.

Mike

NinjaTrader_Dierk

06-07-2009, 03:39 AM

@ctrlbrk, thanks for your feedback. I will come back to you in case.

NinjaTrader_Dierk

06-08-2009, 01:30 PM

Added results to original post

whitmark

06-08-2009, 05:53 PM

Gains

06-08-2009, 07:33 PM

Dierk,
Wow! This is fantastic news, thanks for sharing it.
Well done, I can hardly wait to take my own test drive.
Gains

NinjaTrader_Dierk

06-08-2009, 10:20 PM

Dierk,

That is promising news. Are you looking for chart configuration examples or strategy analyzer examples where the data load can be extensive? In the later case, are templates and workspaces needed?
I'm looking for both. Templates/workspaces would not be needed for strategy analyzer related issues.

In fact market analyzer setups would be interesting as well.

NinjaTrader_Dierk

06-09-2009, 05:52 AM

Add new results to original post

ChiTrader2000

06-09-2009, 06:03 AM

NinjaTrader_Dierk and NinjaTrader development team,

This is great news!

Thank you very much for dealing with this issue!

ChiTrader2000

whitmark

06-09-2009, 07:54 AM

Dierk, impressive results on Setup 3! For extra credit, would you consider loading up the current contract with a year of ES tick data from previous contracts and rerunning the test? The implication from your initial results on memory consumption is that simple strategies on single timeframes with minimal plots/dataseries may realize a tenfold+ improvement in backtesting periods. Someone pinch me :)

Thanks,

Whitmark

NinjaTrader_Dierk

06-09-2009, 07:59 AM

ctrlbrk

06-09-2009, 09:37 AM

Setup3, backtest of SampleMaCrossOver on 50 ticks series, 2009-03-15 through 2009-06-08:

NT6.5: RAM peak usage 1.2GB, 550MB after backtest complete, took > 4mins

NT7: RAM always usage < 110MB, backtest completed after 40secs

Happy Dance!

http://i41.tinypic.com/voykg7.gif

Mike

whitmark

06-09-2009, 10:15 AM

I will try if you could provide an import file (e.g. on some download area, you could try Help->Mail to support+attachment). I would not want to spend time right now to merge stuff together...

Also: the "ratio" on RAM efficiency gained certainly will vary dependent on actual setup. So please take it with a grain of salt.

Are there any eSignal users out there who have been keeping up their continuous ES ##.## symbol or others who maintain exported ES continuous contract data at the tick level for this stress test of NT7 memory consumption? If not, when I have more time, I'll have to splice what I can from backfill data from each separate contract. No need to backadjust for this excersize.

Thanks,

Whitmark

Laserdan

06-09-2009, 10:33 AM

Hey Dierk

I get out of memory exceptions when trying to create a continuous contract from ZenFire & Mirus

Just start from 4 contracts ago and try to merge into the next contract 3 contracts ago with tick data

then take everything 3 contracts ago and merge into 2 contracts ago

I have never been able to do a year without out of memory exception

Daniel

NinjaTrader_Dierk

06-09-2009, 10:36 AM

Thanks Daniel. However continuous contract handling will be much simplified and very different in NT7. Thus you could not compare. We will provide details on NT7 continuous contract handling as we roll first beta.

Laserdan

06-09-2009, 10:37 AM

That what I thought, so I didnt post it b4

I so look forward to NT7

astra

06-09-2009, 10:45 AM

Are there any eSignal users out there who have been keeping up their continuous ES ##.## symbol or others who maintain exported ES continuous contract data at the tick level for this stress test of NT7 memory consumption? If not, when I have more time, I'll have to splice what I can from backfill data from each separate contract. No need to backadjust for this excersize.

Thanks,

Whitmark

I have one year data of ES.##.##. Tell me how and where to send it.

astra

whitmark

06-09-2009, 11:05 AM

I will try if you could provide an import file (e.g. on some download area, you could try Help->Mail to support+attachment). . . .

Thanks astra. I assume its one year of tick data vs just minute data? Export it via Historical Data > Export and then Zip it and send to support via Control Center Help > Mail to Support and add as an attachment. It may be too large to export, in which case you might have to export it in two chunks. Or the export file may be too large to email, in which case you have to use a free service like yousendit.com to get it to support@ninjatrader.com and reference N7 Memory Consumption Test

Thanks again!

Whitmark

NinjaTrader_Dierk

06-09-2009, 11:14 AM

Guys, before you go through that exercise just double check the import format would be supported by NT

whitmark

06-09-2009, 11:22 AM

Dierk, is it safe to assume if astra exports the data from NT6.5 it can be imported into NT7 or is there a complete change? If the later, how would you propose we assist to get ES data into NT7 for a larger period stress test?

astra

06-09-2009, 11:23 AM

Guys, before you go through that exercise just double check the import format would be supported by NT

Dierk,

I am sending you a .txt file. I used the Historical Data -> Export facility of NT as whitmark suggested. I hope is OK.

astra

NinjaTrader_Dierk

06-09-2009, 11:26 AM

- historical data files exported by NT6.5 can be imported by NT7
- please make sure you zip the file since it would be huge. It would become a handling/upload nightmare if you didn't...

astra

06-09-2009, 11:35 AM

- historical data files exported by NT6.5 can be imported by NT7
- please make sure you zip the file since it would be huge. It would become a handling/upload nightmare if you didn't...

Too late.... is still uploading... I'll try to zip it now!

whitmark

06-09-2009, 11:43 AM

What is the size is the file once its zipped? I can't imagine a file of this size being uploaded in this manner. Give yousendit.com a shot, I believe its still free for transfers of up to 1 gb.

astra

06-09-2009, 11:55 AM

What is the size is the file once its zipped? I can't imagine a file of this size being uploaded in this manner. Give yousendit.com a shot, I believe its still free for transfers of up to 1 gb.

whitmark,

The size of the file is 215MB and the zip version is 15MB. Now I am trying to send it via a normal email (should be OK) but in the meantime I'll have a look to yousendit.com

whitmark

06-09-2009, 11:56 AM

Thanks again astra . . . looking forward to some more exciting results with this test.

astra

06-09-2009, 12:07 PM

Thanks again astra . . . looking forward to some more exciting results with this test.

Is done! Should be now with Dierk. Me too I am exited! From the info so far with new GO, Monte Carlo, allowing for future bar projection, SQL, and now speed improvements, looks like NT7 will be a fantastic platform. Thanks whitmark for the tips.

astra

NinjaTrader_Dierk

06-09-2009, 01:45 PM

astra

06-09-2009, 02:05 PM

Unfortunately there only are a couple of weeks in that file. Sorry guys, but I can't denote any more time on dealing with file handling issues: Whitmark, please let me know as you have a file readily available. Thanks

Dierk,

Check the latest email with the link to yousendit.com. You can download it from there for the next 7 days. I send it to support... att: Dierk. Cheers,

astra

Edit: When unzipped you should get a 215.224 KB file: ES: June 2nd 2008- June 5th 2009.

NinjaTrader_Dierk

06-09-2009, 02:23 PM

This is the file I downloaded from the yousendit.com link (215 MB). Here are the last few lines of that file as I opened it in notepad:
...
20080912 203110;1252.5;20
20080912 203110;1252.5;4
20080912 203110;1252.75;1
20080912 203110;1252.75;3
20080912 203110;1252.5;1
20080912 203111;1252.75;15

Sorry, due to bandwidth limitations I can't denote any more time on file handling issues...

whitmark

06-09-2009, 02:44 PM

Sorry, Dierk, thought this would be more straightforward. I'll take care of it and get it to you.

makra081

06-09-2009, 04:59 PM

1. open new chart

2. put only bars on it

3. select cross mouse tool

4. move your mouse constantly in cycle.

5. after some seconds the mouse will stop and hook & move & hook & move & hook.

nice memory leak hey ?

Its getting worse if you add some indicator :)

astra

06-09-2009, 08:04 PM

Just FYI (ie, if someone else tries to export large historical data) the process with NT6.5 Historical Data -> Export is not not so straightforward!

What I have discovered (the hard way...) is when memory peaks to the threashold of 1.52GB aprx, NT6.5 will prompt you that the export was successful and sends a file, but this file does NOT represent the whole period you requested, but only the period it manages to export before the limit of 1.52GB memory is reached. That's why my 215MB file I sent represented only the first 15 weeks of data. Eventually, one year's ES tick data was 1.3GB total!

So, be cautious when you export data to free space in your database to deal with the other NT limitation of 2GB total database limit, to export small files up to 200MB and to always check the file if has the whole period you requested (recently I half my DB from 2GB to 1GB doing that with over 50 stocks of 1 year tick data and I after today I dont have the courage to check what I have done...)

Anyway, 8 zip files with 1.3GB total data are now with Dierk and is up to him to judge how to proceed. From my side, and I believe (from our communication) whitmark's as well, I can't think of another way to send the data in one file with NT6.5. I hope NT7 addresses this problem as well.

astra

NinjaTrader_Dierk

06-09-2009, 10:47 PM

Thanks for your contribution. I added the results to my original post.

whitmark

06-10-2009, 05:28 AM

Setup4, export 1 year of ES continuous contract tick data
NT6.5: User reported was not possible and only could be done in chunks of 1-3 months
NT7: RAM usage < 500MB, took ~5mins

Setup5, backtest of SampleMaCrossOver on 50 ticks ES series, 2008-06-01 through 2009-06-09:
NT6.5: User reported was not possible due to OutOfMemory exception
NT7: RAM usage < 140MB, took < 3mins

More impressive results! Many thanks, Astra, for your persistence in getting the data out of NT6.5 and onto Dierk. The reward for your hard work is an additional test to prove NT7 is up to the task for exporting. (Unclear if an import of the ~1.3 GB file exported by the NT7 export test would yield similar results as large imports in NT6.5 are known to fail)

The results of Setup5 seem to indicate there is plenty of headroom for multiple years of data. If you can share your files, Astra, and if Dierk is willing to test it, I may be able to create multiple previous year test files by modifying the year field on each record.

Whitmark

ctrlbrk

06-10-2009, 05:38 AM

More impressive results!

The results of Setup5 seem to indicate there is plenty of headroom for multiple years of data. If you can share your files, Astra, and if Dierk is willing to test it, I may be able to create multiple previous year test files by modifying the year field on each record.

Whitmark

I agree! I love this thread. It makes me happy. :D

Also, with NT7's largeaddress patch built-in, it automatically doubles the headroom from 6.5 anyway. So you take that and also take the improvements in memory management, and we have ourselves a winner.

I am also particularly impressed with the performance aspect. Dierk, would you mind telling/showing us what your TaskMgr looks like during these backtests -- ie cpu usage?

Mike

NinjaTrader_Dierk

06-10-2009, 06:11 AM

Thanks for your support guys

@whitmark: Let's leave that setup for now. It would not yield new information beyond what we already know. Any further research would dependent on the user actual setup anyway. I'll leave that to the beta testers.

NinjaTrader_Dierk

06-10-2009, 06:13 AM

CPU usage: the details of what's going on under the hood needed to be understood in order to make a proper judgement. I'm willing to expose more info as we actually rolled first beta, but would not want to start something now (it's just premature).

xTrader1

06-26-2009, 03:56 AM

Thanks for your input. You probably are aware of that thread: http://www.ninjatrader-support2.com/vb/showthread.php?t=17699

Please go to my initial post: if you would be able to provide all the info requested by that post, then we would be more than happy to test how your setup would perform on NT7. Thanks in advance

Dierk,

As you referred me to this thread, I'm not sure I qualify for NT7 setup check, because NT does not blow up yet ;) , and I have proprietary indicators, programmed in NinjaScript I'm not interested to post.

I think that an issue I raised in http://www.ninjatrader-support2.com/vb/showthread.php?p=102444#post102444 is not a bug, but a software architecture issue. In any case, nothing can be done, but to wait for NT7 beta and then to complain, if needed.
Thanks,

NinjaTrader_Dierk

06-26-2009, 03:59 AM

Sorry, if my post was not clear. As you stated you feel NT's memory consumption should be much less. We are happy to test if NT7 actually would be more efficient on your setup if you could provide the information required.

eDanny

06-26-2009, 06:50 AM

Maybe running the benchmark strategy, http://www.ninjatrader-support2.com/vb/showthread.php?t=11432&highlight=benchmark, on the same machine with NT6 and NT7 might yield some interesting results for us.

NinjaTrader_Dierk

06-26-2009, 07:11 AM

I don't think so. The benchmark rather is a CPU/machine benchmark than a "NinjaTrader" benchmark. Please contact ctrlbrk (the benchmark author) for details.

eDanny

06-26-2009, 08:57 AM

That is correct. Isnt NT7 supposed to be a little more CPU friendly so to speak?

NinjaTrader_Dierk

06-26-2009, 08:59 AM

We applied several optimizations with NT7. However the benchmark covers standard C# stuff which NT has no impact on. Please contact the author for details.

eDanny

06-26-2009, 09:05 AM

Ok thanks Dierk, that makes sense.

NinjaTrader_Ray

07-09-2009, 11:16 AM

Hi All,

We have been working on a "caching" logic has drastic performance improvements when working with bar series. You will notice serious improvement in areas like:

- NT start up
- Backtesting/optimization
- Working with charts etc...

What NT7 will do is this:

- You request a 10 tick chart
- NT aggregates 1 tick data into 10 tick bars and write this to your disk
- Next time you request 10 tick series, its 10x faster since it does not have to re-construct the bars.

The following video is a comparison of performance on a backtest.

It executes a backtest on about 10 million ticks.

http://screencast.com/t/ID0EbV19o (http://screencast.com/t/ID0EbV19o)

whitmark

07-09-2009, 12:35 PM

Thanks, Ray, for the update and the video is a nice touch. Yes, it is a serious improvement. If you care to, can you address the following questions on this topic:

1) What is the persistence of the cached data? Is it only for this instance of the Strategy Analyzer?

2) How might the caching work when using the walk forward optimizer? Would it attempt to cache the entire data range up front or would it cache the data incrementally as it walks forward.

Thanks!

Whitmark

NinjaTrader_Dierk

07-09-2009, 12:38 PM

Gains

07-09-2009, 02:09 PM

Thanks a lot Ray for keeping us updated like this! It is very much appreciated.

Gains

syswizard

07-14-2009, 05:00 AM

1) Available whenever bars series are built. The cached data then would be available throughout NT.
2) Really does not matter, since cached data is either used (when available) or build (when not available) whenever bars are requested (no matter when that actually would be) = transparent cache
Will NT7's caching accomodate the situation where the 10 tick bars were built say 5 days ago, but now need refreshed with the most recent 5 days of tick history ? Or will it rebuild the entire series all over again from the requested start date ?

NinjaTrader_Dierk

07-14-2009, 05:23 AM

>> but now need refreshed with the most recent 5 days of tick history
correct

KrisS

07-18-2009, 08:39 AM

We applied several optimizations with NT7. However the benchmark covers standard C# stuff which NT has no impact on.

NT Dierk,

Is this to say that NT7 will not run this benchmark with significantly improved hyperthreading and multi-core support because the benchmark uses C#?

I would have expected a significant improvement in calculation time based on the multi-core support in .NET 3.5

NinjaTrader_Dierk

07-18-2009, 08:49 AM

You need to contact the author in order to understand what this benchmark actually measures and what not.

zoltran

07-18-2009, 02:46 PM

NT Dierk,

Is this to say that NT7 will not run this benchmark with significantly improved hyperthreading and multi-core support because the benchmark uses C#?

I would have expected a significant improvement in calculation time based on the multi-core support in .NET 3.5
Hi Kriss
Just a general observation... Using multiple 'cores' or CPU's is a pretty complicated topic.
Many people think it's just a matter of using a new library, or 'dotNet' or OS, but it really is a lot more complicated than that.
Programs in a multitasking OS like Windows break work into many threads or processes. All else being equal, Windows can assign each thread to a 2nd parallel core. But this is not as easy as it sounds, as many things need to be done serially vs parallel.
For example, the calculations in an indicator/strategy need to start from the begining and progress to the end..., and usually require information from the prior bar.
Take a simple moving average over a 400 bar chart .. Splitting that calc into 4 chunks or threads of 100 bars isn't feasible, as each 100 bar chunk needs the value of the all the prior chunks. Plus, there's the overhead of coordinating all these tasks... I used to work on a large transaction processing system and we found the this inter-task/thread co-ordination often stymied the benefits of running multiple threads.

This is one of the reasons why massively parallel super-systems with 10's to 1000's of cpu's are not in common practice.

So in practice, it's unlikely that any given single task, such as an individual chart window will benefit from a multi-core system.

But ... it should be possible to design a system such that each chart or task does have it's own thread, so if you run multiple charts it could be spread over multiple cores.

The NT team has also said they have made multi-tasking improvements in the optimizer, That sounds like some pretty sophisticated programming

kman1

07-23-2009, 03:09 PM

Im wondering why my Murrey Math indicator does not work with Range charts. Only happens on NT. Works fine on other programs. Any help would be appreciated:confused:

NinjaTrader_Kyle

07-23-2009, 04:30 PM

Hi kman1,

Thank you for your post.

I suggest contacting the indicators author for more information.

djohnsonhot

08-07-2009, 07:05 AM

It appears that this may be a good forum on system resource improvements...those that will determine whether I buy NT 7.0 Lifetime licenses or not. Although NT is far superior in many ways compared to other platforms, as it stands today, I cannot use NT without memory dumps, crashes, lost connections and so on. Since I tried using NT as a reference platform, naturally I use my feed and place sim trades. Yesterday, I entered a trade and it took over 8 seconds to appear (right click on chart, Lady-voice confirmation, trade entry line appeared)...that is just ridiculous. Furthermore, the loading times are unreal. I will wait over a minute to fully paint the static workspace, and THEN click connect, and I wait another 30 seconds to a minute for bars to appear. I am using ThinkorSwim's platform and I need to say this: it is not a terrible platform, and in fact it has some things that are better than NT (their trading DOM is simply better as it displays updated volume accumulations that NT does not - - which allows traders who pay attention to VOLUME to note clusters of accumulated large block trades which indicate zones to watch carefully). And, they do have on-chart trading but it is not refined. I know the President there and they claim they are working on it, but NT is far superior in this way. I love every aspect of NT in this area of programming development. Ray, if you are reading this, I am waiting and hoping that you jump past BETA soon and give me a platform that is not a memory hog, offers better OCO "bracket orders on a single click" and other new stuff to make this thing my platform for the long run. Thanks for listening.

kman1

08-07-2009, 01:52 PM

Here, Here, very well said. Im sure you speak for the majority of NT users.
Wont renew unless new version is up and running before next quarter.
Too bad, great platform though.

jam.76

10-09-2009, 01:39 PM

I too, am going through a database corruption problem and no one can seem to put their finger on it. It's always "rename and start over." Doing this wipes everything out ie.. atm, instrument manager and so on.... I have been using Ninja for 1 year in Sim format. I attempt to run many instuments on multiple times with automated strategies and problems are always present. Complete data corruption of database being latest issue.
Although I would like to, there is no way I could commit real money to live trading with the product the way it is. I can only speak for the application of multiple instruments, multiple times and complete efficient automation. Unfortunaely, Ninja has crashed and burned for my application and can not be recommended. Until then, I am open to suggestions on what platform can fulfill my criteria succesfully.

JAM

NinjaTrader_Bertrand

10-10-2009, 01:59 AM

jam.76, welcome to our forums, I'm sorry to hear your disappointment - please contact me using the Help > Mail to Support feature and check to include trace and logs files, we'll review them then and advise further.

Andron

04-25-2010, 09:47 PM

Why is skillful people can not find :> Disable scanning of ougiong email

Why that item can not be found anywere ? Anybody with experiance ?

visounds

07-08-2010, 01:16 AM

Hopefully this reply gets me some help. I have been trying NT7 for months thinking that every beta will improve. I cannot get it working without constant database corruptions and with the current version which I have tried on Windows 7 and Xp I cannot even get any data streamed to my charts at all.

What steps can I take to try and get this resolved?

I have renamed the database and even tried to just bring up a plane old 1min chart and still no data on the chart yet the Doms work fine and i can even trade off the doms, it is purely a problem with the chart data. I have made sure I have updated the Msoft .Net to 3.5 before installing the latest BETA but still zippo comes through. The strange thing is that the charts update every few days with a price resembling the end of day price but no streaming of chart data takes place.

Is this a simple switch somewhere I don't know about?

Cheers,
Andrew