replay grows sluggush with autotrader strategy

[roland_nt]

replay grows sluggush with autotrader strategy

I have not been able to reliably re-create this problem. 95% of the time there is no problem

I have learned the following:
1)Completely removing and re-installing .NET 3.5 does not seem to be a cure.
2)The slowdown will also sporadically occur during market hours. When it occurs during real time trading, minimizing the window containing the strategy decreases the slowdown, frequently restoring things back to normal while the window is minimized.
3)In real time data feed, also disabling/re-enabling the strategy will clear the problem for a while.

I have given NT support samples of the code in question. They have been very helpful; we are working together to see if there is anything obvious in the code which is a stimulus..

[sledge]

I have not been able to reliably re-create this problem. 95% of the time there is no problem

I have learned the following:
1)Completely removing and re-installing .NET 3.5 does not seem to be a cure.
2)The slowdown will also sporadically occur during market hours. When it occurs during real time trading, minimizing the window containing the strategy decreases the slowdown, frequently restoring things back to normal while the window is minimized.
3)In real time data feed, also disabling/re-enabling the strategy will clear the problem for a while.

I have given NT support samples of the code in question. They have been very helpful; we are working together to see if there is anything obvious in the code which is a stimulus..

Create a simple strategy that has 100 trades a day...

make some silly crossover with ZLEMA and EMA or SMA.. whatever.. go to 2 range bars.... I might even try this on this weekend.

[marcow]

please find attached a demonstration strategy to reliably reproduce the replay slowdown as discussed in this thread.

The strategy is named ReplayCrash, because in earlier versions of NT this replay test would cause NT to crash to the desktop. In R11 the crash doesn't seem to happen anymore, so there are definitely some improvements made in the R11 version.

My systems specs: Vista x64 8 Gb RAM, dual quadcore Dell workstation.
During the test below there is always at least 1 Gb free memory.
Version: NinjaTrader 7 R11, Slowdown occurs on both 32 and 64 bit NT.

Also pay attention to NT memory usage. Memory usage increases from 58 Mb at replay start to over 400 Mb after only 1 month of replay. Reported values are peak values, memory usage swings are between peak value and approx. 100 mb below.
Could this be a memory leak ? There is definitely something that gets clogged up.

cpu % usage increases from 2% initially at replay start, to 25% at 7/21/2011 (25% seems to be the maximum, using 2 cores out of 8 )

REAL replay speed decreases from 500X (as indicated in the replay control panel) to a very, very slow 20X after 1 month of replay data. It is nearly impossible to replay more than 1 month using these strategies.

Below are the steps to reproduce the slowdown. In this example I run 3 strategies in separate charts in market replay, running more strategies at once will probably generate the slowdown even faster.


Here's how to reproduce the slowdown:

1) restart NT
2) open 3 charts of the ES contract; 4 Renko , 5 Renko and 6 Renko (default Renko)
3) connect to market replay
4) place the ReplayCrash.cs strategy in each of these charts, enable these 3 strategies. Do not change any parameters, leave all parameters at default settings.
5) start market replay and set speed to 500X

because I used an older expiration of the ES contract, I'll start replay at 6/20/2011 at an indicated speed of 500X. Of course you can also use the current front ES expiration, as long as you've got 1 month of continuous replay data. Values below are for 32bit NT (but again, slowdown also occurs on NT 64bit ).

6/20/2011: replay start, NT memory usage 51 Mb Real replay speed approx 500 X
6/22/2011: NT memory usage peaks at 78 Mb Real replay speed approx 450 X
6/28/2011: NT memory usage peaks at 147 Mb Real replay speed approx 400 X
7/5/2011 : NT memory usage peaks at 186 Mb, sometimes replay halts for a second or 2, and then continues. Real replay speed decreased to approx. 320 X. Short bursts of disk caching audible.
7/12/2011 : NT memory usage peaks at 260 Mb, 2 second halts + disk caching. Real replay speed decreased to 72 X
7/14/2011 : NT memory usage peaks at 305 Mb, 2 second halts + disk caching. Real replay speed decreased to 60 X
7/18/2011 : NT memory usage peaks at 346 Mb, 2 second halts + disk caching. Real replay speed decreased to 32 X
7/18/2011 : NT memory usage peaks at 365 Mb, 2 second halts + disk caching. Real replay speed decreased to 32 X
7/21/2011 : NT memory usage peaks at 414 Mb, 2 second halts + disk caching. Real replay speed decreased to 20 X

In 1 month of market replay, the real replay speed decreases from 500X to 20X, CPU% increases from 2% to 20+% , and memory usage increases from 58 Mb to 414 Mb.

It appears that when the replay halts (and NT is not responding for 2 seconds) it is always at the point when a strategy is opening or closing a position. Is the log file "overflowing" because opening/closing positions creates a lot of entries in the log file ?
Is it possible to completely disable all logging, so the log file remains completely empty ?

Actual replay speed seems dependant on how many strategies have a position. So when 1 strategy is Long and other 2 are flat, replay speed will be higher then when all 3 strategies have a long position.

Let me know if you can reproduce the slowdown as described above.

Marco

[sledge]

After about 2.5 hours

I maxed out sooner than you I believe.

I started on ES 12-11 10/3/2011

by 10:00am, I was already into 4-6 seconds at 500x.

memory usage was stable at first, it appeared to grow stably, then reduced. I started watching VM Size. that was stable too.

by 10/4 3:30 am, memory was jumping huge. 100k swings. (in a few clicks). VM Size is crazy jumps too.

My CPU was 44C, then started to creep into 45. I threw a fan on the case, and I'm down to 38C.

2006 amD athlon x2 2210MHZ (overclocked 1% becuase i can't figure out how to remove the overclock), 3 gb, .. intel 520 ssd 180 gb with 40 gb free, windows xp 32 bit, all up to date.

cpu for NT was 48% stable, now jumpy, 45-80%,... cpu on both cores maxing out now near 100%

I'm at 1,103 trades.... and just got whacked. It looks like i was missing 10/4 data as I zipped fast through the day session until night.

the log file started around 250k, i'm at 3,392kb now. probably 2.5-3 hours worth.

I haven't crashed yet. I don't think my CPU is fast enough. I'm going to have to leave it run overnight.


now MEM USAGE is 250k-600k..


my Virtual Memory under control panel->system , is set to "system managed". so it should expand more when needed and contract. therefore I don't think I'll run out of memory. (well, I'm only on day "1.5", no where near day 30. that might take a week at this rate!!!)

please find attached a demonstration strategy to reliably reproduce the replay slowdown as discussed in this thread.

The strategy is named ReplayCrash, because in earlier versions of NT this replay test would cause NT to crash to the desktop. In R11 the crash doesn't seem to happen anymore, so there are definitely some improvements made in the R11 version.

My systems specs: Vista x64 8 Gb RAM, dual quadcore Dell workstation.
During the test below there is always at least 1 Gb free memory.
Version: NinjaTrader 7 R11, Slowdown occurs on both 32 and 64 bit NT.

Also pay attention to NT memory usage. Memory usage increases from 58 Mb at replay start to over 400 Mb after only 1 month of replay. Reported values are peak values, memory usage swings are between peak value and approx. 100 mb below.
Could this be a memory leak ? There is definitely something that gets clogged up.

cpu % usage increases from 2% initially at replay start, to 25% at 7/21/2011 (25% seems to be the maximum, using 2 cores out of 8 )

REAL replay speed decreases from 500X (as indicated in the replay control panel) to a very, very slow 20X after 1 month of replay data. It is nearly impossible to replay more than 1 month using these strategies.

Below are the steps to reproduce the slowdown. In this example I run 3 strategies in separate charts in market replay, running more strategies at once will probably generate the slowdown even faster.


Here's how to reproduce the slowdown:

1) restart NT
2) open 3 charts of the ES contract; 4 Renko , 5 Renko and 6 Renko (default Renko)
3) connect to market replay
4) place the ReplayCrash.cs strategy in each of these charts, enable these 3 strategies. Do not change any parameters, leave all parameters at default settings.
5) start market replay and set speed to 500X

because I used an older expiration of the ES contract, I'll start replay at 6/20/2011 at an indicated speed of 500X. Of course you can also use the current front ES expiration, as long as you've got 1 month of continuous replay data. Values below are for 32bit NT (but again, slowdown also occurs on NT 64bit ).

6/20/2011: replay start, NT memory usage 51 Mb Real replay speed approx 500 X
6/22/2011: NT memory usage peaks at 78 Mb Real replay speed approx 450 X
6/28/2011: NT memory usage peaks at 147 Mb Real replay speed approx 400 X
7/5/2011 : NT memory usage peaks at 186 Mb, sometimes replay halts for a second or 2, and then continues. Real replay speed decreased to approx. 320 X. Short bursts of disk caching audible.
7/12/2011 : NT memory usage peaks at 260 Mb, 2 second halts + disk caching. Real replay speed decreased to 72 X
7/14/2011 : NT memory usage peaks at 305 Mb, 2 second halts + disk caching. Real replay speed decreased to 60 X
7/18/2011 : NT memory usage peaks at 346 Mb, 2 second halts + disk caching. Real replay speed decreased to 32 X
7/18/2011 : NT memory usage peaks at 365 Mb, 2 second halts + disk caching. Real replay speed decreased to 32 X
7/21/2011 : NT memory usage peaks at 414 Mb, 2 second halts + disk caching. Real replay speed decreased to 20 X

In 1 month of market replay, the real replay speed decreases from 500X to 20X, CPU% increases from 2% to 20+% , and memory usage increases from 58 Mb to 414 Mb.

It appears that when the replay halts (and NT is not responding for 2 seconds) it is always at the point when a strategy is opening or closing a position. Is the log file "overflowing" because opening/closing positions creates a lot of entries in the log file ?
Is it possible to completely disable all logging, so the log file remains completely empty ?

Actual replay speed seems dependant on how many strategies have a position. So when 1 strategy is Long and other 2 are flat, replay speed will be higher then when all 3 strategies have a long position.

Let me know if you can reproduce the slowdown as described above.

Marco

[sledge]

ummm, i'm at 100% now both cores......


msmpeng.exe has kicked in....ok, that was windows defender


NT swings growing bigger at 1.200 trades.


mem delta is now 500k+....


back too 77%-100%

chart trader bid/ask barely updating with trades happening...
PF usage gong bigger now,,,,

at 180,000,000 page faults...

After about 2.5 hours

I maxed out sooner than you I believe.

I started on ES 12-11 10/3/2011

by 10:00am, I was already into 4-6 seconds at 500x.

memory usage was stable at first, it appeared to grow stably, then reduced. I started watching VM Size. that was stable too.

by 10/4 3:30 am, memory was jumping huge. 100k swings. (in a few clicks). VM Size is crazy jumps too.

My CPU was 44C, then started to creep into 45. I threw a fan on the case, and I'm down to 38C.

2006 amD athlon x2 2210MHZ (overclocked 1% becuase i can't figure out how to remove the overclock), 3 gb, .. intel 520 ssd 180 gb with 40 gb free, windows xp 32 bit, all up to date.

cpu for NT was 48% stable, now jumpy, 45-80%,... cpu on both cores maxing out now near 100%

I'm at 1,103 trades.... and just got whacked. It looks like i was missing 10/4 data as I zipped fast through the day session until night.

the log file started around 250k, i'm at 3,392kb now. probably 2.5-3 hours worth.

I haven't crashed yet. I don't think my CPU is fast enough. I'm going to have to leave it run overnight.


now MEM USAGE is 250k-600k..


my Virtual Memory under control panel->system , is set to "system managed". so it should expand more when needed and contract. therefore I don't think I'll run out of memory. (well, I'm only on day "1.5", no where near day 30. that might take a week at this rate!!!)

[sledge]

NT is now overtaking my firefox session as biggest mem usage.... 870kk blip for nt with 670k steady for FF

ummm, i'm at 100% now both cores......


msmpeng.exe has kicked in....ok, that was windows defender


NT swings growing bigger at 1.200 trades.


mem delta is now 500k+....


back too 77%-100%

chart trader bid/ask barely updating with trades happening...
PF usage gong bigger now,,,,

at 180,000,000 page faults...

[sledge]

NT is now overtaking my firefox session as biggest mem usage.... 870kk blip for nt with 670k steady for FF

CRASHED!

started replay on ES 10/3/2011. market replay final time 10/10/2011 3:14:26 am.

2,609 trades .. net profit w/o commisions $6,900, gross profit 61,230, gross loss -54,000.


last log file activity 12:45pm today.. for a total of about 16 hours before crash.

1,993,300,000 page faults
24,000,000,000 I/O read bytes.
9,385,819,000 I/O write bytes.

Chart Trader on a 4 range window lost it's way.. it was bad when I left for work at market replay 10/6/2011 this morning.

So I could only make it through 1 week of market replay.

[marcow]

thank you sledge for reproducing the slowdown -uhhm crash in your case

Can anybody else -or better NT support- confirm the slowdown using the procedure I posted in post #18 ?

Marco

[NinjaTrader_Bertrand]

Marco, thanks for the testing and scenario - in testing my memory used by NT got up to 490MB, so no crash seen due to running into having to less memory available. How do you judge the replay speed going down for you? I''m not surprised to see this artificial performance heavy scenario creating a strong use of resources though.

[marcow]

for clarity, I also don't get a crash with the current version, the crash was with earlier versions of NT. It's just that replay speed gets slower and slower to the point it is unworkable. It is nearly impossible to market replay more then 1 months of data.

How do you judge the replay speed going down for you?

(see post #18)

absolutely horrible. REAL replay speed (not indicated) goes down from about 500X immediatley after starting market replay, to 400X after 1 week, week 2 it's down to 320X , after week 3 it a sluggish 100X and thereafter it is nearly impossible to replay anymore because real speed drops to 30X and below (although indicated speed is 500X !) I mean this can't be normal behaviour ?

in testing my memory used by NT got up to 490MB

So after 1 month of replay, memory rises with about 450 Mb ?
Where does NT need all this memory for ? please explain.

It's not the absolute amount of memory that bothers me, what worries me is that the increase in memory usage is not linear but exponential. If it was lineair, that would be somewhat normal behaviour.

What I cannot understand is why the replay speed drops over time. I've done a lot of 3D rendering, and after rendering an animation that took several weeks of 24/7 rendering, frame render times where almost flat.
NT market replay is the first software I work with that gets tired after 1 month and then starts to melt down.

To me it looks like something gets clogged up big time. Like buffers (log file ? ) or something. That would also explain the disc caching audible when the slowdowns occur.(please re-read post #18 carefully )



ps. I did a new testrun with only 1 strategy on a 4 Renko chart, and on 8/3/2011 - 6 weeks of replay - real replay speed is now down to 60X (although indicated speed is 500X), mem usage is 400Mb.

Marco

[sledge]

Marco, thanks for the testing and scenario - in testing my memory used by NT got up to 490MB, so no crash seen due to running into having to less memory available. How do you judge the replay speed going down for you? I''m not surprised to see this artificial performance heavy scenario creating a strong use of resources though.

Judging replay speed.. Market replay at 500x was down to 2-3 seconds on DAY #2. It was NO LONGER going a couple minutes at a time, but 2-3 seconds.

I believe my crash came as a result of NT blipping memory usage to over 3GB. It was doing 1 gig blips before I left for work. 300,000k one second, 1,000,000+ the next.

[marcow]

exactly sledge ! That's the symptoms I'm talking about.

Right after starting replay it advances approx. 8 min between each update in the market replay window. After the replay has slowed down considerably, it advances maybe 1 min every update followed by a halt/freeze of 6 seconds where nothing happens at all, except a lot of disc caching. Thus effectively reducing replay speed to approx 30X.

It seems as if market replay gets tired and starts to fall asleep

[NinjaTrader_Bertrand]

Thanks guys for the added details, I'll be back in touch with an update on our extended testing next week.

Good weekend.

[NinjaTrader_Bertrand]

Thanks again for the testing and scenario provided guys - we were able to reproduce parts of the slowdown created due to excessive order submission over an extended data playback range. But I still feel this is a valuable testing scenario and as such we will use it internally looking into working enhancements and optimizations into our MR engine for our next major update of the platform.

[laredo78676]

CPU usage high,

Where do i find the check list for high CPU usage issues please?