SuperDom CPU issue

[tonchi]

I have had 2 stop orders in as many days that took so long to exit that I lost two ticks on each exit. I noticed in the beta release notes there is a problem with excessive cpu usage on version .7 for the dom.

When I went to download that beta, it is no longer there. Why was the beta link pulled, and what is being done to fix this superdom issue cpu usage?

[NinjaTrader_Ray]

We continually imrprove NinjaTrader to be as efficient as possible and therebye lowering CPU usage. If you are using Patsystems, IB or MBT CPU usage on average is below 5%.

We removed the beta due to some installation issues. A new beta will be available over the weekend.

Stop orders are not triggered by the SuperDOM or NT. They reside with your broker or at the exchange and are triggered there. If you have experience slippage, it is due market conditions or external factors outside of NT.


Ray

[tonchi]

This is from the release notes of the beta

Improved: Lowered CPU usage on dynamic SuperDOM

All due respects, but the market conditions were not unusual, only the amount of time taken to close the trade. I have a hyper-threaded P4 with 1 GB of ram, so I dont think the problem is with the computer. I have also noticed up to a 1-2 second delay when just placing a limit order to enter a trade.

[NinjaTrader_Ray]

No problem was not disrespected.

Stop orders are at the exchange and are triggered by the market. NT only displays your orders. Any performance issues that may exist for any reason on your PC will not affect the price your stop order is filled at.

If I am not understanding your concern please let me know.

Ray

[tonchi]

It was a sim stop so it was resident on my machine.I should have said that in previous post. I totally understand its' risks, as I have used it for over a year. Up until this version, I have never lost more than 1 tick, and I have not noticed this behavior, where the execution of the stop goes in what looks like computer slow motion. I am making sure no other program that is TSR is running, and just two data collection programs are sharing the computer, NeoTicker and MarketDelta.

[NinjaTrader_Ray]

I see, thank for clarifying. We will release a new beta version by Tuesday, please upgrade to that version and report back. We have made a signnificant amount of changes, several in the area of performance and would prefer to see if you experience any issues with the new release.

Ray

[Antraman]

Getting back to CPU usage, I am sitting here now with 4 charts, 2 doms, and the control center open, and the Windows Task Manager window open. NinjaTrader, by the minute is eating up Memory Usage like its going out of style. When NT first opens, its about 105MB in size, already double my Firefox browser with 90 extensions and several tabs open. After 10mins, NT is another 50MB in size.

The CPU usage clicks from as low as 1 to as high as 26%, and the higher the Mem Usage gets, the greater the CPU usage jumps to. And the CPU time gets slower and slower.

I got another "Failed to execute DB job error" tonight, even after doing a DB repair, and when I checked the Task Manager, NT froze at 226MB. I think the DB errors occur because NT is getting too big and unmanageable...

Check out the screen shot of Task Manager to see what its doing.

I realise that NT is downloading price data in real time, but surely NT has to write some data to disk so it doesn't overload the Memory while its open?

And if you were about to ask, my computer is quite capable - a 2.6GHz Pentium 4 Sony Vaio laptop, with 1GB Ram, a 64MB Radeon graphics card and plenty of HDD space.

Any thoughts???

Cheers,

Antraman
0

[NinjaTrader_Ray]

It would be great if you could help isolate the culprit. Peel off charts one by one and see which chart is causing the problem. Then we can look there.

[Antraman]

OK. I closed my 4 charts, then the 2 DOMS, 1 by 1 gradually. It appeared that none of those actions affected the Mem usage at all. It still kept growing. So right now my Control Centre is open. The Mem usage now, is 334MB.

I then reopened a 1min chart, and as it loaded the price data, it jumped from 330MB down to 108MB, then when the data download completed, it jumped to 200, so there was a considerable change. This was loading in 10 days previous data. Opening DOMS again didn't seem to affect it.

I then closed the charts and DOMS, and reopened a minute chart, this time loading 40 days previous data. The mem usage shot back up to 313MB, and is continuing to grow.

Closing them off doesn't seem to lessen the memory usage, but reopening them with a less historical data does. However, I was running minute charts with about 20 days data when the memory had grown to 330MB, but when I closed all and reopened with 40 days data, it did not immediately increase anymore than with the 20.

It seems that just collected data real time is what causes it to grow the most.
0

[NinjaTrader_Ray]

What indicators do you have on your charts? Any custom indicators?

[Antraman]

1. 90min line with bollinger, SMA and Vol

2. 15min line with bollinger, SMA and VolColoured a few lines and a few fibs

3. 5min bars with SAR, bollinger, SMA, MACD, stochastic, ADX, Aroon, and VolColoured, a few lines and a few fibs,

4. 1min bars with SAR, bollinger, SMA x3, MACD, stochastic, Williams, and Split Vol.


Trading on the Euro


000

[Antraman]

Happened again tonight. As the night wore on, computer slowed down to a crawl at times, then started getting DB job errors. Task Manager showed the Memory usage @ 521MB, and CPU usage up to 75%. (See screenshot)

I am sure this is not what is supposed to happen.

I tried removing various indicators from charts, particularly user ones, didn't affect the readings at all.

Eventually had to close down, and when I reopened, Memory Usage was 79MB. All the charts and DOM's came back, with the same historical data. Just like nothing had happened. Once I connected to IB, then the memory usage started ticking over again.

Only problem now is the trade I had on when it froze missed the stop loss and I am 12pts in loss. Not good.

Any ideas?

cheers,

Antraman


0

[NinjaTrader_Ray]

Only idea is to remove indicators and add them one by one and try and figure out what specifically eats memory. I will do this based on your suggestion but it will be a few weeks since I am on vacation.


Ray

[Antraman]

I may have a reason/solution to this problem. Since I upgraded to the latest release the problem has thankfully not been recurring, however it happened last night after about 4 hours of trading.

I think it is related to using a computer that has an integrated video graphics card. This means that the motherboard memory is shared with the graphics processor.

My laptop has a second monitor out jack, and I am running a 23'' screen at 1600x1200 resolution. With lots of programs open, the screen flickers and breaks up occassionally, when there is some HDD or processor activity.

NT5 I think is quite processor hungry, and therefore requires full use of the computers resources available to it to run happily, and having the integrated graphics card demanding some of the resources to power the large screen takes something away from NT, causing to create Failed to execute DB errors...

Minimising (not closing) all NT open windows to the taskbar every couple of hours has the effect of reducing the MemUsage and CPU usage down to below 10% and 10MB, almost like a refresh, then maximising all the windows again starts the slow but steady increase of memory usage by NT.

It would be interesting to see if others who have had this Execute DB error problem
(spenbm01, tonchi et al) also run a computer with an graphics processor integrated with the motherboard.

As a result of this, I have decided I will build myself a dedicated trading computer (partly for fun, partly for the necessity) that willl have a high power graphics card (such as my newly acquired Matrox Quad Head) running in its own slot, plus a dual processor with heaps of RAM (like 16GB) Check out the Tyan Tiger or Thunder motherboards for this. Nice...

cheers,

Antraman


0

[NinjaTrader_Ray]

Do you use the crosshair on your charts?

[Antraman]

Yes, a couple of times. Why?
0

[NinjaTrader_Ray]

Try not using the crosshair. We did resolve a memory leak issue with the cross hair. This is for V6. I am curious if not using the cross hair would make a difference.

[Antraman]

OK, I'll try again on Monday when the Euro starts again. I don't recall a connection between the two though...

thanks,

Antraman



0

[copodon]

Antraman wrote: I think it is related to using a computer that has an integrated video graphics card. This means that the motherboard memory is shared with the graphics processor.



I'm having similar issues with NT regarding heavy mem usage. I haven'texperienced it to the levels your describing, however I only run a dome for ES plus an 80 Tick chart (2 days backfill) and two indicators (boll bands and stochastics). When I start up in the morning it runs between 50-68K and creeps upward all day, usually by mkt close it is running at between 110-130K.

NT charting has come a long way.With a few more improvements here and thereI think I will be able to ditch my other charting platform but the current levels of mem usage are definately a concern.

Have youseem any improvementrunning it ona more powerful computer with a dedicated graph card? The reason I ask is becauseI have a NVIDIAQuadro Card and 1.5G of ram so I am not sure if that is your problem. I also noticed flickering, but to a minimal extent (basically the Link button on NT windows disappear and reappear when clicking back and forth between different NT windows and other apps). I'm not sure if this is related to a video card or not because I observed it on two different machines, this one and another one with an intergrated card.

[NinjaTrader_Dierk]

a) on mem usage: Please try disabled cross hair. Does it make a difference?
b) link button flickering: not related to your video card nor related to mem or cpu usage. This is "standard" behaviour since link button painting is outside the scope of regular .NET painting (link button is on caption bar). We have not yet found a way to get around it.

[copodon]

Dierk Droth wrote: a) on mem usage: Please try disabled cross hair. Does it make a difference?
b) link button flickering: not related to your video card nor related to mem or cpu usage. This is "standard" behaviour since link button painting is outside the scope of regular .NET painting (link button is on caption bar). We have not yet found a way to get around it.


A) No I don't particulary use crosshairs, but I read before to stay away from them until V6 so I will keep that in mind

B) I see.I should add that through experimenting Ifound the screen flickering was not really related to NT more so when I run too many graphic intensive programs. Thank you for the explanation.

[Antraman]

I'm still having problems with Ninja crashing, giving "Falied to execute DB job error" messages. I haven't been using the crosshairs.

I've tried freeing up CPU and Mem resources by closing down other programs running in system tray etc. whichseems to help, however it really only prolongs the inevitable.

I get some respite by minimising all windows then maximising them again, which instantly reduces Mem usage to below 10MB, but it slowly creeps up again.

I get about an hour or two trouble free trading, then the DB errors. Mostly I remember to min/max all windows all windows, but when market action gets busy or at a crucial point, you can't afford to be thinking about your memory usage...and sometimes I've been caught.

I do hope this problem is understood, and sorted for v6.

cheers,

Antraman
0

[NinjaTrader_Dierk]

1) "Failed to execute DB job error" is not related to any chart functionaility. Did you try Tools->Options->Misc->Repair DB?
2) Mem usage you exerienced (< 10 MB) is caused by windows swapping logic. NT (as a process) still is using "its" memory, however it got swapped to disk and thus "Mem usage" went down.

[Antraman]

Hi Dierk,

re:
>1) "Failed to execute DB job error" is not related to any chart functionaility.

Thats not what I have observed. If I just have a DOM open the mem usage does not go very high, but as soon as I add charts, the rate of mem usage increases dramatically.

>Did you try Tools->Options->Misc->Repair DB?

Yes, I tried that, and other things like reset account, deleted the mdb files, started all over again. I tried removing various indicators on charts, put other ones on. It seems there is no particualr tool or indicator that is to blame, just an endemic problem in the system when you have more windows open and more indicators...then the more the mem usage increases, and the quicker it crashes...

cheers,

Antraman


0

[NinjaTrader_Dierk]

1) Although your statement about mem usage going up as you open chart is correct, the error you experienced is not related to mem usage nor charts (to the best of my knowledge). Please let me know if and how you can reproduce.
2) Did you check for right version of Jet and MDAC software: http://www.ninjatrader.com/web/trading_software_support_installationguide.htm
3) What windows version are you on?
4) How much RAM does your machine have and how much mem do you still have available when NT is running?

Thanks

[Antraman]

1) Although your statement about mem usage going up as you open chart is correct, the error you experienced is not related to mem usage nor charts (to the best of my knowledge).

I can prevent DB errors by regularly minimising then maximising all windows together, this keeps the mem usage down.

Please let me know if and how you can reproduce.

I can reproduce anytime. I have open:
1) Control Centre
2) DOM x 2
3)120min bar chart - 45 days, with Bollingers and volume
4)30min bar chart - 15 days, with Bollingers and volume, Previous day OHLC
5)5min candles - 4-5 days, with Bollingers, MACD, Stoch, pivots, OHLC, splitVOL
6)1min bar chart - 1-2 day, with Bollingers, some SMA's, MACD, stoch., split vol and pivots, OHLC

Running 2 monitors, 1028x768 and 1600x1200 on 2.66GHz Pentium 4 with Radeon 128MB onboard graphics chip.


2) Did you check for right version of Jet and MDAC software: http://www.ninjatrader.com/web/trading_software_support_installationguide.htm

As far as I can ascertain, they are the latest.

3) What windows version are you on?

XP Pro service pack 2 upgrade


4) How much RAM does your machine have and how much mem do you still have available when NT is running?

It has 1GB installed. Ninja usually crashes when mem usage is around 300MB.

Thanks0

[NinjaTrader_Dierk]

1) On your charts: Which interval are you using? 1 min? else?
2) As you get a crash there usally is more info available on sub-windows. Can you please post this information into here or send a mail to support (and refer to this thread).

Thanks

[Antraman]

Intervals on charts are shown below in previous post.

The sub window messages include 'ExecuteInsertJob', 'AccountUpdateJob' 'StrategyUpdateJob' all unspecified errors.

When the problem occurs, I seem to get 4-5 error messages for each order. For example, if I have an order in place, then the problem occurs, everytime there is a change to the orders on the DOM, like a trailing stop moving, or a target filled, stop filled, or placing another order etc.,, each order made will get a series of 4-5 error boxes, each saying:

Failed to execute DB job error: 'ExecuteInsertJob' unspecified error
Failed to execute DB job error: 'AccountUpdateJob' unspecified error
Failed to execute DB job error: 'StrategyUpdateJob' unspecified error

It seems that at a certain point, either NT, or the system gets overloaded, and NT is unable to write all the order details to the DB. I can still make orders on the DOM, but I get that scary warning sound and about 10-15 error boxes popup over my charts, which I have to close out one after the other. (erros boxes, not charts).

When this happens, I can only stop it occurring again by closing all positions, exiting NT and starting again.

It appears several others are also receiving same errors:

http://ninjatrader.mywowbb.com/view_topic.php?id=667&forum_id=7&highlight=failed+ to+execute
http://ninjatrader.mywowbb.com/view_topic.php?id=683&forum_id=10&highlight=failed +to+execute

cheers,

Antraman


00

[NinjaTrader_Dierk]

>>>Intervals on charts are shown below in previous post.
Sorry, did overlook that.

Other posts you referred to are different, since they are strategy storage related only. Your issues appears on allmost any DB-managed object (apart from orders). Also: I hadn't a report on these issues for months.

- Which instruments do you have on your charts/DOMs?
- How have you setup your strategies? (targets, stops ...)
- Which provider are you connected to?
- I suppose you are on most current NT version?
>>>It has 1GB installed. Ninja usually crashes when mem usage is around 300MB.
- How much RAM is available at that point in time?
You can contact me at dierk@ninjatrader.com.

My idea is to setup your scenario and let it run for a while on a test machine -> need complete setup info.

Thanks

[Antraman]

Will send a pm with more details.

> - How much RAM is available at that point in time?

Not sure what you mean by this, or how I should check it...

cheers,

Antraman


0

[Antraman]

PM sent.



0

[NinjaTrader_Dierk]

I suggest sending me a mail as by mail address below

Thanks