NT7b18 crashed without reason, complete freezing

[metaltrade]

Yesterday, while I was in a trade on a live account NT crashed without error message, it just froze it was completely unresponsive. I was planning another trade. CTRL-ALT-DEL took me out of it. When I restarted NT and loaded up the charts I was in loss because I needed to enter the market.

I can email the log file but I have no clue if it will say anything since it was a complete freeze, I wasn't doing anything, not clicking anything, just staring at charts since several minutes.

This Ninjatrader experience is getting very expensive for me.

[xTrader1]

But we don't know about. Sometimes developers don't know either. :D

More seriously, live trading is not recommended on Beta.

[metaltrade]

Yeah I tried 6.5 that even crashed more when I loaded up my 9 charts

[xTrader1]

And I'm sure eventually they will get a stable application. From my point of view it's worth to wait.

[NinjaTrader_Jason]

Hello metaltrade,

Sorry to hear about the issue.

Can you please send me your log and trace files for the date the issue occurred. You can find log and trace files at the following locations.

Start Menu--> My Documents--> NinjaTrader 7--> Log--> log.20100720 for today. (log.yyyymmdd for any other date)

Start Menu--> My Documents--> NinjaTrader 7--> Trace--> trace.20100720 for today. (trace.yyyymmdd for any other date)

At the time of the crash, did you receive an error message or did NinjaTrader just became unresponsive without an error?

You can send your log and trace files to support[at]ninjatrader[dot]com. Please reference this forum thread and put 'Attn Jason' in the subject header.

[powermoney]

Yesterday, while I was in a trade on a live account NT crashed without error message, it just froze it was completely unresponsive. I was planning another trade. CTRL-ALT-DEL took me out of it. When I restarted NT and loaded up the charts I was in loss because I needed to enter the market.

I can email the log file but I have no clue if it will say anything since it was a complete freeze, I wasn't doing anything, not clicking anything, just staring at charts since several minutes.

This Ninjatrader experience is getting very expensive for me.

I have had the same freezing issues as described above - though trading in simulation (SIM101)

[NinjaTrader_Jason]

Hello powermoney,

Can you please tell me if you received an error message or did NinjaTrader just became unresponsive without an error when you experienced the issue?

If you received an error that stated to send us your trace file, can you please send me your trace file.

Start Menu--> My Documents--> NinjaTrader 7--> Trace--> trace.20100722 for today. (trace.yyyymmdd for any other date)

You can send your trace file to support[at]ninjatrader[dot]com. Please reference this forum thread and put 'Attn Jason' in the subject header.

[dave1992]

It happened to me as well, today, with two live trades on. No errors. No clues in the log/trace. Beta 18, Mirus/Zen. 5 strategies running.

[NinjaTrader_Jason]

If you experience freezing/unresponsiveness without error, you will need to reduce data/resource request to avoid it and improve performance.

Check if you are running workspaces in the background (these will request data & resources as well) and close them. Please restart NinjaTrader and go to File-->Workspaces and check what workspaces are listed below 'Save Workspace As'. The workspace with the checkmark is the workspace opened on your monitor. Any additional workspaces listed, are opened in the background. Select a workspace without a checkmark and subsequently go to File-->Workspaces-->Close Workspace. Repeat this process until only one workspace is opened.

Close some charts, SuperDOM's or other NinjaTrader windows. Reduce the amount of indicators in your chart or set 'Calculate on bar close' to True. Or reduce the amount of historical data in your charts by configuring the Format Data Series window.

More tips can be found at the following link.
http://www.ninjatrader.com/support/helpGuides/nt7/performance_tips2.htm

[dave1992]

I'm sorry but you use the CPU/mem/resources argument ad nauseum and as far as I can tell in 98% of cases it just doesn't hold water.

[NinjaTrader_Jason]

Hello dave1992,

Market volatility plays a role as well, so the same setup runs fine most of the time. However when there is increased volatility, you experience lagging/freezing behavior.

If you experience such behavior without any error or other clues, it usually means your NinjaTrader setup requested too much memory/CPU frm your PC.

Did you notice anything else when you experienced the freezing behavior? If you restart NinjaTrader and use the same setup, does the issue persist once connected to your data feed?

[powermoney]

I agree with dave1992. During normal operation and during the freeze, I observed Ninjatrader.exe process using no more than 5% cpu .... and the overall CPU usage was low. The machine running Ninja has 8 processors w/ average 20% usage and no memory issues.

[NinjaTrader_Jason]

Ok, can you please check if you can reproduce this behavior. Do you notice any similarities between the occurences so far?

If so, please send me your workspaces, so I can test the same setup on my end.

[dave1992]

Jason,

You'll find more info in this thread http://www.ninjatrader.com/support/forum/showthread.php?t=31291

Not every hang is reproducible and there are other ways to investigate and fix this.

Market volatility plays a role as well, so the same setup runs fine most of the time. However when there is increased volatility, you experience lagging/freezing behavior.

This is almost certainly not market volatility. If it was, the following things would likely be true:

1. It doesn't tend to happen through the night, during low volatility
2. It is fairly easy to reproduce at exactly the same time during market replay

If you experience such behavior without any error or other clues, it usually means your NinjaTrader setup requested too much memory/CPU frm your PC.What evidence do you have for that assertion? In probably more than 90% of cases I have seen on this forum, there has been no evidence for this. Can you describe exactly what typical symptoms are on someone's PC when Ninja does this? Too much memory/cpu implies that the PC is running out of resources. There would be other evidence. But there isn't. Everyone describes it exactly the same. Ninja just stops responding entirely - no other clues - and has to be killed by the OS.

[NinjaTrader_Jason]

NinjaTrader can become unresponsive to any action you try to perform. Or it does respond to actions, but in a delayed manner.

You can see 'not responding' in the Control Center header.

[dave1992]

Sorry Jason, you'll have to expand on that - I've no idea which part of my post you're responding to.

[NinjaTrader_Jason]

Sorry about that - it refers to the question below in your last paragraph.

"Can you describe exactly what typical symptoms are on someone's PC when Ninja does this?"

[powermoney]

At no time during the freeze did 'not responding' show in the Ninjatrader control center window or in the windows task manager. The Ninjatrader.exe process continued to run - only the application GUI was frozen in place.

Also, I should mention that during one freeze I left it alone for 1 1/2 hours to see if it would clear itself up and it did not.

[dave1992]

Sorry about that - it refers to the question below in your last paragraph.

"Can you describe exactly what typical symptoms are on someone's PC when Ninja does this?"

Well in that case you've just proved that it's not the performance issue, since in our examples nothing happens - no messages, no 'not responding' message, no slowing down of response, no slowing of the OS, no build up of CPU/mem

Now can you finally take seriously that it's not performance related?

[powermoney]

From a programmer's perspective this almost seems like an endless loop.

Another thing that I noticed was that the memory usage of Ninatrader.exe slowly continued to increase after the freeze of the GUI interface.

[NinjaTrader_Ray]

metaltrade, powermoney, dave1992,

I would like to first see what the common items each of you might share during the locks ups:

- Did you all have NinjaScript strategies running when the freeze occurred?
- Real time or on market replay or both?
- Did you all have working orders open? If yes, Sim or live account?

We did release B19 today and it would be great if you could all upgrade to this version so we know we are testing against the latest code base.

First, thanks for posting your findings here. Second, thanks in advance for working with us on this.

[toddaclark]

I just had NT7-B18 totally lock up on me, even in this "low volatity time frame".

I was on live data at 4:30pm or so EST, I had no open orders and no strategies running. I was just switching between charts in my workspace. There are 14 charts in my workspace. I just switching from one chart to another and NT would hang up for a few seconds before switching. Then I tried to switch to another chart and it hung up for a minute so I opened the windows task manager and watched memory climb from 500k to 1.3k over a period of 5 minutes while NT7 was frozen.

I finally killed it and started over but I don't think that fixed my problem.

Ryan told me to re-create the account connections, re-create all of my workspaces, etc. I'm willing to do all of that but do you think that is really the source of the problem or the solution to whatever is causing the problem. Re-creating all of my workspaces, etc will take over an hour and I don't want to go thru that if it's not really the issue.

Any suggestions would be appreciated.
Thanks,
Todd

[NinjaTrader_Ray]

I just had NT7-B18 totally lock up on me, even in this "low volatity time frame".

I was on live data at 4:30pm or so EST, I had no open orders and no strategies running. I was just switching between charts in my workspace. There are 14 charts in my workspace. I just switching from one chart to another and NT would hang up for a few seconds before switching. Then I tried to switch to another chart and it hung up for a minute so I opened the windows task manager and watched memory climb from 500k to 1.3k over a period of 5 minutes while NT7 was frozen.

I finally killed it and started over but I don't think that fixed my problem.

Ryan told me to re-create the account connections, re-create all of my workspaces, etc. I'm willing to do all of that but do you think that is really the source of the problem or the solution to whatever is causing the problem. Re-creating all of my workspaces, etc will take over an hour and I don't want to go thru that if it's not really the issue.

Any suggestions would be appreciated.
Thanks,
Todd

Todd,
Thanks for reporting. No need to re-create workspaces or account connections at this time. At this time, we have no clue what the root cause would be but hopefully some answers lie in your workspace.

Could you kindly do the following:

- Start NT
- Help > Mail To Support
- Send us workspace, trace and logs
- Please include in this email answers to the questions in my prior post
- Attention to Josh

We will follow up with further questions once we have this initial information and files.

In the meantime, please upgrade to Beta 19 which was released earlier today.

[NinjaTrader_Ray]

metaltrade, powermoney, dave1992,

I would like to first see what the common items each of you might share during the locks ups:

- Did you all have NinjaScript strategies running when the freeze occurred?
- Real time or on market replay or both?
- Did you all have working orders open? If yes, Sim or live account?

We did release B19 today and it would be great if you could all upgrade to this version so we know we are testing against the latest code base.

First, thanks for posting your findings here. Second, thanks in advance for working with us on this.

Info I collected from earlier post:

Dave1992 - Live Trading, B18, Zen-Fire, 5 Running Strategies

[toddaclark]

Ray..

I've sent the email to Josh. I will upgrade to the latest version and wait to here from Josh via email.

I use zenfire. I use windows XP.

Also, I was using the workspace titled "current1-smaller" when I had the issues but I closed that and opened "current1" and NT still seemed to be hanging up.

Thanks,
Todd

[azuul]

I had the same problem yesterday:

NT 7.0.0.18, live trading /IB/ forex: - after about 2 executed trades NT froze completely in place, I could not move any window or disable the strategy, cpu/mem was not an issue.
It was a simple crossover for testing purposes and I remember one strange detail, maybe it is important:
NT was frozen showing a short position, while in TWS I had a long position!
That means NT must have submitted a close + buy order wich arrived in TWS and it froze before /while trying to show the change in the control center. I think it was that way, not the other way round.

I did not encounter this behavior with live data + simulation any time before.

I was using IB datafeed, TWS was behaving fine, no error messages or other general performance issues.

I tried the same thing and it froze again. ( I had only one strategy running the 2nd time and only one datafeed)

Today I tried a different strategy live and no freeze at all.

[toddaclark]

I had the same problem yesterday:

NT 7.0.0.18, live trading /IB/ forex: - after about 2 executed trades NT froze completely in place, I could not move any window or disable the strategy, cpu/mem was not an issue.
It was a simple crossover for testing purposes and I remember one strange detail, maybe it is important:
NT was frozen showing a short position, while in TWS I had a long position!
That means NT must have submitted a close + buy order wich arrived in TWS and it froze before /while trying to show the change in the control center. I think it was that way, not the other way round.

I did not encounter this behavior with live data + simulation any time before.

I was using IB datafeed, TWS was behaving fine, no error messages or other issues.

I tried the same thing and it froze again. ( I had only one strategy running the 2nd time and only one datafeed)

Today I tried a different strategy live and no freeze at all.

What is TWS?

[NinjaTrader_Josh]

azuul,

Do you still have the log/trace files for the incident you described? Can you please send them into support[at] ninjatrader [dot] com with Attn Josh and a link to this thread? Thank you.

Where were you viewing the short position from? A chart? Positions tab in NT Control Center?

When you say you tried again, does that mean you tried again just now and got it to reproduce? If so, can you send us that particular strategy you are using (the original one that has seen freezing 2x, not the different one that was working)? Also, what version of TWS are you using?

toddaclark,

TWS is IB's Trader's Workstation client that we connect to when people establish IB connections. FYI: I sent you an email with clarification emails needed for your case.

[azuul]

I just sent you the trace and log

I am not 100% sure if I was long or short in TWS, but I am rather sure that the TWS position was the opposite position from what was shown in Ninjatrader control center /strategy tab which I had visible.
I cannot say exactly how/when, but I realized at one point that NT GUI was frozen, I think after the 2nd crossover, and that
the position I had for real was different from what the frozen NT control center showed.
I do not remember if I had "immediatelly submit live working orders" activated or not
that time - if that is of any relevance.
Maybe the discrepancy in positions has other reasons, but it came to my attention and I was wondering about it.
The strategy was running on a 20 seconds bar chart.

I tried it a second time shortly afterwards that day and it happend again.

I did not try to reproduce the same scenario yet, but a different strategy yesterday ran without freezing for a longer time, (1 Minute bars)

The strategy I do not want to send, but it is the same I used on other instruments /timeframes etc. - in live data simulation I never had any freeze of that kind during that time, so the strategy itself does not seem to be the problem. It is not big or complex
There are no print" " statements in it, CalculateOnBarClose = true and traceorders is not activated.

TWS Build 900.6

I hope this helps to determine the source of the problem.
Thanks for your efforts!

[powermoney]

metaltrade, powermoney, dave1992,

I would like to first see what the common items each of you might share during the locks ups:

- Did you all have NinjaScript strategies running when the freeze occurred?
- Real time or on market replay or both?
- Did you all have working orders open? If yes, Sim or live account?

We did release B19 today and it would be great if you could all upgrade to this version so we know we are testing against the latest code base.

First, thanks for posting your findings here. Second, thanks in advance for working with us on this.

Ray,

Strategies were running during the freeze
Real Time
Working orders were open - in Sim

[NinjaTrader_Ray]

Hi All,

It would seem that 4 of 5 of you were all running NinjaScript Strategies. Next steps are:

- Please make sure you upgrade to B19
- Next time you experience the the freeze please confirm that NinjaTrader is unresponsive (does not respond to any mouse action, can't move windows etc...)

a) Your CPU is pegged consuming high %
b) Your CPU usage is really low

Thanks in advance.

[azuul]

Strategies were running during the freeze
Real Time
Working orders were open - in Sim

@ Powermoney : do you mean that strategies were still being executed (=running?) by NT while the GUI was frozen? (that would explain my experienced discrepancy in positions)


The GUI was 100% frozen in place, unresponsive to any action like a windows background picture (Different from other freezes like during optimization with print statements in the code - then NT would freeze but become grey when I try to move an NT window and eventually windows asked if I want to close the application. This was not the case here.)
Windows 7 and TWS were running smoothly at any time. Not performance related in this case.

[powermoney]

No - To clarify, strategies were running until the point of the freeze. An order placed by the strategy moments before the freeze may not have been properly filled. After the freeze, the strategies were not executing and (NinjaTrader) had lost control of the current working orders. They remain in place as is - unmanaged. Positions have to be closed manually after restarting NinjaTrader.

[powermoney]

Hi All,

It would seem that 4 of 5 of you were all running NinjaScript Strategies. Next steps are:

- Please make sure you upgrade to B19
- Next time you experience the the freeze please confirm that NinjaTrader is unresponsive (does not respond to any mouse action, can't move windows etc...)

a) Your CPU is pegged consuming high %
b) Your CPU usage is really low

Thanks in advance.

I upgraded to B19.

The freeze occurred today at 12:01:44 PM.
This is a different computer than previous.
There were only 2 charts open and 1 strategy running at the time.
The CPU usage is normal - around 10%
The Ninjatrader.exe process CPU % was 2-3%
The Ninjatrader.exe process was using 215704k memory

Looking at the trace files, this and previous freezes occurred shortly after a trade - and the last entry in the trace files were:

Yesterday - Machine 1 w/ B18:
2010-07-22 08:30:07:619 (Esignal) Cbi.ExecutionEventArgs.Process: Execution='c0ce2e4205f741b08e528d7fc1527c1a' Instrument='CL ##-##' Account='Sim101' Exchange=Default Price=76.79 Quantity=1 Market position=Short Operation=Insert Order='68f25acd1bc541f9a8b6d398b0620033' Time='7/22/2010 8:30:07 AM'

Today = Machine 2 w/ B19:
2010-07-23 12:01:44:020 (esignal) Cbi.ExecutionEventArgs.Process: Execution='e382bd5db5b242e5bc6b9ae506252512' Instrument='YM ##-##' Account='Sim101' Exchange=Default Price=10307 Quantity=2 Market position=Long Operation=Insert Order='487e63e194364a2c9c8f2a5d6f70b6b5' Time='7/23/2010 12:01:43 PM'

Powermoney

[NinjaTrader_Ray]

powermoney,
Could you send us an email with contact information? We would like to follow up via telephone.

[dave1992]

Of all the freezes I've seen -

- all were while running multi time frame strategies in the strat tab, but I don't use charts particularly anyway.
- all happened during market replay, except for one which was live
- I'm not sure if all the replay ones had a position open when the freeze happened, but I think it's possible
- the live freeze did have two open positions - since my positions usually last 1-5 mins, and are 5-20 trades per day, it is possible that the freeze is more likely soon after a trade, or while a position is open.
- all had no clues in the logs/trace
- all had low cpu/mem
- they may be more likely at higher replay speeds
- I don't think I've seen one while running just one strat, but I don't do that often anyway.

I downloaded B19, and had one complete 20 day replay run through of 5 strats at 500x speed without a freeze. In my experience that would usually have frozen at some point. (By the way, do multiple running strats get to take advantage of multi core, or is it just backtesting?)

regards
Dave

[NinjaTrader_Ray]

Of all the freezes I've seen -

- all were while running multi time frame strategies in the strat tab, but I don't use charts particularly anyway.
- all happened during market replay, except for one which was live
- I'm not sure if all the replay ones had a position open when the freeze happened, but I think it's possible
- the live freeze did have two open positions - since my positions usually last 1-5 mins, and are 5-20 trades per day, it is possible that the freeze is more likely soon after a trade, or while a position is open.
- all had no clues in the logs/trace
- all had low cpu/mem
- they may be more likely at higher replay speeds
- I don't think I've seen one while running just one strat, but I don't do that often anyway.

I downloaded B19, and had one complete 20 day replay run through of 5 strats at 500x speed without a freeze. In my experience that would usually have frozen at some point. (By the way, do multiple running strats get to take advantage of multi core, or is it just backtesting?)

regards
Dave

Thanks. Multiple running strategies will not take advantage of multiple cores. Please stay tuned, we will likely provide you a custom installer since we have identified something that might cause a dead lock in Market Replay.

[azuul]

Version 7.0.0.19

Complete Freeze

I was simulating live 4 strategies (from chart) - single timeframe each - IB is Dataprovider
CPU idle 1-3%
Memory consumption of NT about 250MB that time, not changing during freeze.
Nothing special in the logs and trace, last entry in log prior to freeze was an execution.

Very likely same issue as during first freeze with 7.0.0.18.

Could the freeze be related to latency issues causing a deadlock ?
Had these issues only with IB Data so far.

Sent trace and log (freeze was around 1:39 PM)

[powermoney]

azuul, what instruments were you trading/simulating when NT froze?

[azuul]

only Forex: EUR/USD

Freeze occured rather shortly after starting the strategies, since then I am running the same setup/strategies without problems.
I was only connected to IB

[NinjaTrader_Ray]

We believe we have figured out what the issue is. Its a deadlock situation. We are currently internally testing this and if it looks good, we will provide a custom build for those of you have experienced this issue. Thanks for your patience and we will update this thread once we are ready.

[dave1992]

We believe we have figured out what the issue is. Its a deadlock situation. We are currently internally testing this and if it looks good, we will provide a custom build for those of you have experienced this issue. Thanks for your patience and we will update this thread once we are ready.

This is good news. Thanks.

Could we learn some lessons from this whole issue? This was written on this forum over two weeks ago:

"If I had to have a guess, I'd say it's a database or resource deadlock. That certainly fits in with the symptoms"

It was completely unacknowledged, and since then we've had to argue against numerous posts saying it's your cpu/mem etc. It was very frustrating.

thanks

[NinjaTrader_Ray]

This is good news. Thanks.

Could we learn some lessons from this whole issue? This was written on this forum over two weeks ago:

"If I had to have a guess, I'd say it's a database or resource deadlock. That certainly fits in with the symptoms"

It was completely unacknowledged, and since then we've had to argue against numerous posts saying it's your cpu/mem etc. It was very frustrating.

thanks

Sorry for the frustration, for sure it was not intended. We will try to do a better job in the future.

[NinjaTrader_Ray]

We have a custom build that we would like to provide to those of you in this thread that have experienced the freezing.

Please send an email to our support team, attention Josh and reference this post and provide your forum ID. We will provide further instructions.

[paulg]

Ray, do you still provide a custom version --or the latest beta 22 fixed those issues?

And you guys, dave1982, azuul and powermoney are you experiencing no more freezes?

[dave1992]

I'm on standard beta20 and I don't get it any more. It's def fixed for me.

[paulg]

I'm on standard beta20 and I don't get it any more. It's def fixed for me.
Thanks for your feedback. Unfortunately I'm on beta 22.

[NinjaTrader_Josh]

This issue has already been resolved several betas back as dave1992 pointed out. Most likely then the issue for you paulg would be strategy related. Please check your CPU load to ensure the strategy is not simply pegging the CPU with calculations. Please check memory is not being chewed up by some custom resources the strategy is using or some incredibly high amounts of loaded data.

[paulg]

This issue has already been resolved several betas back as dave1992 pointed out. Most likely then the issue for you paulg would be strategy related. Please check your CPU load to ensure the strategy is not simply pegging the CPU with calculations. Please check memory is not being chewed up by some custom resources the strategy is using or some incredibly high amounts of loaded data.

I'm not using any strategy, actually.
CPU is an i7 860 and the 4 Gb RAM are usually free for 2 Gb. Of the 8 logical cores, I never saw one of them busy more than 40/50% and the remaining cores at 5-10%. Also the internet connection even fast market does not hit more than 80/120 KB/s while my bandwidth is saturated at 800 KB/s (7 Mbit). Only one workspace is open with no T&S window, only one DOM, 5/6 instruments are involved and 17 charts overall. I'm using Vision 32 bit feed on XP Pro x64 SP2. The software most of the times if left alone resumes working after a few dozens of seconds or minutes as it has to retrieve 'cached data' or something like that: this is the impression.
Other times instead, there is no way to regain control of the software and I have to kill the process with Task Manager. Again during these times there is plenty of free ram and cpu cycles available.

[NinjaTrader_Josh]

Then your issue is unrelated to this thread. This thread is very specific in relation to the running of strategies.

I suggest opening a new thread as to avoid any confusion in this regard. We can assist you there. What exactly are you loading chart wise? What kind of series and how many days back? Any indicators on the charts? CalculateOnBarClose = true or false?