Memory bloat just from opening NT7

[bluelou]

I'm getting memory bloat of nearly 1 GB just from opening NT7. Here's a pic from task manager. Also, I've sent my log and trace files to support. Please advise on how to stop this.

Every time I try to run a live strategy the memory blows up to >1 GB and NT shows up as "not responding". Now, even after shutting down NT7 and my PC and restarting the memory has remained bloated - and this has been for several hours.

Thx,
Lou

[NinjaTrader_Chip]

Hello bluelou,

Thank you for your post.

Please re-send your Log & Trace files and include this thread/link in your email.

Once we have an opportunity to review them & see what might be taking place here, we will be able to respond.

Thank you,

[bluelou]

Chip,
I tried resending the log & trace files but NT just hung there for 30+ min. I couldn't even use my browser. Any other ideas?

[NinjaTrader_Chip]

In this case, just isolate the individual Log & Trace files for the last day you experienced this issue.

You can find these in your (My) Documents folder in the sub-folders: (see screenshots)

1. NinjaTrader 7 > Trace > trace.2010MMDD.txt

2. NinjaTrader 7 > Log > log.2010MMDD.txt


Just attach these two text files to you email & send it to support AT ninjatrader DOT com


Once we receive them we will evaluate & try to understand what is going on.

[bluelou]

Chip,
Yesterday's log file was 342 MB and the trace file was 6 MB. How do I send this to you? Even YouSendIt.com won't allow me to send a file of this size w/o paying them $10. Do you know of another way to get this file to you?

[NinjaTrader_Kyle]

Hi bluelou,

Please compress the files in to a .zip or .rar and then upload using the free web service http://senduit.com/

[bluelou]

Kyle, I just sent the trace files since that was only 6MB.

senduit has a 100MB limit so that won't work for a 342 MB log. A few other services max out at 300MB free. I just looked at the log and I don't think it will be of that much help. It has over 1/2 milllion lines of print statements from my backtests but that's probably it.

I'll see if I can split the file in notepad++ or something like that.

[bluelou]

BTW, I've been using MSVS 2005 to debug a live strategy in NT7. During the entire process, even after terminating the MSVS debug process, NT7 shows "not responding". That means the only way for me to end the debug process in NT7 is to terminate the program in Task Manager. Is there a cleaner way to terminate debugging in NT7?

Also, I tried sending a zip version of the log file using senduit. I think I might have crashed their site during the upload b/c now the site is reporting technical difficulties.

[NinjaTrader_RyanM]

Hello bluelou,

The size of your log file indicates there is some issue with your custom items. This is a text file meant to display debug information related to orders. If it has grown over 300 MB, then there is a lot of unnecessary logging and may explain why you're having memory issues with NinjaTrader.

Your trace file shows this line, repeated for many pages and may help you isolate what's happening here.

2010-08-02 14:37:32:954 ERROR: OBU Test

[bluelou]

Ryan,
I use the log as a place to monitor the output of my custom indicators. The log info you're seeing is mostly from StrategyAnalyzer output. The exception to this is where you see OBUTest - (OnBarUpdate test) - which is a test for live trading.

Are you saying that using the log to monitor strategy output will cause NT memory to bloat? If my indicator output should not go in the log then where do you recommend I print it to?

[NinjaTrader_RyanM]

Hi BlueLou,

300 MB of text output per day is not typical and may cause issues loading. Using the log tab for monitoring output is fine, but something in your code or simply a large amount of tests is generating way more output than is probably useful.

If you have the need to output this much data related to your strategy analyzer tests, a better approach may be to export to an external file.

You can use the principles here:
http://www.ninjatrader.com/support/forum/showthread.php?t=3475

If the output is generated by standard tests and you have no control over where it goes, then you may need to manually monitor this file. If it gets too large transfer the contents outside of NinjaTrader's internal files.

[bluelou]

Ryan,
When you say a large log file "may" cause issues loading are you guessing that this is a problem or is this a known issue? If so, I wasn't aware that the size of the log file could affect performance.

Thx,
Lou

[NinjaTrader_RyanM]

Hi Blue,

It's your tests that are generating such large output.

NinjaTrader reads and writes to this log file all the time. This isn't a known NinjaTrader issue - it's a tennant of computing that larger files loaded into memory will impact performance.

You're using an essentially open programming language that allows you to create tests that can potentially tax your system to the the extreme. Let us know if you feel one of your scripts is generating this output abnormally or if it's just related to the number or type of tests you're running.

[bluelou]

Ryan,
I think you've misunderstood my last comment. So, you're saying that each day's log file is held in memory for that day? For example, if I run 50 simulations and that generates 100 MB of log files (due to print statements I've put in my code) and then let's say I shut down NinjaTrader and then reopen it later the same day - are you telling me that when I reopen NT that it will also load the 100MB of log files into memory?

[NinjaTrader_Josh]

Correct. It is not recommended you flood the log file with NinjaScript messages that are not critical. The log tab for NinjaTrader should be reserved for critical error reporting and things like that. If you are just trying to have some output, I suggest you output it to your own txt file instead of the NT logs. Please see here: http://www.ninjatrader.com/support/forum/showthread.php?t=3475

If you don't need these outputs for storing purposes you can even just use the Output Window and completely bypass saving these messages altogether.

[bluelou]

I'll follow your recommendation but I still didn't get why my messages "spam the log file" and affect NT's performance. Is the following T or F? If, during the same day, the log file has grown to 100 MB due to my "spam", and then on the same day I subsequently shut down and restart NT, is the 100 MB of "spam" loaded into memory?

[NinjaTrader_RyanM]

Yes, it loads the contents of the log files into the program for the current day. The items you see on the log tab of the control center come directly from the most recent log file.

[bluelou]

But, if I close NT and reopen it during the same day then the log tab appears empty. That's why I don't understand how the contents of the log file effect NT memory. If, upon reopening NT during the same day, the log file were loaded into memory, I should see it in the log tab, right? So, maybe the log file isn't loaded into memory upon reboot, but, nonetheless, a large log file effects performance in some other way. Does this make sense? I'm not trying to be a pain, I'm just trying to understand what's happening so I don't repeat the mistake in some other way.

[NinjaTrader_RyanM]

Yes, you should see the contents of the most recent log file appear in the log tab. You should see this even if you shutdown and restart NinjaTrader. If you're not seeing the contents here, check the Right Click > Filter by Category selection to verify these items are all checked.

There's also a possibility that Ninja encountered difficulty opening and displaying your large file. Please try this with a more reasonable log file size.

[bluelou]

That's not what happens w/NT7.19 on my end. I just checked everything in Filter by Category and the only thing that's in the log tab is a small portion of the most recent backtest and none of the previous backtests. I guess it's one of the many "special features" of NT on my PC that isn't supposed to exist according to NT Support.

[NinjaTrader_RyanM]

Note that it only loads the contents for the most recent log file(today's file). If this still isn't working the way you expect, please send a note to support@ninjatrader.com Attn:RyanM.

Reference this thread:
http://www.ninjatrader.com/support/forum/showthread.php?t=32069

We can work over remote assistance to determine why yours is operating differently.

[bluelou]

Note that it only loads the contents for the most recent log file(today's file). If this still isn't working the way you expect, please send a note to support@ninjatrader.com Attn:RyanM.

Reference this thread:
http://www.ninjatrader.com/support/forum/showthread.php?t=32069

We can work over remote assistance to determine why yours is operating differently.


Ryan,
As I said, the current day's log doesn't load into the log tab as you've described. But, this isn't a big deal to me now that I understand that the log is somewhere in memory even if I can't see it.

However, I would like to take you up on the offer of remote assistance. As you can read in the following link, http://www.ninjatrader.com/support/forum/showthread.php?p=183296#post183296 I'm having several problems with bugs that are not being caught by the NT compiler and are not showing up when using StrategyAnalyzer, but only show up in live trading. Thus, I can't trade and I'm trying to use Visual Studio as a work-around to the shortcomings of the NT compiler. Please send me a private mesg so we can coordinate regarding remote assistance.

Thx,
Lou