Market Analyzer Error

[Greg1]

I get fairly frequent (multiple times a day) "out of memory" exception errors when using the market analyzer. It appears to be accumulating memory eventually leading to this condition. I have also noticed that when using the "reload script" feature it seems to add memory at that point also.

Only way to solve things is to eventually shut Ninja Trader down completely and restart it from scratch.

This is on the latest beta (beta 9 of NT 6). Is this a known issue ? If not, what additional information can I provide ?

Thanks

[NinjaTrader_Dierk]

Lets try to figure out some details:
- are you using custom indicators?
- if so, could you please remove those custom indicator columns from the market analyzer and try to reproduce the problem?

Thanks

[NinjaTrader_Dierk]

Today we fixed some memory issues which likely could have caused the trouble you experienced.

Please:
- uninstall NT6 via control panel
- install custom build from here: http://www.ninjatrader-support.com/ninjatrader/nt6.custom/NinjaTrader.Install.msi

[Greg1]

It does seem better as of tonight. I am able to work on my indicators and reload them without running into persistent memory related issues.

The big test will be on Monday when the indicators update as the market is live. I will post a further update at that point.

There are some other suggestions related to market analyzer that I will post under a different topic.

Thank you so much for looking into this issue.

[NinjaTrader_Ray]

Thanks for reporting back Greg.

[Greg1]

I've got a further update. I still get periodic out of memory exception errors (although not as many as before). I get these errors even though there is plenty of available memory in Windows Task Manager (my WinXP machine has 4GB RAM total). There are a few things that I'm noticing as I do testing and look at memory usage, time utilization as I go step by step:

1) First off I started with a clean db (removed the old one) and an empty workspace. Started NT 6, beta 10+ (install from today that fixed an issue with a performance issue on loading data with different timeframes). Connected to ESignal. At this point memory utilization was about 70MB. Note that I don't think this latest build is causing this issue as I was having these issues periodically (out of memory exception errors) during the last week on the official beta 10, just haven't had time to do more detailed testing until today.

2) Loaded a market analyzer window and setup 5 columns (MA1000, MA500, MA200, MA50, MA20) all on a daily timeframe. Loaded the default DOW and Nasdaq lists. It took a few minutes to load the 100+ stocks and at this point memory utilization was around 115MB.

3) Closed the market analyzer window and the memory utilization went up slightly to about 118MB. The reason I mention the increased memory utilization on window closing is that I've noticed that when I have multiple market analyzer windows running and I close them one by one the memory usage goes up instead of going down as I would expect.

4) Then, I closed the app and started with a clean db again. Loaded the same market analyzer setup BUT changed all 5 MA's to an hourly timeframe. Loading the DOW list took a LONG time (15+ minutes). MUCH longer than loading the daily data. Loading the NASDAQ list took a very long time (2+ hours). Again, MUCH longer than loading the daily data. And the memory utilization in task manager is significantly higher (500MB+) when the only thing that is different is an hourly timeframe versus a daily timeframe. Note that I can go into ESignal and load the same stock, same timeframe, approximately same # of bars and that stock will load within a second but watching the same stock load in Ninja Trader takes at least 30 times longer.

Bottom line:

1) From a time standpoint, I don't understand why loading hourly data is exponentially longer than loading daily data (only variable being thedifferent time interval)

2) I don't understandwhy memory utilization is so much higher to store hourly data in market analyzer versus daily data (again same columns, same indicators). Even the memory utilization on the daily data for 5 simple indicators, small list of stocks seems very high (my calculations showed about 45MB memory usage)

3) I get intermittent out of memory exception errors (cannot reproduce consistently or at the same point) even though my machine shows plenty of available memory. But, it really seems like an awful lot of memory is being consumed for simple indicators, relatively small list of stocks

4) Closing the market analyzer windows doesn't seem to be releasing the memory that window is using. It finally gets released when I close the NT app or I get some sort of exception (many times I'll get an out of memory exception error when I try to close the market analyzer window, other times I'll get out of memory exception errors when loading the list of stocks or when the indicators refresh during the live trading day).

Note that I also have quite a bit of experience with TradeStation Radar Screen (similar capabilities as Market Analyzer)and I ran through the same type of scenario there and I don't see any of these issues:A) list of stocks with same MA's loads extremely quickly; less than 1 minute for both timeframes with 500 stocks instead of the 130 used here, and B) memory utilization is significantly lower.

Again, I am getting LESS of these memory related errors than I was getting before but the speed issues and the memory issues described here are still causing problems. So, I think further work is needed in this area of the application so that it is more stable and more responsive.

Let me know if I can provide further information or any additional assistance.

[NinjaTrader_Dierk]

1+2) Hourly bars are built of minute bars which are kept in memory. Daily bars are built of daily bars which are far less.
3) .NET is a garbage collection system, meaning you really can not look at the memory utilization to detect potential deficits, since the .NET runtime may decide to not free memory (which is not in our control). However, the .NET runtime will free memory, as there no longer is other memory available. In case no more memory is available you will get execptions you experience which then is
a) a bug in NT, since objects are not freed (and garbage collector has no chance to free the used memory)
b) you just use too much memory since large data series are loaded.

[Greg1]

It sounds like I'm out of luck? This would mean I would need to stick with TradeStation as there Radar Screen doesn't have these speed issues or memory utilization issues? I'm really surprised there isn't something that could be done on both of these fronts as supposedly C# is much more efficient than a 4GL language which is what TradeStation uses.

I definitely understand why hourly data takes more time than daily data but it is an exponential difference. I mean to load hourly data for 100 stocks took over 2 hours. I can walk through those hourly charts using ESignal one by one and it takes no more than a second each especially if I'm doing something simple with indicators like just moving averages. It's faster for me as a human to walk through the stocks one by one scanning for opportunities than using a computer. That just doesn't make sense to me.

The other thing I should point out is that 5 minute data loads exponentially faster than hourly data. The 5 minute data is about the same response speed as the daily data.

Also, note that when I get the out of memory exception error the Windows Task Manager is showing over 1.5GB of RAM still available and I can launch other applications just fine when I get that exception error.

Hopefully I'm not understanding your reply correctly. I can't believe I'm exceeding the limits of what NT can do (this is a fairly simple scenario). Before I give up on NT can you just confirm there is no hope for me on fixes to this issue.

Thanks

[Greg1]

Doing a bit more testing on other time intervals:

1) 5 minute loads about as fast as daily

2) 15 minute is also fast

3) 30 minute is slower than either 5/15 but still much faster than hourly

Just seems like all the "minute" based intervals should have about the same level of performance.

[NinjaTrader_Dierk]

Let's try to clarify:
- NT internaly only knows 3 types of data series: 1 tick, 1 minute, 1 day. All other data series are built of the base data series, meaning your hourly data series below is built of a 1 minute series. This apporach makes sense, since these 3 series types are the common denominator on all popular data providers.
- Your comparison of load times between NT and eSignal exposes a difference, since likely eSignal will download hourly data (I really don't know) but NT has to download 1 minute series which could be slow on eSignal.
- Can you please post (or PM me) the market analyzer template which takes you 2 hours to load. The other one you sent me via PM the other day loads in minutes.
- It makes a huge difference if you first time download e.g. hourly data for an instrument on a market analyzer, since then the complete data sets needs to be loaded. But on reconnect only the new data since last connect will be downloaded.
- According your post below NT uses about 130 MB to manage 100 instruments on a market analyzer. This is normal and by no means an indicator of excessive memory usage.
- In case you again experience out of memory exceptions based on latest custom build I provided to you I would need to know again exact setup. Also: As you experience the problem, please go to task manager and let me know what the ememory usage is.

I will follow up on other issues via PM shortly.

Thanks

[NinjaTrader_Dierk]

Zoltran, Greg1,

My findings so far:
a) I just increased the speed of the eSignal adapter up to 10 times. This should make a significant difference.
b) There is a bug in how data is loaded in the market analyzer as historical data re requested for the first time: basically NT fails to go far enough back and then has to re-request the data series again. This issue will be fxied shortly.
c) as you today connect to eSignal and have a market analyzer open, NT will detect that the last data downloaded is of thursday. NT then will reload thursday (the last day of recorded data to make sure there is no data missing), friday and today/saturday resulting in 3 calendar days to check for data. Since there is no calender/holiday concept in NT, NT is not aware of dates where there is no data. Howevere, this issue will become negligible which fix on issue (a).

[NinjaTrader_Dierk]

Ok, let's take next step: I created a new custom addressing issues (a) - (c) below.

This custom build adresses some memory leaks issues on the market analyzer also.

Please:
- uninstall NT6 via control panel
- clear out your browser cache
- install latest custom build from here: http://www.ninjatrader-support.com/ninjatrader/nt6.custom/NinjaTrader.Install.msi

[Greg1]

Initial impression is a major speed improvement !! I will do more testing over the weekend on different scenarios and will post further results. Thank You.

[zoltran]

NT crashes for me with this build if I try to load multiple MktAn's at once.
This happens at startup as I have 3 MA's in this test

No feed is running at this time.

Deleting the default.xml workspace file, allows NT to restart

Opening MA's individually and adding a list seems fine,

But opening 2 or more, and then adding a instrument list to both of them quickly, so they are loading at the same time seems to be a way to consistently recreate the crash

I have take screen shots of the registers, dump etc and will attach as a word document below.

if you know of any way to grab these crash files as text, pls let me know.
Windows would not let me copy/paste them

[NinjaTrader_Dierk]

I can reproduce issues and need to think about it. I will come back to you.

[NinjaTrader_Dierk]

Guys, working my way through...

I would want to learn something: Why would you need several market analyzers running? Why could you not just throw all columns on all timeframes you need onto one market analyzer?

Thanks

[zoltran]

One reason is screen space/layout and another is for separation of function by account type.

For example, I have one small list in the foreground all the time for primary R/T emini trading, and wanted another for my slower paced swing and covered call portfolio. This 2nd one would be moved off from the primary screen.

[Greg1]

My reason is similar to Zoltran in that I have very short-term trading (futures) where I look at various indexes like Tick, Advance/Decline, various eminis. I have one monitor where I keep all my short-term trading charts, scanners, order placement, etc.

On the other monitor I have charts that I use for longer-term trading anywhere from an hour to multiple days/weeks. Originally when I started to use the market analyzer I did have all column timeframes for my longer term trading in the same market analyzer and was having problems with speed/memory utilization (I think many of those issues have been addressed over the last week to two), so I broke things down into smaller chunks so I could try to use the scanner during the day on a few timeframes but not use all the timeframes I desired.

Since various issues have been addressed recently, I am going to take all my longer term trading timeframes and put them back into a single market analyzer and see how that works out. If it does, then I would be at the point where I need two market analyzer windows (1 for futures trading and 1 for options/longer term trading). I'll load up my options market analyzer with the various timeframes (weekly, daily, 4 bars per day, hourly, 30min, 15min, 5min) and see how the speed/memory usage is (a big test would be during the trading day tomorrow).

Hope this explanation helps.

[Greg1]

Just to confirm my current understanding. If I load up a single market analyzer with different timeframes (i.e. hourly, 30min and 5min) the only columns that will get recalculated every 5 minutes are the 5 min columns? Is that correct?

[NinjaTrader_Dierk]

Guys, thanks for clarification.

Greg1: Yes your understanding ist correct.

I'll keep you posted.

[NinjaTrader_Dierk]

Greg1: I stand corrected. We recently changed the default calculation to CalculatyeOnBarClose=false, meaning calculation is triggered on every tick.

Please set it to True on the column properties if you want to trigger calculation only once ber bar.

[zoltran]

Dierk

I can confirm that the esig loading time is signifgantly reduced!

Is there any way you could apply that magic to the OpenTick adapter?
I know they are slow, but maybe ..... ;-)

Thanks for all the help over the past week or two.
Much appreciated.

[NinjaTrader_Dierk]

Sorry no. The delay you experience on the OpenTick adapter is related to their server speed. Nothing we can do on our end.

[Greg1]

Just wanted to check in and see how things look for a fix to this in the next beta possibly? Thanks.

[NinjaTrader_Ray]

Hi Greg1,

Be out after market close tomorrow.

Ray

[Greg1]

Thanks for beta 11. With this build I have been able to have 2 market analyzer's load upon program restart (first time I've been able to do that).

This beta appears to have quite a number of significant improvements in the areas of performance, memory utilization and general program stability. The live markets will be another good test and will let you know if I have further problems.

Thank you for all that you do.