Did I hit a limit in the number of symbols in Market Analyzer?

[1reason]

I loaded up about 850 stocks (all done one by one) into my list of stocks and loaded them into market Analyzer with two custom indicators and it sent my machine on to its knees.

I am using DTN as the data source and the machine is the same that I was doing the same screening with Tradestation (the program I am trying to replace)

During loading of the data if I tried to open or change a chart (on another window/monitor) it appeared to lock the system and I had to reboot.

the indicators use daily prices and they look at the last High/Low/Close data for the past two weeks.

Should I expect that NT takes more resources than TS and a bigger machine will solve the problem?

Is there a way to optimize the market Analyzer so that it doesn't have issues with that many stocks?

Am I simply running too many stocks in the program at once?

Thanks for any and all help

[NinjaTrader_Dierk]

a) There is no inherent limitation in NT which would restrict it's usage to any # of instruments.
b) Your data provider subscription (DTN) may or may not have such a limitation. Please clarify with DTN.
c) Dependant on your actual setup you may hit the wall due to PC resource excess. Please read more here: http://www.ninjatrader-support.com/HelpGuideV6/PerformanceTips1.html

[1reason]

a) There is no inherent limitation in NT which would restrict it's usage to any # of instruments.
b) Your data provider subscription (DTN) may or may not have such a limitation. Please clarify with DTN.
c) Dependant on your actual setup you may hit the wall due to PC resource excess. Please read more here: http://www.ninjatrader-support.com/HelpGuideV6/PerformanceTips1.html

Thank you for your quick reply.

I am using DTN IQ (I am thinking thats what your asking but not sure as they seem to offer a lot of different products)

I will also be trying to fine tune the process today and see how that goes. I would like to follow up with another question.

I do my scans based on the last 11 days of data and I think based on the performance tips that I could make the look back period either 12 or 11 but I do want to add one more filter in my scans. That is the 90 day moving average. is it possible to look at the last 11 days AND have a 90 day moving average filter AND only look back 11 or 12 days?

Thanks

[NinjaTrader_Dierk]

Not sure I follow. The lookback in the market analyzer goes by bars not days. It will be applied to any filter on the market analyzer.

[1reason]

Not sure I follow. The lookback in the market analyzer goes by bars not days. It will be applied to any filter on the market analyzer.

In this example that I am having issues with each bar is a day. I am thinking then that if I am looking at the last 10 bars AND want to use a built in 90 day moving average filter that i will need to look back 90 bars (assuming they are day bars) and not just the last 10 bars??

[zoltran]

Sorry to jump in here .. but MA performance is a sore point with me too.

Dierk ...
Try this example ...

Load a MA with the attached template, and add the NASD100 list (or the Dow30 if you want to start small). Do not have a feed enabled yet.
The initial load will take several minutes ..if you have any data at all in your database.

Attach to IQfeed ... ... the reload will take 5 minutes or longer. And on my machine, the nasd100 never completes.

During the two examples above, the 'Loading Data' dialog is displayed .. which pretty much cripples NT for doing anything else. For example .. you can't change a symbol or timeframe in a chart. If you do forget, and do this anyways .. there's a very good chance that NT will never complete the data-loading function and you'll have to restart it.

In comparison, using the same list in QuoteTracker, same feed ... it is faster by several orders of magnitude. I do not think it's a feed performance issue.

And .. just as a recap
I have tried this on several machines .. all exhibit the same symptoms.
The primary reason that I and many others run a MA with an indicator is to make NT collect replay data. I use two simple indicators for this .. one on Minute data, and the other on tick/second data.

It would be much simpler and less resource intensive if NT did not require an indicator column in order to do this.

I know you are adding additional traces in this area, I hope they help.
As it certainly seems like something is amiss in NT's historical data fetching method.

NT6.5.4 latest beta.

[1reason]

I played around with it all day and my indicator appears to work at about 100 symbols without a problem. its when I load up 700+ symbols that it just sits there. doesnt appear to hang or freeze if I allow it to run through the A to Z data uploading (using DTNIQ). but where the plot0 'should' change from a zero to the variable it just stays a zero.

This is even after allowing it to run for 20-30 mins. I even made it the only thing running (no charts or other software etc...)

I have tried the performance tips and changed the bars to only be the required amount. I would think that if the machine was too small that it would still work but work very slow.

Is it possible that NT really isn't meant to have a market scan of 800 symbols? I am running a dell P4 2.8g with 1 gig of ram with a quad monitor video card.

Thank you

[NinjaTrader_Dierk]

>> Is it possible that NT really isn't meant to have a market scan of 800 symbols?
No, there is no such NT internal limitation. However, dependent on your actual provider you may experience limitations.

[1reason]

>> Is it possible that NT really isn't meant to have a market scan of 800 symbols?
No, there is no such NT internal limitation. However, dependent on your actual provider you may experience limitations.

Thank yo for your reply. I am using DTNIQ. Do you think the issue is that I should change data providers? As I write this I see the loading data going on and on and on. I assumed the slowness was NT processing the data and not the data source though.

By limitation I mean that the design was not meant to have that many scans at once, not that their was some code to prevent it.

thanks

[NinjaTrader_Ray]

DTNiQ is not bad at all, changing providers likely will not provide the change in performance you are looking for.

Loading data for 800 stocks just takes a lot of time. Just load one symbol, 15 days back in a chart for 1 minute data. Maybe it takes 5 seconds? Then multiply that by 800 etc...

1GB may also be not enough...as soon as you hit that wall, you will start diskswapping. As soon as this starts, it can sit there for hours.

[zoltran]

Ray ...
Understand that there is a lot of data to load.. But that's not the whole issue. NT is rather slow doing the load. QuoteTracker, Sierra etc are all many times faster loading files from disk.

The problem is made worse as NT hangs up the UI when it's loading.. making the problem worse (yes, the dialog box is asyncronous.. so you can click on other menus .. they just don't do much until the data load is done)

[NinjaTrader_Ray]

I see. I believe Dierk is running some tests in this area and we will report back on anything we find.

[NinjaTrader_Dierk]

Ray ...
QuoteTracker, Sierra etc are all many times faster loading files from disk.


My findings are different. Please PM me at "dierk AT ninjatrader DOT com" for follow up. Thanks

[1reason]

DTNiQ is not bad at all, changing providers likely will not provide the change in performance you are looking for.

Loading data for 800 stocks just takes a lot of time. Just load one symbol, 15 days back in a chart for 1 minute data. Maybe it takes 5 seconds? Then multiply that by 800 etc...

1GB may also be not enough...as soon as you hit that wall, you will start diskswapping. As soon as this starts, it can sit there for hours.

Thank you for your reply.

I just bought a Dell DUAL Xeon 3.2 ghz with 4 gig ram and a 10K SCSI hard drive workstation. This hopefully will arrive to me by the end of next week. I would say that I really want that new computer to be able to do what I want NT to do.

What are your thoughts on that computer being able to perform the calculations of 800+ stocks?

Thanks again.

[NinjaTrader_Dierk]

>> What are your thoughts on that computer being able to perform the calculations of 800+ stocks?
Unfortunately we don't have any experience on processing this number of stocks. You would need to try by yourself.

[SirHC]

Hello,

I'm sure i will experiencing same problem of that thread.
I just want to add my thought.
1)Microsoft Access is easy to use and a good DB for small processing.
But when you need to process millions of lines, ( for example, 6 millions lines for 20 years of daily bars on 1200 stocks, just daily...), application need to be well designed but overall database must be the most. Did you do performance tests before choosing Access with SQL Server or other real DB ? What are you thinking about your choice?

2)You said it takes 5 seconds to load 10 hours* 60 min * 15 days = 9000 lines. Did you try to optimize this?

Regards and congratulations for your work.

[NinjaTrader_Dierk]

a) Jet engine performance is irrelevant for that scenario.
b) Not sure what you mean by optimize.

[SirHC]

a) you re right. "For larger database needs, Jet databases can be upgraded (or in Microsoft parlance, "up-sized") to Microsoft's flagship database product, SQL Server 2005"
So what sort of limitation can you suggest to your costumers for having a good experience of NT ?

b)I mean : Did you try to reduce that 5s? 10000 lines must be processed in less than 0.25 s, isn't it?

[NinjaTrader_Dierk]

>> So what sort of limitation can you suggest to your costumers for having a good experience of NT ?
You don't have any other option than using the JET engine.

>> Did you try to reduce that 5s? 10000 lines must be processed in less than 0.25 s, isn't it?
Sorry, not sure what math you are referring to. I suggest trying by yourself to gain your own experience on the actual implementation.

[SirHC]

I would just know how you have considered this part of your application.

Thanks for your answer.