Why does NT6 download new data when I already have history in the DB?

[zoltran]

Nt6 has a long pause/hang when connecting to either IB or Opentick.
I already have valid historical data, but it seems like NT wants to download and overlay this .

Both IB and OpenTick are slow backfillers, so this action for NT is a real pain.
And it can and does lockup NT from time to time.. especially if I get impatient and try opening another chart. This starts another backfill .. and that often locks NT up and I have to kill it via XP TaskMgr

I've attached a MarkAnalyzer template that will help show the problem.
Nothing special, just 3 columns with a 5//15/60 minute CCI
And I only have about 8 instruments in it.

Try this ..

Open 3 charts .. say YM, ER2, and 6E (euro)
Add this Mkt Analyzer

Connect to Opentick ...

IF NT survives this (50/50 for me) then disconnect and reconnect OpenTick
You get the same wait again.
Infact, you get wait just changing periods .. from 5 to 10 to 1 minutes for example.
Which is bad, as the minute data is already in the DB.

Change the symbol on a chart while it's got the 'data loading'
for the Mkt Analyzer .. usually good for a hangup.


I don't get any hangs if NT is not connected to a feed. It uses the historical data in the DB as expected.

[NinjaTrader_Ray]

I will check to see if there is anying unusual from the set up you experienced. A few things:

- Please make sure you are on the latest beta 9 as there were changes to internal bar data management
- Any time you open a chart, NT will take your bars back value and check if there is historical data in the database on that day. If it is empty, it will request the entire period up to today of data. If this day falls on a Saturday where the dbase is sure to be empty, then it will always re-request data. Assuming there is data there, it will only request the current days data. This must be done since you need to have the most current data. OpenTick and IB can be extremely slow, especially OpenTick. I have seen it backfill on 15 days for over an hour, I have seen it hang. This is their current level of service. They are working hard on improving it but thats what you get for afree service.

Ray

[zoltran]

I have attached a screenshot with steps to recreate the NT hang on backfill.

The trick to making it hang seems to be to adjust the period of a chart .. After it has backfilled, but before NT has finished backfilling all the symbols in the mkt analyzer

Yes ,, this is Beta 9

[NinjaTrader_Ray]

Zoltran,

Thanks for the detailed step by step walkthrough. I can reproduce and will forward to development.

Also, I will respond to your CCI post later on once I get clarification on a few items.

Ray

[zoltran]

Thank you.

Could you also inform dev't that NT is generally unstable anytime the 'Loading Data' dialogue is displayed. I've had several hangs other than the one I detailed below, just from trying to do 'things' while a backfill was proceeding elsewhere.

By "things" .. I mean modify a chart, change period, load a template elsewhere ...
At one time I thought it was when another dialog had to be opened .. but I can't be more specific.

fwiw,
Machine is XP-MediaCtr, SP2 , AMDx2, 3G Ram, tons of disk.

[NinjaTrader_Dierk]

The lock up problem when toggling chart intervals while NT is loading data will be fixed with next beta/release. Thanks for pointing that out.

[zoltran]

Another test case for you.

Disconnect from esignal
Create 3 charts
Create 3 Market Analysrs using the 3cci template posted previously
mktAnlz 1 - 1 - default list
mktAnlz 2 dow30
mktAnlz 3 nasd 100

Connect to esignal

Wait ... Wait Wait ... it still takes a loooonnnnngggg time, even with esig

And eventually NT will crash.

OR at least it has the last 3 times I've tried it.

[NinjaTrader_Ray]

I could not reproduce. Here is what I did step for step.

- Start NT
- Open 3 charts ES, ER2 and NQ, all 1 min with 15 days back
- Open 3 MA's with your template
- Default instrument list, DOW and NAZ lists
-Connected to eSignal
- Yes, it did take a long time to load and I will look into possibilities to improve this but after it was all loaded, it did not crash

See anything that I may have done differently than you? Also, next time it happens, is there error message with a details button? If yes, please press and copy the text and email it over.

Thanks

Ray

[zoltran]

I will try again later this weekend

[Antraman]

>I will try again later this weekend

Hi Zoltran. How did it go. I have a similar problem and if you resolved it I would like to know how. thx.

Antraman

[NinjaTrader_Ray]

We did improve historical data download for Market Analyzer with Beta 10. The problem prior to Beta 10 was that there were redudant requests for download taking place.

Ray

[zoltran]

Hi Ray

Sorry to say, NT6 Beta 10 still has some problems in this regard.

For example, I currently have a 5 min chart, with a Format Data Series back to early 2004. This was orginally opened using Opentick .. I wanted to see how far back I could get data.

This morning, I connected to Esignal .. and NT tried to backfill the whole 3 years again .. taking about 20 to 30 mins to do so.

I'm not sure of the logic being used.
Most programs I am familiar with only backfill the gaps and do not even refresh the current day. This is usually a very-good-thing, especially with provides like IB that have differing methods for r/T vs historical (.2 sec snaps vs 2 sec snaps)

To my thinking NT should only fill the most recent hole from the last existing quote in the DB to now. This would cover 'most' situations. And we should have a simple way to 'request' a longer backfill from the chart for those situatations the call for it.

[NinjaTrader_Ray]

Yes, eSignal can be slow on minute bar requests.

The current logic backfills based on the first day of the historical data set. If there is data in the database, it will only load the current day, if there is no data, it will load the entire set again.

The way to resolve this particular case is to change the lookback period to fall ona day where data exists in the database. I also suggest for long lookback periods not to have the start data fall on a Saturday where there will never be historical data.

Ray

[zoltran]

I"m not sure it's working as you say Ray.

It always seems to backfill, regardless of state or starting date.
For example, I just tested with charts that started last Monday.

Load NT, no connect - charts display fine.. the data is there
Start esig .. blank-out while it backfills
Stop esig (not NT) .. charts are fine
Restart esig connection ... same blank-out/backfill

Does it on opentic and IB as far as I can tell.
Of course, with IB, it's pretty certain you'll hit their throttle with NT working like this.

[NinjaTrader_Ray]

In your example, is the last data (Format Data Series) set to today? If yes, then this is correct behaviour. NT will always load the current days data into the chart on start of a new connection if the last day is set to today.

Ray

[zoltran]

Yes, it's todays date, But it seems to reload all the chart data. Not just todays
At least thats what it looks like, as it displays each date it's loading in the dialog box.

And why does it load todays if it's already there?

tia

[NinjaTrader_Ray]

Here is what I suggest:

Close all windows in NT.
Shut down.
Open a 1 minute chart with start date last Wednesday, end date today.
--> You should see a chart (assuming you have data in the database)
Then connect to eSignal, NTshould only load today's data.

Why? Since for real-time, you always need to have today's data refreshed so the chart is up to date to the second otherwise, you would have a gap in data from the last point in time you were connected.

Ray

[Greg1]

I believe what I am observing is similar to the other member's observation. Here's what I did to duplicate the problem I am having:

1) Closed Ninja Trader and renamed the .mdb file (to start with a clean db)

2) Started NT (version 6, beta 10) and connected to my data source (ESignal)

3) Opened a market analyzer window with just the instrument columnand a chart window (AAPL stock, 60 min timeframe,45 days back of data). Linked the two windows. On the chart, I added 4 simple moving averages (1000, 500, 200, 50). The chart window is looking at timeframe from 9:30 AM EST to 4:00 PM EST

4) In market analyzer, I loaded the Dow 30 stocks

5) Clicked on AA symbol. Took over 10 seconds for chart window to load

6) Clicked on AIG symbol. Tookover 10seconds for chart window to load

7) Click on AA symbol again (my understanding is that this data is now cached in the mdb file?). Still takes over 10 seconds to load the chart

Note that loading the same chart in ESignal with data since the beginning of the year (about 60 days of data) only takes about 3-4 seconds for every chart even if I go back to a previous symbol. I have avery fast computer and a very fast Comcast Cableinternet connection. I am fairly certain that ESignal does not do any caching of previously requested data.

What is really weird is that if I use daily data timeframe (even if I load 1500 days of data) it takes 1-2 seconds for any symbol on the first request and is almost instantaneous on subsequent requests.

Unfortunately, some of the charts with certain timeframes (like 60 min for example, there are others also)load so slow during the trading day that I don't use them which is contrary to my Ninja Trader goals. Any ideas on how to resolve this would be appreciated.

[NinjaTrader_Dierk]

I guess I know what's going on. Let me thing about potential resolutions for a while. I will come back to you.

Did you get my other PM of yesterday regarding your issue in the other thread? Could you please follow up by PM? Thanks in advance.

[NinjaTrader_Dierk]

We created a custom build which should address this issue. Could you please:
- uninstall NT6 via control panel
- clear your browser cache to make sure you get latest download
- install custom build from here: www.ninjatrader-support.com/ninjatrader/nt6/NinjaTrader.Install.msi

[Greg1]

Yes, I believe this problem is solved !! I'll try to run the desired charts, market analyzer windows during the live trading day on Monday and see if things are improved during live action. If not, I'll provide further details but this is a definite improvement.

THANK YOU

[NinjaTrader_Dierk]

Great!

[zoltran]

It's better, but still has issues.

Switching charts using a link'ed MarketAnalyser and chart now works at a reasonable speed, approx 2 to 3 seconds.

However, loading a Market Analyser still take a long time.
First load of the Dow 30 using the attached template took approx 6 minutes.
2nd Loading took 3 minutes .. you still saw the dialog box counting thru the bars


Nasd 100 first load is at GENZ and has been running for 9 minutes.
I suspect it will take 30 min or more to do the initial load. '-(


The template is pretty simple. Just 3 columns with a 5 , 15 and Hourly CCI value displayed. You can tell it's the Hourly taking the time from the way the Loading Data box counts down.

So .. first load . Some slowness is expected, but not this much. It could be an Esig issue .. but as reported in another thread, esig itself loads this quickly

But 2nd load .. should be really fast


And I still don't understand why NT has to reload the days data even if it's already been stored.


I am using the special build posted below, and esignal for backfill.

[NinjaTrader_Dierk]

Yes, we fixed a hanging issue with B10.

Please:
- uninstall NT6 via control panel
- install latest custom build from here: http://www.ninjatrader-support.com/ninjatrader/nt6.custom/NinjaTrader.Install.msi

[zoltran]

I am running the special build you posted on Fri APr 6
Is the one you just posted different ?

[NinjaTrader_Dierk]

Sorry, some forum handling confusion on my end.

Let's focus the issues you and Greg1 experience in this thread: http://ninjatrader.mywowbb.com/forum12/1802.html

I'll post my findings there.