The database file is larger than the configured maximum database size

[gena72]

Hello,

I know this issue came up a few times, but all the responses did not help me to find a solution. I'm getting a lot of "The database file is larger than the configured maximum database size. This setting takes effect on the first concurrent database connection only. [ Required Max Database Size (in MB; 0 if unknown) = 257 ]" errors. Typically it occurs when strategy order is executed, but it can also occur randomly at any time during a day.
When it happens, NT freezes, sometimes up to a level when datafeed connection is lost (I use Kinetick). When it un-freezes, prices are no longer updated, and i have to restart NT and my strategy.
When I look at NinjaTrader.sdf, I see that sometimes it jumps to more than 250Mb in size, but most of the time its close to 8Mb.
My strategy does not place/execute too many orders, but sometimes it has to cancel/re-place an order many times. However, I do not think its the main issue here.
The problem may be related to the fact that my strategy uses very large data structure, and NT probably tries to serialize the strategy along with all its data and place it into the database. Just a guess.

Please assist in resolving this issue.

[NinjaTrader_CalH]

Hello gena72,

Thank you for your post.

This will have to be isolated down in the strategy script to see what is causing the exact issue.

When is the enlargement of the database file witnessed?

You said your strategy is using very large data structure, could you expand on that?

You would want to systematically go through, comment out some sections of code and run some Print() statement in place.

Debugging your Script

[gena72]

Well, from what I see, the database increases in size sharply when an order is executed. The error message pops up, and than after I press Ok, the database size goes back to 8Mb.
The strategy uses various data-structures, including arrays and Lists. I've not done any memory profiling, but cumulative size of these arrays may well exceed 250Mb. I still do not understand how this may be related to database size, unless NT is somehow trying to serialize the entire strategy along with all class members and place it into DB. Is this the case?
Not sure that putting "Print" statements all around the code (which is large and complex) will help to isolate the problem, as strategy code is not using the DB directly, and there's no API that I know of, which may help to access and measure DB code from my code as well.

[NinjaTrader_CalH]

Gena72,

I would recommend commenting out sections at a time and running the strategy. This will allow you to see at what point the DB is getting large and thus allow you to determine what part you just commented out could be causing the issue.

[gena72]

As I mentioned, the strategy is large and complex, with parts of the code dependent on each other. Commenting out various sections would take forever, especially since the problem manifests itself only when orders are executed in real-time.
Is there a more intelligent approach? Is my strategy being serialized into DB?
I believe NT development team could easily list all possible reasons for DB filling up.

[NinjaTrader_CalH]

Gena72,

There is no direct answer to this.

There a couple scenarios than can cause this but the main ones are going to be Database manipulation or excessive Order submission, this includes submission and cancellation.

Is this strategy run on Live data or using Market Replay?

Do you see a lot of logging or orders from the orders and log tab?

[gena72]

The problem occurs on Live Data, but I believe it happens on Market Replay as well. There're orders being logged, including cancelled ones, but not too many. Around 20-30. The problem occurs, however, only if the order is being executed. Frequent cancellation does not seem to matter that much.
Not sure what do you mean by database manipulation. I'm not accessing the DB in any way from my code.

[NinjaTrader_CalH]

Gena72,

Would it be possible for me to send me the strategy to test this on my end as well?

[gena72]

The strategy requires a very specific environment to run (uses external files/etc..), and it would not be easy to recreate on your end. I can try to create a "reduced" version, which still causes the same problem, but it may take some time.
Is there any directions/recommendations to look at on my side? I still believe that the problem is somehow related to the datastructure size, as it started to occur only when my strategy became "large". Is there a way to examine this large database? Maybe its would be helpful if I send this large DB file to NT support for examination.

[NinjaTrader_CalH]

Gena72,
You can send me the DB file to me by going to Help ->Mail to Support and select the Database check box.

Put Attn Cal in the subject

[gena72]

For some reason NT fails to send the e-mail along with the DB file. Maybe file is too large? The window just freezes after I press Send. Still I managed to copy the SDF file (its over 250Mb in size).
Is there any other way to upload it to NT support servers? I also can use any file exchange if its appropriate.

[NinjaTrader_CalH]

Gena72,

WeTransfer should be able to do this for you.

[gena72]

Ok, thank you for the tip. Just sent a message using WeTrades with two sample DB files. Hope you'll be able to provide a solution for this problem, since currently my strategy is not running due to this.

[NinjaTrader_CalH]

Gena72,

Thank you for your DB.

I looked into the DB and didn't see anything outstanding. However, without the strategy script and the ability to test and run it on my end I cannot give you a definitive answer as to what is occurring.

More than likely something in the strategy is causing the system to flood the DB.

Point is the DB falters if your strategy instances are not needed anymore

If you can get me a skeleton example of where the issue is being replicated this is something that I can look into and either find a solution or report to Development about the handling.

[gena72]

If you don't see anything outstanding, than what exactly is taking over 250Mb? Orders, strategy instances, something else?
Can you elaborate a bit on "Point is the DB falters if your strategy instances are not needed anymore"? When exactly the instances of my strategy are not needed anymore?
Also, I must point out that my strategy uses only documented NT API, in a documented way, and in theory a problem like this should not be occuring at all, unless there's an issue within NT. At this point its a total show stopper, as my strategy can not run at all.