Optimization time

[bobby1001]

I am optimizing 7 variables. If I optimize over 4, I have a drag time of about 6 to 22 hours. All variables have between 6 to 10 possibilities. If I only do 4, it takes about 3 minutes. If I do more, the please wait box time to go fluctuates wildly, and the process does not seem to work.

I'm running an intel quad processer with 4 gbs of ram, so I don't think it's a system issue.

Please advise as this wait period does not make sense to me, and it is to long to wait.

Thank you.

Bobby

[NinjaTrader_Ray]

Bobby,

I would need clarification.

You state:

Optimizing > 4 input variables takes 6 to 22 hours

This could very well be the case depending on the range of time of your optimization, the underlying data series and the complexity of your strategy and embeded indicators.

What do you mean "fluctuates wildly" and process does not work.

Thanks.

[bobby1001]

What I mean by fluctuates is that the time that has passed will work fine, but the next time, time to completion, will generally start at a much larger number, 24 hours, give or take seems to be the number, and will start counting down during the test. The number will jump back up again, sometimes going from 4 hours to go back up to 24 hours to go.

If I do make it through to results then sometimes it won't show the trades on the chart, or list the trades, etc., only show the summary.

If there is a way to speed up the process, please let me know.

Also, when testing multiple variables, wouldn't each variable affect the next variable, or could you test it one variable at a time?

Thanks.

[NinjaTrader_Dierk]

I suggest amending your optimization strategy to ensure the tests completes within "reasonable" expectations. Reason: apparently you're experiencing issues related to potential memory excess or excess of other system ressources. As soon as this happens, .NET no longer might operate as expected and NT (based on .NET) might not longer operate as expected.

Also: Please double check your strategy does not consume memory or other system ressources. This could happen as you use "extended" .NET features and do not release resources in the Dispose method. (see docs)

[bobby1001]

What would be the best way to check what system resources are being used and how resources are being released?

[NinjaTrader_Dierk]

One way to test system resources is the task manager.

I suggest trying this: start with a simple as possible strategy and add parameters step by step until the optimization of your strategy finally goes "out of whack". This would provide you an idea what causes the issues.

You also might consider testing on a smaller time frame to isolate the issue.

Forgot: Please make sure you are on NT 6.0.1000.4.

[bobby1001]

I'm running the system on 150 tick bar charts. This seems to be the problem. However the system is based upon tick charts as opposed to time charts. I'm only optimizing 3 days worth. What would you recommend to clean this up?

Thanks.

[NinjaTrader_Dierk]

That depends on your actual strategy. Try with a simple as possible version and take it from there.

[bobby1001]

I'm running my system on 150 tick bars, but am looking at 2 additional time frames within the system. I've done this using the barsarray method.

This seems to be the problem with the tick charts.

Also, in the slippage box, I am using 1. Is that 1 tick on a round turn, or 1 tick per side? I just need clarification on that, and can't find it in the help documentation.

Thanks.

[NinjaTrader_Dierk]

- slippage is per side
- are you on 6.0.1000.4?

[bobby1001]

Yes, that's the version.

[NinjaTrader_Dierk]

Please PM me to "dierk AT ninjatrader DOT com":
- your zipped DB by Help->Mail to support (enter my mail address)
- a simple-as-possible version of your strategy
- a screen shot of your strategy settings

Thanks in advance

[bobby1001]

Also, when optimizing the default is max prof. factor, and there is max net profit. I'm assuming net prof. ignores drawdown, while factor takes it into account. Is that correct?

[bobby1001]

You said to PM you. Do you mean e-mail you? I'm not familiar with PM.

[NinjaTrader_Dierk]

PM = Private Mail = eMail. Start NT->Help->Mail to Support and proceed as per below

Profit Factor = gross profit / gross losses
Net profit = gross profit - gross loss

[bobby1001]

I sent it. Will you reply here, or to my e-mail?

[NinjaTrader_Dierk]

To your email

[bobby1001]

All right, thanks. I also sent a blank e-mail should the first not get through for any reason. I have a new setup and things are still a little touchy.

Thanks.

[Jenny]

Hi, NinjaTrader_Dierk,

I really want to know the result, what is the solution for this question? 4 variable take 3 minutes but 5 variable 6~22 hours. Only 1 variable different, why running times are so much different?

And an intel quad processer computer sounds very powerful. 4 variablesX10 possibilities=10X10X10X10 only need 3 minutes.

Thanks && Happy new year
Jenny



I am optimizing 7 variables. If I optimize over 4, I have a drag time of about 6 to 22 hours. All variables have between 6 to 10 possibilities. If I only do 4, it takes about 3 minutes. If I do more, the please wait box time to go fluctuates wildly, and the process does not seem to work.

I'm running an intel quad processer with 4 gbs of ram, so I don't think it's a system issue.

Please advise as this wait period does not make sense to me, and it is to long to wait.

Thank you.

Bobby

[KBJ]

You might check to see if the number of Page Faults greatly increases when you increase the number of variables being optimized.

This may be checked by using the Windows Task Manager, clicking on the "Processes" tab, and then "View", "Select Columns", and selecting "Page Faults".

Then check for number of page faults before and after running each series of optimizations.

If the number of page faults is not proportional (within reason) to the total number of combinatations (calculated from the product of the numbers of possibilities for each variable being optimized), this would indicate that the optimizer is running out of working set memory and creating I/O's to the page file as part of its virtual memory utilization.

Each page fault causes an extra I/O, so if this is occurring, this would be greatly slowing down the optimize run.

Check out this definition of virtual memory, which describes what page faults are all about: http://en.wikipedia.org/wiki/Virtual_memory

[Jenny]

Dear KBJ (http://www.ninjatrader-support.com/vb/member.php?u=1297),

Thanks for your information. I will monitor the page faults.

I am running a program, about 3.5 months data by minute, only the back test take about 3 minutes. That means 120 combinations need about 6 hours. Do you think that is normal?

Thanks a lot
Jenny

[Pete S]

Jenny:

That doesn't sound normal to me. My primary strategy, which I run on 5 minute bars, takes .05 seconds to run one iteration as timed in the code. You are saying it takes 3 minutes for one single iteration? If so, you must be using a bad algorithm in your code somewhere. I'd suggest doing a little manual profiling of your strategy code.

[NinjaTrader_Josh]

Sometimes I find that running backtest/optimizer will draw upon your connection to try to get data. I find it runs much faster if you first download all your data and then disconnect from your datafeed and drive your backtest from your database only.

[Jenny]

Thanks for Pete S and Josh replying my question.

I had downloaded data, but it is still the same.

I had some other code, it works fast. Maybe I have too many Variables? Here is the session:

#region Variables
private int siganlSMA =1041;
private int pe =8;
private int p1 = 44; // Default setting for P1
private int p2 = 205; // Default setting for P2
private int b1 = 81; // Default setting for B1
private int b2 = 3; // Default setting for B2
double Low2=0;
double High2=0;
double Low1=0;
double High1=0;
int LowBar2=0;
int HighBar2=0;
int LowBar3=0;
int HighBar3=0;
#endregion

In my code, I also use :
MRO(delegate {return (CrossAbove(SMA(5), SMA(10), 1));}, 1, 260);

Would you please give me some advise?

Happy new year

[NinjaTrader_Josh]

Sorry Jenny. I am out of ideas as to why your optimizing is taking so much time.

Here is a benchmark from my system you can check against:
SampleMACrossOver backtested on 1min from 1/1/2007 to 1/4/2008 took 6 seconds.
My computer is 1.84ghz with 1gb ram on WinXP SP2.

[Pete S]

Extra variables will not, in general, make your code run slower. What really makes your code run slower is two main things:

1. Excessive memory allocations
2. Bad algorithms -- algorithms where the performance decreases in a worse than linear fashion as the input size grows

You definitely don't want to be doing this in your code: MRO(delegate {return (CrossAbove(SMA(5), SMA(10), 1));}, 1, 260);

What you should do instead is, in the variables region, do this

private SMA mFastSMA;
private SMA mSlowSMA;

and in your Initialize method:

mFastSMA = SMA(5);
mSlowSMA = SMA(10);

Then you would reference them like

CrossAbove(mFastSMA, mSlowSMA, 1)

in your code. This will not matter for running live, but will make a noticable difference when running thousands of iterations on an optimization.

Note, however, this alone will definitely not account for what you are seeing. Like I said in my last post, a little profiling is in order so you can find out exactly what is happening. The first file I have attached is an instrumented version of the standard optimizer. To install it, save the file to your Ninja Trader 6.5\bin\Custom\Type directory and recompile your strategy. Then, when you run your optimization, pick 'Default with Timing' as the Optimizer. Post the output here and I will help interpret it; it will look like this:

FCSX: SampleMACrossOver: starting optimization at 1/6/2008 10:34:34 AM: total iterations: 820
FCSX: Iterations: 100/820 in 0.1 minutes (0.03 sec per iteration)
FCSX: Iterations: 200/820 in 0.1 minutes (0.03 sec per iteration)
FCSX: Iterations: 300/820 in 0.1 minutes (0.03 sec per iteration)
FCSX: Iterations: 400/820 in 0.2 minutes (0.02 sec per iteration)
FCSX: Iterations: 500/820 in 0.2 minutes (0.02 sec per iteration)
FCSX: Iterations: 600/820 in 0.2 minutes (0.02 sec per iteration)
FCSX: Iterations: 700/820 in 0.3 minutes (0.02 sec per iteration)
FCSX: Iterations: 800/820 in 0.3 minutes (0.02 sec per iteration)
FCSX: finished optimization at 1/6/2008 10:34:53 AM, iterations: 820, total time: 0.3 minutes

The second file I have attached is an instrumented version of the SampleMACrossover strategy. It's an example of how you might instrument your own code to find the areas that are taking too much time. You'll also note I put in a section to demonstrate what I explained above; you can see it is roughly 100% faster in a backtest.

I've spent a bunch of time doing this for myself and my primary strategy runs 10 times faster than it did before optimizations.

[Jenny]

Pete S,

I really applicate you help. I already modify my code according to your advise, Here is the result:

$NZDJPY: LowHigh12e: starting optimization at 1/7/2008 12:08:02 AM: total iterations: 4
$NZDJPY: finished optimization at 1/7/2008 12:10:30 AM, iterations: 4, total time: 2.5 minutes

Thanks a lot
Jenny

[Pete S]

So each iteration now is taking ~37 seconds, that is still a shockingly long time compared to anything I have seen. I'll note that I don't know anything about trading Forex & if that has anything to do with it.

I'd recommend one of these things:
1. Follow my example from the MACrossWithTiming strategy. Take a look at the code, put those timing blocks around each individual section of your code. Find out where it is spending its time. Then you can focus on that section of code or post that block here for help.

2. If you don't mind sharing your strategy, you can email it to me and I'll take a look at it.

I suspect there is one section of your code doing something unintended.

[Jenny]

Pete S, I have result after embed your timing code, but it is too big to post here. Can I sent to your private email address?

Thanks for time

[Jenny]

Pete, I already send the result to your email.
Thank you very much

[Jenny]

Hi, Pete,

I modify my code, here is the new result:

100000: avg cross: 90.938 avg cross2: 404.219 avg orders: 6.719 [times in microseconds]
200000: avg cross: 91.641 avg cross2: 368.359 avg orders: 5.703 [times in microseconds]

I only have such so good little data.

Thanks

[Pete S]

Jenny:

Whatever is inside the section labeled cross2 is taking, if my math is correct, ~80 seconds of time:

~400 microseconds * ~200,000 runs = ~80,000,000 microseconds or 80 seconds. If you are still running 4 iterations, that's 20 seconds per iteration. So that is probably the first problem area to look at. Can you give me as much detail as you can about what you are doing in there?

[Jenny]

Pete,

I just want to find 2 most recent Low and High points, then compare them to see good for Buy or Sale. I use MRO() and LRO(), MIN() and MAX().

Maybe you have some way much easier?

Thanks

[Pete S]

There is only so much feedback I am going to be able to give without seeing code...in general, as I said earlier, your code is probably using a bad algorithm and looking at the input data too many times. Just curious; how many bars back are you looking?

Even if you are just using min and max functions, you can do optimizations. MIN and MAX each loop through the input data. Here's an example. My breakout strategy uses the DonchianChannel indicator. The main section of the standard out-of-the-box DonchianChannel looks like this:

Value.Set((MAX(High, Period)[0] + MIN(Low, Period)[0]) / 2);
Upper.Set(MAX(High, Period)[0]);
Lower.Set(MIN(Low, Period)[0]);

Each call to MIN or MAX is looping through the data Period number of times. I created a new indicator which uses this as the main logic:

int n;
double high = -1, low = 1000000;
for (n = 0; n < actualPd; n++)
{
if (High[n] > high)
high = High[n];
if (Low[n] < low)
low = Low[n];
}

MyMean.Set((high + low) / 2);
MyUpper.Set(high);
MyLower.Set(low);

Notice there's only one loop. This was roughly 300% faster, as well as degrading in linear fashon, where the stock code degrades in a worse than linear fashon.

Ideally, you wouldn't need to worry about this stuff, but optimizing for performance has really been a key part of my productivity improvement.

[Jenny]

Pete,
You are right!!
I use a loop insteads of MIN, MAX, MRO() and LRO(), it works much fast.
Thanks a lots and good luck in the market!
Jenny

[Learning1]

Thanks for this Pete S ...
I can see where this could speed up my optimizations significantly ... one question - with this indicator, I'm a bit confused on the use of actualPd ... Is this a vairiable you have named above that is the desired lookback period of the indicator (replacing the period in MAX / MIN) that is an input for the normal Donchian Channel?

Thanks again ...

Learning1

[Pete S]

I use an intraday-only version, so for me it contains either the Period or bars since the beginning of the day. You can safely ignore that and just use the standard period parameter, just make sure you check CurrentBar > Period.

[Sunrise]

Does this mean that the Max and Min calls could be improved by Ninja so they are not so 'intensive'? Certainly sounds like it!

I am off to change my coding!

Sunrise.

[Pete S]

It depends. For example, if you need to call a bunch of different functions that are going to iterate over 20 bars of data, there is no way for NT to restructure those calls so that can all be done at once.

I'll just emphasize again in my opinion it's not a problem per se, it's just that when you are running long backtests anything you can do to pick up efficiency really helps.