Has anyone run some backtests/optimizations in the Strategy Analyzer

[NinjaTrader_Ray]

Would love to hear some feedback on anyone that has been testing the performance/memory of backtesting/optimization of strategies.

[PrTester]

Would love to hear some feedback on anyone that has been testing the performance/memory of backtesting/optimization of strategies.

I try, if i use the default strategies, great, if i use one of mines lots of chums, and lot of message, need to strip them down and start adding code to see if they work, but never end an optimization :-(

[daqu40]

My first tests aren't working. It seems the OnBarUpdate is not doing anything.

I tried the SampleMACrossOver and nothing.

I'm digging through the forum to see if there are any configuration issues I can look at to fix this.

Well 10 minutes later I figured it out. I didn't have all the data loaded for my strategy. Seems to be basically working.

Execution speed is initially slower. However my strategy is not producing the same results either so I'll need to get the same results before offering a fair estimate.

I am very happy to see that all 8 CPUs (Quad Core with HyperThreading) are about equally active during the backtest. This is a very welcome change and I expect I'll get much faster tests once I get the code changes ironed out.

Memory impact seems to be fairly minimal. I run test on ES futures going back to 09-07; 9 full sets of data on 1, 30, 60, 1440, and daily SP500 data series. I'm just using Windows Task Manager, but I only got a bout 6% increase in physical memory usage when running these tests.

I'll post new entry when I get this working properly.

[laparker]

A rough test optimizing on 2 params gave me a speed increase by a factor of 2.7 using default optimizer

[laparker]

These are some of the issues I have come across:

On back test tabbed grid, after running test and selecting a tab other than summary, on returning to summary tab, Long Trade and Short Trade detail is blank

Genetic optimizer (GA) crashes with bool params

GA doesn't find anywhere near optimum params. I'm using 20 generations, population of 256. On 6.5 am using Swig's MOGO optimiser and piersh's. Both say that they have started with "Perter's" work and taken it further. Both seem to converge on decent params using 20 gens and 256 pop. Perhaps you could look at their code and see how it differs from yours.

On Trades tab, the Trade# is all 1. The numeric Bars column does an alpha sort

[oclop]

Hello,

I have also done some tests with NT7 Beta2. The issue I have found is that the genetic optimiser only performs the first generation and it stops after it. This may explain why the optimiser fails to find close to optimum results.

I was also glad to see that both CPUs were used when I tried the optimiser. No problems with the memory for the time being. I have not quantified yet the gains in execution speed.

I would also like to make some suggestions for improvement:

It would be nice if when selecting one optimisation iteration to have the option to reuse those parameters directly on the same underlying for a backtest with a range of dates defined by the user. This would reduce significantly the time needed to check the results of the parameters proposed by the optimiser on a data sample that has not been used during the optimisation.
For the genetic optimiser it would be good to have a column with the generation number and the iteration for each optimisation. This would help to identify clearly and keep track record of the optimisations that work.
It would also be good to find a way to better present the parameters that were used for each iteration, especially the ones that have been optimised. I have seen no improvement from NT6.5. The way it is presented now with all parameters in alphabetical order separated by comas is not user friendly at all.Regards, Oscar

[NinjaTrader_Dierk]

...
On back test tabbed grid, after running test and selecting a tab other than summary, on returning to summary tab, Long Trade and Short Trade detail is blank
...
On Trades tab, the Trade# is all 1
...
The numeric Bars column does an alpha sort


Just reproduced these ones. Will look into. Thanks

[NinjaTrader_Ray]

Thanks for all the feedback, suggestions and issue reports.

[NinjaTrader_Dierk]

Guys, please hold off trying the GeneticOptimizer. There are a couple of issues which I just addressed. Thanks for reporting you findings.

[paschall]

I use the Backtest feature each programming session. It is invaluable to my work. Although, right now, I have to optimize parameters manually, it does give me a very nice objective performance evaluation for my strategies. One of the reasons I chose NT for development work.

cheers,
D

[zweistein]

I did try some backtesting, but usually I do some automation because I feed the backtest results into a Neural Net.


So my code is something like:

ninjaserver.Instance.BackTestDialog("SampleMACrossOver");
ninjaserver.Instance.SetBackTestParameter("Instrument","FDAX 12-09");
string dt = DateTime.Now.AddDays(-30).Date.ToString();
ninjaserver.Instance.SetBackTestParameter("BackTestFrom", dt);
ninjaserver.Instance.SetBackTestParameter("BackTestTo", dt);
ninjaserver.Instance.RunBackTest(60); // 60 s timeout
object results = ninjaserver.Instance.BackTestResult("Performance.AllTrades.TradesPerformance.Currency.C umProfit");


and due to automation I one cycle is at least one second.

Therefore I cannot talk about speed...

Best regards
Andreas

[dnoff]

I have tried running the optimizer several times tonight and it constantly fails to run beyond the a few seconds and throws the following errors:

11/1/2009 8:13:48 PM Default Error on running optimizer: Value cannot be null.
Parameter name: value
11/1/2009 8:14:00 PM Strategy Error in setting strategy parameters: Input string was not in a correct format.

The settings I am trying to optimize can be run successfully in 6.5 (just takes a while), and I enter the values (checked several times) by expanding the parameters and typing the values into the boxes.

Thoughts

[NinjaTrader_Dierk]

Please make sure you are on latest NT7B3. If issue persists, then please post your strategy and any info required to reproduce the problem. Thanks

[paschall]

I just hand-checked some backtest results against a strategy. I only ran 2 days worth so I could scroll back and hand-check things. Over the past 2 days, my strategy generated 3 long trades. Backtest results? It says there was only 1 long trade (albeit a successful one). Same erroneous result with the short trade data. Hand-checking shows 8 short trades. Back-test output Summary window indicates 5 short trades. I double-checked the date range, time frame of the backtest compared to the chart window, etc. Used default values in both instances.

Any ideas why the discrepency?

cheers,
D

[NinjaTrader_Josh]

Please provide the strategy and relevant backtest settings used including instrument, dates, time frame, etc. Thank you.

[trend]

On a separate note, I can't find an appropriate combination of
"From" and "To" entries under "TimeFrame" to generate results for
a single day on intraday data series. For example, if you set:
From 11/3 to 11/3 you get trades for both 11/3 and 11/4.
You might want to expand the From/To entries to include minutes
(11/3/09 12:00 am - 11/3/09 11:59 pm ?)

[NinjaTrader_Josh]

The dates are controlled by the session template you choose to use in your backtest. If you want 24 hour session please select 24/7 or 24/5 for the session templates.

[trend]

Well, I tried various templates beyond the one that was
set for the instrument ("Forex"). When I set From and To
to 11/3/09 I still get All trades that exited on the 3rd and 4th.
I guess this is a suggestion to incorporate minutes in the
From/To entries.

Also, is there a way to track real PnL intraday?
I'm using Perfromance.AllTrades.TradesPerfromance.Currency.C umProfit
but this assigns PnL of trades at their exit bar. For real time PnL
we would need to track CumulativePosition and AverageEntryPrice
for Shorts/Longs of all signals in the script and marktomarket at every bar.

Thanks,

[NinjaTrader_Josh]

Not sure what you mean. If you are using intraday data you will get timestamps of minutes and even seconds if your data has seconds granularity.

If you want to track real trades use Performance.Realtrades.

[trend]

Wasn't clear. Even for Historical trades,
Performance.AllTrades.... assigns the whole PnL at the
DateTime where the Exit of the Trade is stamped.
So you can't track the PnL resolution intraday .

Another way would be to track PnL on a bar to bar basis
by accessing the Cumulative Long/Short position and their
corresponding average prices as they are adjusted with every
execution, and marktomarket at every bar intraday.

[NinjaTrader_Josh]

Correct. Performance is a measure of your overall trading. If you want individual trades you need to be looking at the Trade class.

[trend]

From the examples I see the Trades collection refers to Completed
trades. Is there any way to track PnL (i.e., M2M at each bar)
while the Trade is open, even on Historical data?

That is, you might have several trades generated during the day
(let's say beggining session to end session) but
they all exit at close of market. Can you track the M2M at intraday granularity (so you know what's your PnL at 12:00 pm-even thogh there
where no exits till that time)? Also, to apply these on backtests (historical data).

[NinjaTrader_Josh]

You can try: http://www.ninjatrader-support.com/HelpGuideV6/GetProfitLoss.html

If you need even more granularity you need to track it yourself based on your IOrder's filled price.

[dnoff]

I have tried running the optimizer several times tonight and it constantly fails to run beyond the a few seconds and throws the following errors:

11/1/2009 8:13:48 PM Default Error on running optimizer: Value cannot be null.
Parameter name: value
11/1/2009 8:14:00 PM Strategy Error in setting strategy parameters: Input string was not in a correct format.

The settings I am trying to optimize can be run successfully in 6.5 (just takes a while), and I enter the values (checked several times) by expanding the parameters and typing the values into the boxes.

Thoughts

So I have been playing with the optimizer this afternoon and the issue I noted above is still present. It is however only present with the "default" optimizer, not the newer genetic optimizer.

So I have attached the screen shots of the strategy I have been running to optimize. It fails with the error1 first, and then error2 (screen shots). I checked the trace file and their is not errors at all in there, just entries getting data from Zenfire.

If I switch the optimizer from "default" to "genetic", it runs to completion.

Note: I am on NT 7.0.0.4, Win 7-x64

I can send the strategy if required - let me know.

[NinjaTrader_Dierk]

What data type is "EntryType" and "ExitType"? It needed to be string/int/double/bool. Any other data type if not supported.

[dnoff]

What data type is "EntryType" and "ExitType"? It needed to be string/int/double/bool. Any other data type if not supported.

It is a custom enumeration of strings/values. I suspect this is the problem then. Is this new/changed from 6.5 as this strategy used to work just fine in this. Also, why would the genetic optimizer still work?

Thanks for the reply

[NinjaTrader_Dierk]

We only ever supported string/bool/int/double for strategy parameters (see strategy analyzer). I may or may not have worked in 6.5.

[trend]

"You can try: http://www.ninjatrader-support.com/H...rofitLoss.html"

The above is for Unrealized PnL
Is there sthg tracking Realized PnL?

Thanks,

[NinjaTrader_Josh]

That's what the Performance and TradeCollection classes track. I am not sure what you are trying to get since you started from there to begin with.

[trend]

Well basically, I'm trying to do marktomarket at every bar intraday
for a case with multiple signals.

Position.GetProfitLoss(Close[0], PerformanceUnit.Currency)

tracks unrealized Pnl. The issue is that at a given intraday bar you might have some trade closing while others are still open.

As I understand, when a trade closes
Performance.AllTrades.TradesPerformance.Currency.C umProfit

i.e, RealizedPnL is updated.

[NinjaTrader_Josh]

Correct. As a trade finishes the performance will be aggregated into that CumProfit as realized PnL and subtracted from GetProfitLoss's unrealized PnL.

[trend]

Got it,
Performance.AllTrades.TradesPerformance.Currency.C umProfit +
Position.GetProfitLoss(Close[0], PerformanceUnit.Currency)

for M2M on each bar,

Thanks Josh,

[piersh]

I have just released an update to my GO that works in NT7. It supports optimizing bools & enums.

You can find it over on Big Mike's forum (http://www.bigmiketrading.com/free_downloads/ninjatrader/strategies/303-download.html).

Some things I'd like to see in a future release of NT7:
- allow an optimizer to find out if it's in 'walk-forward' mode.
- copy bool & enum parameters from the OptimizationMethod.Parameters collection into the strategy. i've used hacks in my GO, but it wouldn't take much more code in SetPropertyValue...
- if 'Optimize()' returns before calling 'WaitForIterationsCompleted (true);', or throws an Exception, assume that it has been called. don't leave the app in a broken state.

[tradejockey]

64bit Strat Analyzer works great performance wise... 32bit not so hot. I wouldnt use 32bit for anything (especially SA) but zenfire connection (and running the script daily). For some inexplicable reason 32bit version of SA takes longer. What I dont understand is that my pc has the horsepower but it is not being utilized.. I checked the CPU utilization/ Memory / Disk IO. They all indicate very low levels of activity.. perhaps one single thread is chugging away in a sequential manner in 32bit version.

I believe zenfire is a marketing brand for some other company's api. Dont remember what that company's name is (starts with an "R") Any idea as to when a 64bit version of it will be implemented (by whom I dont know).

Business functionality wise:

Ofcourse, my chief complaint with SA is that BackTester is missing some trades that live sim picks up. So backtesting is not entirely accurate.

Secondary complaint is Minute vs Second timeframe. Say you use 180 minute interval for example.. results will be nice. But multiply that my 60 and you get 10800 seconds.. results will be horrible even though it is the same exact interval.

MonteCarlo seems to work more like an optimizer than a true monte carlo engine. In 64bit, I was able to do 500,000 sims while NT7's memory foot print grew to 7 gb in the ram. Doesnt let go of the memory after it completes. In 32bit version, If NT7's memory size reaches in the 2.5 gig range, Monte Carlo freezes.

Since MC is dependent on trades list, what should be taken as more accurate? MC run off of
1. list comprising unabridged (non back adjusted) data for longer time period such as 2 years or
2. list for a quarter's worth of data
3. list for two or more quarters worth of data (start date is say second (globex) /third (eurex) thursday of previous quarter(s) )

Some key metrics such as Sterling ratio, Sortino ratio, Calmar ratio etc are not provided. I do not know if this is an oversight or a purposeful omission.
There are a few other ratios that I cant recall at the moment. Also, Drawdown on initial capital is missing. Max drawdown is useful but it doesnt capture the accurate risk to your initial capital outlay..

I suspect that Daylight Savings time is not being considered between Europe and US. For a fortnight in march and perhaps in fall, there is a one hour gap and start datetime stamp is somehow in sync when it shouldnt be.. ie 2am in europe will be 3am est for a fortnight or so until europe catches up. But datetime stamp on the tick will still be 2 am on historical tick data. In reality there wont be any data until 3 am.

[NinjaTrader_Josh]

piersh,

Thank you for the suggestions.

tradejockey,

Backtesting is and always has been fundamentally different than real-time. This is why people recommend taking backtest results with a grain of salt and to make sure that proper forward testing is also done.

Minutes vs seconds: Different data sources can result in different results. Minute data comes from minute bars. Seconds data comes from tick data.

Thank you for your MC suggestions.

Thank you for your suggestion on additional analytics.

Timestamp of data comes from your data provider on historical bars. In real-time, depending on the data provider, it may be timestamped by them or by your local machine on receiving the ticks.