Optimization function - System Quality Number

[Ragingbull]

Dear NT,

I have been using for some tests your strategy analyzer module. Looking to the optimization part of it, I realized rapidly that it would be of very little interest. Your program proposes as optimization function the max net profit, the max av. profit, the max W/L, the min drawdown, the max profit factor……all very nice to have but totally useless from a real trading point of view.

The only way to seriously qualify and optimize any system is through its System Quality Number (SQN). I advice you to refer to Van Tharp for reference on the subject.

Assuming a set of N trades (N>30 for being statistically significant), SQN is defined as follow:

SQN= Squareroot(N) * Average (of the N Profit&Loss) / Std dev (of the N Profit&Loss).

The large the N, the more trading opportunities you have.
The large the average P&L, the better you are obviously.
The smaller the Std dev (P&L), the more regular are your results and the smaller are the drawdowns.

Note here that if you optimize for the largest SQN, you maximize in fact the product N*average P&L and you minimize the Std dev (P&L) and the drawdowns at the same time.

This is exactly what all good traders should be looking for their system.

Since it is so simple to program, I believe that it could be rapidly integrated in NT.

Thanks in advance.

Best regards

[NinjaTrader_Dierk]

Thanks for your suggestion. We'll add it to the list.

[Pete S]

Ragingbull:

Although it's not documented, you can in fact write your own optimization types. I've done this a couple of times for the reasons you are describing -- the base ones aren't quite adequate. If you give me a little more information, maybe an example, I'll see if I can create one. It will be dependent on the required data being available in the stock data model, which it sounds like it is.

[Pete S]

Actually, assuming that the avg. P&L are in points, and not percentage, try this. If not, please let me know and I'll make the changes.

To use:

1. Copy this to My Documents/Ninja Trader 6.5/bin/Custom/Type
2. Recompile any of your indicators or strategies -- this will cause SQN to be compiled as well
3. "my system quality number" will show up in your list
4. The SQN value shows up in the Performance column on the grid

It's a bit more computation-intensive than the standard optimizations, but not much.

I only tested this on 6.5 beta 4, I have no idea if it works on 6.0. Also, I would expect that since I am using an interface that's not officially documented by NT, it may break from release to release.

[KBJ]

Very clever.

However, it doesn't work for 6.5. I've attached the error. I'm not suggesting you fix it, this is just FYI.

[Ragingbull]

Hi Pete,

Thanks for the file but it does not seem to work for me either. None of my strategies are working with this new file.

I am running on the 6.0 regular version. Is this the problem?

Concerning the computation of the average P&L, using the values in points is OK. Using values in currency or in % can also be done but should be coherent for both the average and the std dev computations.

Although I would suggest that you use "net P&L" values for the calculation of the average and the std dev since some scalping method can lead to very large N and therefore also to very large commission costs.

I hope you can find out what is the problem, it would be very nice if we could have this tool working.

Best regards.

[Pete S]

KBJ -- that is bizarre, first, because I have been running with this for the past day on 6.5 beta 4, and second because that field comes directly from NT's standard code. Are you sure you didn't mean 6.0? Take a look inside bin/Custom/Type/@MaxNetProfit.cs and tell me if the line of code they use for that standard calculation matches what is in my file.

Ragingbull -- why not install the 6.5 beta? I'd rather not reinstall 6.0 at this point. Are you running live? Also -- I believe that the calculation I am using does take into consideration commission costs as long as you have that value turned on when you run the strategy. Again, I am using the same calculation for net profit that NT uses in their code and it respects commission costs.

[Pete S]

Oh, and KBJ, I forgot to thank you for the file handling code you posted in one of my earlier threads, that came in handy when I debugged my custom optimizer class.

[Pete S]

As I suspected, KBJ, I think you meant 6.0. Because I can't stand leaving the impression that I put bad code out there, I attempted to run this on 6.0. It was a one line change, they've changed the net profit calculation for 6.5. Everything else appears to work ok, but I did not do intensive testing on 6.0.

[KBJ]

Big ooooops .... you're right; I meant 6.0.1000.6, and I still think its clever.

[Ragingbull]

Hi pete,

Thanks for the effort for the new file. I have installed it and its perfectly running on my 6.0. I will make some tests.

You suggest that I should install the 6.5 beta 4 version. Is it up and running for day trading? Where can I find it (I have a lifetime license for 6.0)?

Best regards,

[Pete S]

Thanks for your original post -- I think this will be useful for me as well.

Forget my comment about the beta for now, that was just me trying to avoid recoding the file for 6.0, which I ended up doing anyway (and was trivial).

BTW -- if you notice any bugs with the algorithm in your testing please let me know so I can fix them.

[Ragingbull]

Hi Pete,

I made some test yesterday and have noticed that the SQN number that we read in the column "Performance" do not correspond to the SQN calculated manually from all trades.

Basically, when an optimisation sequence was completed, I have

[Ragingbull]

Hi Pete,

I made some tests yesterday and noticed that the SQN numbers that we read in the column "Performance" do not correspond to the SQN calculated "manually" from the trades.

Basically, when an optimisation sequence was completed, I have extracted the trades and calculated in Excel the corresponding SQN. It never compaired with the ones produce by the program. The program seems to produce SQN systematically too large.

Not being aware of how to program in C#, I can't help you much more than that.

I hope you can find where is the catch...

Note that the order of magnitude for SQN, for a very average system, is between 1 and 2.

Best regards,

[Pete S]

What are you using in excel? Your own formula? If so, can you post it here?

[Ragingbull]

Pete,

I am using:

SQN= sqrt(N)*average("Profit")/stddev("Profit")

Please note that since I have a french version of excel, I have translated the name of the function in what I believe is the excel english name.

Where N is the number of trades and ("Profit") means the N values from the column "Profit" extracted from NT.

[Pete S]

OK, if it is that simple, i will do a little testing with Excel here and see what I come up with. I wrote a version which uses percentage instead of points and the results weren't very different.

Are you sure you have your commission set up correctly in NT and have use commission = true when you run your strategy?

[Pete S]

Ragingbull -- one thing that jumps out at me right away is that I am using the Excel formula for "STDEVP", not "STDEV", in my code. The difference is STDEV is over (n-1), while my STDEVP is over (n). Can you do a quick test, and substitute STDEVP in your formula, and see if that is what is causing the difference? If you are using a reasonably small n it is possible.

Next suspect would be the commissions...if this doesn't work out I will investigate further.

[Ragingbull]

Pete,

I am about to leave and won't be back before saturday.

Effectively, this could make a small difference but don't forget that we are looking here for a set of trades larger than 30.

To give you an idea, a manual calculation was giving me a SQN of 3 and the program 13 (really too good to be true...) on hundreds of trades! So, in this particular case, the error is probably coming from another place and of another order of magnitude than "n" instead of "n-1". I will check at my return which one should be used anyway.

Best regards,

[Pete S]

Hmmm...I certainly haven't seen anything like that.

I am willing to try and figure out what is going on, but I think I will need an example of your data which is producing these results. Is it possible for you to strip out any personal and/or proprietary information from your spreadsheet and email it to me? I'll PM you my email address.

[Ragingbull]

Hi Pete,

No problem. How do we proceed?

[Pete S]

PM sent w/my email.

[Pete S]

Hm, so you are running with futures, which I don't trade (I developed and tested the optimizer using stocks). After running SampleMA cross against the symbol in your spreadsheet, I can reproduce the behavior you are seeing. I must be making some assumption about the NT data model which is different for futures and stocks. I have what I need to diagnose.

[Pete S]

OK, let's try this. Ragingbull, you want SQN6, that is the version for 6.0. The one called SQN is for 6.5 beta. Instructions are the same as above, make sure you recompile an indicator or strategy after copying the file.

I should have just asked you what symbol you were trading, I would have noticed this immediately. I've verified that an excel spreadsheet produces the same result as the trades for both stocks and futures.

[PrTester]

Thanks for sharing guys!

PRtester

[Ragingbull]

Bravo Pete,

Excellent job. I made a test on futures with SQN6 and got coherent results with the strategy analyser and a manual cross-checked with Excel.

I guess that you have modified the code to make it compatible with stocks and futures now.

Have good trades and let me know if it is working fine for you.

[Pete S]

I reposted this here http://www.ninjatrader-support.com/vb/showthread.php?t=4520 in the file sharing section, please post any followup discussion there.

[Ragingbull]

Hi Pete,

You have to correct the formula in the post because it is not correct. Should be: SQN= sqrt(N)*average("Profit")/stddev("Profit")


Best regards,

[whitmark]

Great work Pete S/Raging Bull! It has been on my list to improve the existing "optimizer types" with a similar type of multiple objective fitness function. Thank you for taking up the charge.

As you know, the current "default" optimization method is brute force (calculating all parameter iterations), it would be nice to use this type of fitness function to direct the optimization search algorithm and reduce the number of iterations for larger scale optimizations. Are you guys, or anyone else for that matter, planning on taking on development of a genetic algorithm? I have implemented a GA DLL on another platform and the availability of different fill models within NinjaTrader presents some interesting possibilities.

Regards,

Whitmark

[Sunrise]

Yes, It is a pity that Ninja people did not take the opportunity to build in a Genetic optimiser into their platform. It would have made it one of the best Strategy platforms on the market.

Sunrise

PS Ragingbull and Pete S - thank you for this contribution. It has opened my eyes to new ways to work with NT

[Ragingbull]

The only method I know to avoid to have to run systematically all cases in an optimisation problem is called the "Simplex" method. It never mathematically guarantees that you will find the "best optimum" in a space of N dimensions (represented by your N variables that you want to optimise) but if you take some precautions you can pretty much rely on the proposed answer by the algorithm. It does require a consequent amount of programming... If Ninja would integrate this features, combined with SQN defined as an objective function it would become a very powerfull tool too define strategies.

[Sunrise]

Hi Raging Bull.

No need to worry now as a Genetic optimization option will be available with NT 7 (when it is released later this year?!)

Sunrise

[Ragingbull]

Hi Sunrise,

Thanks, I know about the genetic optimiser in V7. In fact, I have been using one for about a year now. I think, it was provided by Pete S.

Best regards.