Beta22 backtesting mess

[dave1992]

It seems Beta 22 is even worse than beta 21 when it comes to backtesting. I get wildly different results from the last stable backtest release - beta 20. I'm not even going to start to analyze the mess, just going back to beta 20. If you have a serious multi timeframe strategy to execute in your testing I'm sure you will see all the problems.

Here's a quick example. I'm logging on execute

19/05/2010 15:18:35   6A: psar4 6A Sell short 1 at 0.8371 - pos: Short 15:18 
19/05/2010 15:18:35   6A: psar4 6A Stop loss 1 at 0.8461 - pos: Flat 15:30
19/05/2010 15:28:45   6A: psar4 6A Sell short 1 at 0.8371 - pos: Short 15:28 
19/05/2010 15:28:45   6A: psar4 6A Stop loss 1 at 0.8461 - pos: Flat 15:30

That last column is the trade time. So at 15:18 my strategy sold, then immediately executed a stop at a wildly different price, and a timestamp of 15:30. Then ten mins later, does the identical same thing, with the same prices.

Here are the orders for the above trades

            6A 12-10             SellShort             Limit             1             0.84             0             Filled             1             0.84             0             Sell short             
            NT-00002             
            4daefd85414e433bb75876241790c8d4             19/05/10 15:28                               6A 12-10             SellShort             Limit             1             0.84             0             Filled             1             0.84             0             Sell short             
            NT-00000             
            8159a53f4a524b12b37b1b8b225651af             19/05/10 15:18                               6A 12-10             BuyToCover             Stop             1             0             0.84             Filled             1             0.85             0             Stop loss             NT-00002             NT-00003             
            f2f29a9be8b6444098127a551686e474             19/05/10 14:45                               6A 12-10             BuyToCover             Stop             1             0             0.84             Filled             1             0.85             0             Stop loss             NT-00000             NT-00001             
            bbe1f0b9db81410d929ddeb96ea39528             19/05/10 14:45

the orders have the two stops executed an hour earlier at 14:45, at a different price again!!

I've also seen double trades.

Dave

[NinjaTrader_Austin]

Dave, can you please post the code you're using so we can test this out here?

[dave1992]

Sorry, I can't. It should happen with any reasonably involved multi timeframe strategy (using minutes and ticks and entering on the ticks)

Dave

[NinjaTrader_Austin]

Dave, we'll need a reproducible strategy in order to isolate and debug this, so would it be possible for you to create a stripped-down version of your strategies for us to test?

[dave1992]

Austin,

I feel that your test strategy should be able to pick it up, but it may be that your test strategy is not complex enough. For example, it should have picked up this bug created from b20 -> b21


4078
NinjaScript

Backtest orders and executions had been marked with timestamps of incorrect series on multi-series strategies with multiple series on the same instrument.

This bug certainly looks related to the ones I have. I think it shows that your testing is inadequate in the area of multi series. As I said, I'm going back to b20, because multi series backtesting is completely broken in b21 and b22, and until you test it properly I don't think it's fair to expect me to test it and write reproducible scenarios for you. I realise what the beta testing process is, but it would save much time and effort on both sides if you just had a comprehensive test suite. Please don't just tell me that you have a comprehensive backest strategy test. It's inconceivable that bug 4078 above could have passed it. It will save you so much time and hassle from users in the future to sort this out. Especially once in production.

Sorry to sound uncooperative but I honestly believe this is the best way forward.

regards
Dave

[NinjaTrader_Bertrand]

Dave, please test with this intrabar granularity sample, this works as expected here on my B22 - http://www.ninjatrader.com/support/f...ead.php?t=6652

Thanks

[dave1992]

Apologies, but I don't have beta22 installed any more.

A quick glance at the strategy you posted and my feeling is that it's not complex enough. It doesn't even have ticks. You need a few minute series, and 1-tick one, and enter on the tick. Use limits and cancels.

Trust me - a short amount of time spent writing a more involved strategy and you get to do more useful testing for ever more. What's the downside?

regards
Dave

[NinjaTrader_Bertrand]

Hi Dave - ok I see thanks for the clarification and added details - so from your testing it broke going from B20 > B21 and was not resolved with B22? I'll amend the sample and compare 20 vs 22.

[dave1992]

I imagine the problem from 20->21 may have been fixed, since it was the bug mentioned. 22 looks even worse.

[NinjaTrader_Bertrand]

Hi Dave, I'll monitor it on the latest B22 but the issues arisen with the incorrect execution stamping introduced with B21 are overcome by B22 for me here. If you have a stripped down strategy sample highlighting the issue you've noticed or think are 'broken' please let me know or contact me at support at ninjatrader dot com.

Thanks

[dave1992]

Oh dear, I waited for RC1, but the same issues are there. See attached trades list. The longer one, RC1, is just a complete mess. It has a list of trades that are theoretically impossible, and incomprehensible. These two lists were produced with the same strat and data, either side of an upgrade from beta20 to RC1.

Oh well, reproducible scenario etc etc. Don't worry, I'll debug and sort it out and tell you where it's going wrong.

[dave1992]

OK, here is the first bit of analysis (it's the same problem as last time)

Here is how it is able to trade so many times while seemingly not flat

08/10/2010 13:34:53 6A: psar6-b20 6A Buy 1 at 0.9725 - pos: Long 13:34
08/10/2010 13:34:53 6A: psar6-b20 6A Stop loss 1 at 0.9659 - pos: Flat 14:00

These are printouts from OnExecution. As you can see, the stop loss is executed immediately at 13:34, with a trade time of 14:00!! and at a completely different price.

I should add that five months of backtest before 8th Oct appears to be identical, so maybe it's something changed from beta20 in how Ninja handles or interprets bad data?

[dave1992]

Right, I've written a strategy which does it. If it doesn't do it for you, maybe you need my data files for that hour. I could email them. Run this against 6A 12-10, 45 Min bars, for 8th Oct 2010

namespace NinjaTrader.Strategy
{
    public class t1 : Strategy
    { 
        protected override void Initialize()
        { 
            Add(PeriodType.Tick, 1); 
            SetStopLoss(CalculationMode.Ticks,32);
            TraceOrders = true;
        }
        
        protected override void OnBarUpdate()
        {
            if (BarsInProgress == 1) {
                Tick();
                return;
            }
        }
                    
        protected void Tick() {
            if (Position.MarketPosition != MarketPosition.Flat)
                return;
            if (ToTime(Time[0]) == 134432)
                EnterLong();
        }    
    }
}

[NinjaTrader_Bertrand]

Dave, thanks for sharing the testing snippet - I'll check into here - why feed are you using?

[dave1992]

It's IQ Feed data, but I'm not connected at the time - it's using NT's db.