Any way to backtest an automated strategy?

User avatar
Euler
Posts: 24700
Joined: Wed Nov 10, 2010 1:39 pm
Location: Bet Angel HQ

I'd say that was a good summary.
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

foxwood wrote:
Wed Nov 21, 2018 10:03 pm
To me, I would model it by thinking what would happen to the market if the certain winner suddenly looks like losing. Follow that line of thinking and also consider how to manage the delay and the risk of only one runner trading at 1.01 and you end up with a strategy that could be nicely profitable with very little downside.

Backtesting is looking at consequences - the data values and moves etc shown in the data are the result of something that happened and just because there is correlation doesn't mean common causation.

Modelling is looking at causation and using the data to confirm your view of what the consequences would be.

Simples ? Probably not - may be coat time ;)
Blimey, that's gonna take some time to get my head around (getting my head away from causation having to be linked to something fundamental)

Really really good insight there though Fox, appreciate it!

No, it defo isn't simple - no-one else was able to explain that view so concisely
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

foxwood wrote:
Wed Nov 21, 2018 10:03 pm
To me, I would model it by thinking what would happen to the market if the certain winner suddenly looks like losing. Follow that line of thinking and also consider how to manage the delay and the risk of only one runner trading at 1.01 and you end up with a strategy that could be nicely profitable with very little downside.
Here's the way the way I think - if you (or anyone) can explain the error I'd be so so grateful.

We know laying 1.01 generically, back-tested is (technically) an edge - although done IRL, probably ins't (hence the dangers).

Let's say, I watch a race live, & notice 1.01 is hit extremely early, before the race has even reached half-way - & the fav gets caught.

I subsequently make a hypothesis that 1.01s hit early in a market tend to fail - & backtest it.

What's the error, basically why isn't that a model?

Cos, that's how I think, & those are the types of ideas I generate daily (albeit using technicals, rather than fundamentals)
LinusP
Posts: 1871
Joined: Mon Jul 02, 2012 10:45 pm

I don’t think I have read a thread on this forum with so many conflicting opinions, which if you ask me is a good thing.

I backtest but looking back it has been very good at telling me what definelty doesn’t work, helpful? Yes, very. I follow Peter W’s advice In getting money through the market ASAP, however the advice being given is very much dependant on the market, sport and preplay/inplay. Without knowing what different forum members specialise in you are in the dark. The sensitivity of backtesting can vary a huge amount and you want to limit this to as close to zero as you can.
sa7med
Posts: 800
Joined: Thu May 18, 2017 8:01 am

Euler wrote:
Wed Nov 21, 2018 2:14 pm
But to model the market and understand how it works with a view to exploiting that understanding of the market.
Would you explain in a little more detail what that process is? Would love to see a video on modeling, are there any existing ones? Is the result something like soccer mystic or tennis trader? How would it work for racing? Are you modeling likely odds or momentum of trends or everything and anything?
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

Interesting read:

https://blog.quandl.com/interview-with-a-quant-part-one

My pick:

"I use Excel for back-testing. This is a deliberate choice: Excel is not as powerful as Python, and this means there is an upper bound on how complex I can make my trading rules. This is a good thing: a strategy that requires complexity to be profitable is probably not a good strategy in the first place.

Excel also allows me to see my assumptions made explicit; it’s easy to lose track of such things when you’re working in code. It allows me to visualize performance statistics (risk, return, drawdowns, capital efficiency, Sharpe ratio and so on) quickly and clearly. Even if my model “works”, there’s no guarantee that a trading strategy built around the model will be economically viable, so these statistics matter."
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

Btw Peter; enjoyed the video ;)
User avatar
ShaunWhite
Posts: 9731
Joined: Sat Sep 03, 2016 3:42 am

ruthlessimon wrote:
Thu Dec 20, 2018 12:06 am
Interesting read:
"I use Excel for back-testing. This is a deliberate choice: Excel is not as powerful as Python, and this means there is an upper bound on how complex I can make my trading rules.
This makes no sense, just because Python executes faster (it's not inherently more 'powerful') there's no need to overload it with rules. Simple rules in Excel are no different to simple rules in Python.
ruthlessimon wrote:
Thu Dec 20, 2018 12:06 am
Excel also allows me to see my assumptions made explicit; it’s easy to lose track of such things when you’re working in code.
If your Python output goes to Excel then there's no difference, visualise to your heart's content. And aren't you working in code in both?
Use whatever suits you, both are fine, no need to justify Excel to any tech-snobs.

-------------------------
I do find the continual bluring between analysis and backtesting strange for a community who live and breathe this stuff. I'm sure others here who've worked in tech esp financial tech, must feel the same way.

Simon's analysis on 1.01 above isn't backtesting, it's data analysis or modelling.

The whole purpose of a backtest is to see if a solution found by analysis would have suceeded in a live environment. Otherwise, what are you testing? And if you call that backtesting, then where's your analysis? Backtesting requires the best available simulation of the live environment. Not necessarily amazing, but the best you can piece together (Something easier to do in Python than Excel btw).

The analysis would show there might be an edge. The backtesting would show it's not obtainable.

That's presumably why in the video, Peter says he looks at lot of data (analysis) but doesn't do much backtesting. Two different things. I suspect that's because when Peter started trading, backtesting would have been really difficult, so he 'grew up' on analysis and then going straight to live. But if you have the capacity to do backtesting of your analysis, then why not? Peter started in the 20th century, times move on, tech improves and the subject matures. Someone looking at this afresh in the 21st century certainly has more options available to them. He'd probably be the first to admit he's not a 'techy' and doesn't do any/much hardcore coding. But if you can............I wouldn't be so keen to dismiss backtesting.

I think Jimi gets what I mean, Jimi can Excel the pants off anyone, but lobbys for the ability to replay data. He does his analysis, now he wants to backtest, I guess? The forum has got lots of threads on analysis, I've never seen one about backtesting and it would be interesting if there were.

I think I've laboured that point enough now :)
User avatar
jimibt
Posts: 3641
Joined: Mon Nov 30, 2015 6:42 pm
Location: Narnia

ShaunWhite wrote:
Thu Dec 20, 2018 5:06 am

I think Jimi gets what I mean, ...... I've never seen one about backtesting and it would be interesting if there were.
this is exactly my point Shaun, you arrive at your conclusion(s) via the analysis and even if the automated rule is super simple, it's still a pain to have to backport it into another coding language (which in turn could introduce small differences that behave in a different way to the BA rule).

having a replay facility literally would assert whether your analysis and more importantly, your BA rule, was functioning as intended. this replay facility would also highlight areas that needed defensive coding etc. the aim (from my perspective) is not to hack our way thro' random code changes, but rather to test that the assumptions are correct in the environment where it will all play out.
Bet Angel
Bet Angel
Bet Angel
Posts: 3999
Joined: Tue Apr 14, 2009 3:47 pm

jimibt wrote:
Thu Dec 20, 2018 9:34 am
having a replay facility literally would assert whether your analysis and more importantly, your BA rule, was functioning as intended.
The issue here is what would you replay? We would not be allowed to sell a product that allowed you to replay your own self-captured betfair data, it would have to be the official data sold by betfair itself. The only data that they sell which is of the quality and detail to reflect the actual market is £199 per month per customer per sport. I don't think we'd get many users willing to give betfair that amount of money for data.
We've looked at the cheaper data that is available that would be allowed to be resold, but it's pretty useless and would give far too many false signals. Imagine the differences between current Practice and Live mode but magnified ten-fold. It's such a fine line between any profitable and loss-making strategy, that such differences are too significant.

Considering the amount of development effort required, any replay service would need to be actively used by 100+ customers to justify spending the time on the feature. We just can't see that many people willing to purchase the £199 data on a regular basis. Perhaps if betfair get more realistic with their data pricing, we can revisit this.
User avatar
jimibt
Posts: 3641
Joined: Mon Nov 30, 2015 6:42 pm
Location: Narnia

BA - i get you. The cost, per seat, does seem prohibative. oh well, make do and muddle thro ;)
User avatar
ShaunWhite
Posts: 9731
Joined: Sat Sep 03, 2016 3:42 am

Forum debates are tricky enough without us all working from the same glossary. (There's an idea for a sticky if ever there was one. The recent difference of opinion about a simple term like 'weak market' is another example.)

Have we all now agreed that there's a difference between data analysis (usually in Excel) and what's more traditionally known as 'backtesting'....and that although the word backtesting is frequently used here, almost nobody does any?

Where I'll agree with BA Dev, is that adding that functionality to BA wouldn't just be prohibitably expensive, it's also bloody difficult. It's in the category of sounds simple but doing it in such a way as to make it meaningful is a real headscratcher. From a purely selfish perspective I wouldn't be keen to see that in BA, afterall those of us who arrive here not especially interested in sport, but interested in tech need some way to level the playing field a little :ugeek: That said, anything that encourages market participation is a good thing, so as with almost everything else, both positions have their pros and cons that could be debated long into the night, or long into a dull winter's afternoon.
foxwood
Posts: 390
Joined: Mon Jul 23, 2012 2:54 pm

Still not on the same sheet - imho data analysis is not modelling as was suggested in an earlier post - my definitions and sequence for developing a strategy ...

1) Modelling - predicting how the market will behave in response to a real world event or unusual market event - then looking at data to check and develop strategy - all about causation of changes in the market - not just chasing a statistical edge.

2) Data analysis - if not an extension of modelling then is effectively the same as data mining and/or artificial intelligence ie looking for profitable statistical combinations in the data - often with lots of filtering to improve confirmation bias ;)

3) Validation (backtesting) - applying the rules of a data based strategy to a totally different dataset to that used while developing the strategy - proves /eliminates whether the strategy has been backfitted to the original data. Vital step often omitted.

4) Testing - feeding data to a strategy implementation to ensure the strategy performs according to specification (ie what jimb asks for - I think). In practice, a bit of thought and code walkthoughs should fix most things.

5) Run live with small stakes.

6) Scale up gradually if works as intended else rinse and repeat from appropriate step.
User avatar
ShaunWhite
Posts: 9731
Joined: Sat Sep 03, 2016 3:42 am

From a quick scan I think I'd be happy to sign-off on that one foxwood.

Sounds like a strong framework and could be used as is, or by skipping the odd step if it's not appropriate for the idea or your facilities. I think everyone from a pure quant to a purely manual trader could perm a few of those and wouldn't be far off. I mention those just as examples of extremes of trading styles, there's no intention to suggest that either are superior or inferior to each other.

...that said, I think 4 is more than a code shakedown, it's what I call the backtest/simulation against an historic data feed. Eg Would I have actually got the prices I need? am I fast enough? how did the outages and api blips affect it? It's the step that happens in the big firms that your average one-man-band doesn't have the facilities to do and gets replaced with your step 5.

yep I see your step 4 as being the unit test phase of the (seperate) development cycle, ie does the code perform to spec.

jeez this feels like being at work again. :shock: I started trading to leave all that paperwork behind :)

4 Code/rule testing
5 Simulation (aka my backtest)
6 Start small in live
7 Rinse and repeat.
Last edited by ShaunWhite on Thu Dec 20, 2018 5:00 pm, edited 1 time in total.
User avatar
ruthlessimon
Posts: 2094
Joined: Wed Mar 23, 2016 3:54 pm

ShaunWhite wrote:
Thu Dec 20, 2018 5:06 am
This makes no sense

If your Python output goes to Excel then there's no difference, visualise to your heart's content. And aren't you working in code in both?
Use whatever suits you, both are fine, no need to justify Excel to any tech-snobs.

I think I've laboured that point enough now :)
I haven't got a clue lol; I was just quoting the woman :)

But I do like seeing "metrics" - i.e. that excessive winrate thingy
Post Reply

Return to “Betfair Data”