Mark, did you ever get/find an answer to your optimisation problem? Or is it still a case of "there's no (known) optimal way to optimise"?
Thanks.
How do you know you have a profitable system?
I recently did a video on assessing the profitability of a system in a simple way: -
https://www.youtube.com/watch?v=S-i3fSkT9LI
Returns should regress to mean the longer you run a system.
https://www.youtube.com/watch?v=S-i3fSkT9LI
Returns should regress to mean the longer you run a system.
-
marksmeets302
- Posts: 527
- Joined: Thu Dec 10, 2009 4:37 pm
I have a deal with the university where they will present my exact case to students. This will happen in november.Mark, did you ever get/find an answer to your optimisation problem? Or is it still a case of "there's no (known) optimal way to optimise"?
Oh, yeah, sorry. I got my dates mixed up and thought that was from last year November!marksmeets302 wrote:I have a deal with the university where they will present my exact case to students. This will happen in november.
I'm so used to time flying nowadays that I got confused. Just ignore me!
-
marksmeets302
- Posts: 527
- Joined: Thu Dec 10, 2009 4:37 pm
Very nice video; it captures what the "excel method" tries to do. As you gain more samples, the average return divided by turnover should stabilize. That is, it converges to a number, and the wiggles in the line become smaller: the standard deviation gets lower. When you get to a point where the average minus 2 standard deviations is still positive, then you have shown that with 97.5% confidence you can state that your strategy works.Returns should regress to mean the longer you run a system.
You can also reverse this, and this is maybe a nice solution to a problem addressed in the video: you're in a drawdown and your gut tells you to call it quits. If you do the same calculation and the average plus 2 standard deviations is negative, then yes, with 97.5% certainty this is a bad system. If not, it might just go through one if its natural down cycles and it might be better to hold on.
-
ricardodeano
- Posts: 205
- Joined: Tue Oct 06, 2015 2:03 pm
This really is a brilliant video and has given me renewed hope. Time to go back and start all over againEuler wrote:I recently did a video on assessing the profitability of a system in a simple way: -
https://www.youtube.com/watch?v=S-i3fSkT9LI
Returns should regress to mean the longer you run a system.
One thing I will say, is that something that would significantly aid this would be the ability to identify different rule sets in the export csv files i.e. by rules name present against each line as per my suggestion here
-
marksmeets302
- Posts: 527
- Joined: Thu Dec 10, 2009 4:37 pm
My students got back to me with their results. Their approach was somewhat different than what I expected: instead of using a t-test they created new time series based upon real, historical ones. By doing this several hundreds of times and run a strategy on them, they could then estimate confidence levels.
While I'm happy they did it this way, because I gained some knowledge, it's also a bit disappointing because it can't be directly used for the betfair strategies. The difference is that with betfair we only have one single time series: the odds. In the financial markets we have one time series per asset, and they go through regimes with correlations with each other.
The students report can be downloaded here: http://newhighs.nl/download/2MMR10___Gr ... ersie2.pdf
One of the new forum users said he had a background in maths and statistics - I'm hoping he will add his comments in this thread.
While I'm happy they did it this way, because I gained some knowledge, it's also a bit disappointing because it can't be directly used for the betfair strategies. The difference is that with betfair we only have one single time series: the odds. In the financial markets we have one time series per asset, and they go through regimes with correlations with each other.
The students report can be downloaded here: http://newhighs.nl/download/2MMR10___Gr ... ersie2.pdf
One of the new forum users said he had a background in maths and statistics - I'm hoping he will add his comments in this thread.
Thanks for sharing the students' report, Mark. I finally got round to reading it.
It wasn't quite the question I was expecting to be answered either (and not the method as you already mentioned). It was quite an interesting approach though.
But I was more expecting you to ask them how to safely select the best parameters to a trading model. And/or compensate a significant test based on the fact you had backfitted the best input parameters. (Surely the answer isn't that you don't have to?!)
I'm actually warming to method the students used now actually. I can see why they did it. It's just very dependent on your timeseries model and relying on what's happened in the past (which is obviously a big problem). And not very helpful for our sports scenarios as you already mentioned.
It wasn't quite the question I was expecting to be answered either (and not the method as you already mentioned). It was quite an interesting approach though.
But I was more expecting you to ask them how to safely select the best parameters to a trading model. And/or compensate a significant test based on the fact you had backfitted the best input parameters. (Surely the answer isn't that you don't have to?!)
Maybe that question is just impossible to know the answer to since you can never know how much you have overfitted. Perhaps the age old method of using out-of-sample data for cross validation is the way to go. But even that is fitting to data to an extent.marksmeets302 wrote:Right now I'm looking at the following question: I have a system that works, and I can make it even better by tweaking one of the variables. By testing it on historical data I can pick the best value. I know that this can lead to overfitting, so if I do a test for significance again I have to compensate for the fact that I already picked the best variation. From the internet I've already found 3 approaches on how to do that and they all give different results...
I'm actually warming to method the students used now actually. I can see why they did it. It's just very dependent on your timeseries model and relying on what's happened in the past (which is obviously a big problem). And not very helpful for our sports scenarios as you already mentioned.
-
marksmeets302
- Posts: 527
- Joined: Thu Dec 10, 2009 4:37 pm
After describing the problem to the students they called me the next day, pointing out that I was asking a weird question (although they were very polite, that's what I thought at that moment 
). Basically I was asking for confidence intervals on something that wasn't a stochastic process - I had just one fixed set of data. So, they said, we can tell you that with 100% probability you had a sharpe ratio of x in the past. I know, I said, but what about the future? That's how we got from the original problem to the one they solved.
I'm still a bit uneasy about how they generated the new time series, because now the problem shifts to "how confident can we be that those new time series describe new possible futures". In a sense the whole process left me more confused than I was before.
I guess the difference with testing whether a cure is effective is that the way people react to the cure shouldn't be dependent on time. The way a strategy reacts to the market is something that changes, because the market changes over time. That's why you can do a statistical test on drugs, but it gets a little trickier in "our" case. Not sure if this is a good analogy, but it makes sense to me.
). Basically I was asking for confidence intervals on something that wasn't a stochastic process - I had just one fixed set of data. So, they said, we can tell you that with 100% probability you had a sharpe ratio of x in the past. I know, I said, but what about the future? That's how we got from the original problem to the one they solved.I'm still a bit uneasy about how they generated the new time series, because now the problem shifts to "how confident can we be that those new time series describe new possible futures". In a sense the whole process left me more confused than I was before.
I guess the difference with testing whether a cure is effective is that the way people react to the cure shouldn't be dependent on time. The way a strategy reacts to the market is something that changes, because the market changes over time. That's why you can do a statistical test on drugs, but it gets a little trickier in "our" case. Not sure if this is a good analogy, but it makes sense to me.
-
marksmeets302
- Posts: 527
- Joined: Thu Dec 10, 2009 4:37 pm
I send him a PM, hope he takes the bait
I think Euler's video on testing is pretty good. (The guy is making a living off this, so we are not surprised...)
I also agree with the comment that profits of a strategy will often decline over time. So it is best to have multiple **profitable** strategies running. If a strategy loses enough times, you should have a mechanism to stop you trading it further. Remember Betfair has data scientists who can easily look into profitable accounts and reverse engineer your logic.
I saw someone mention "optimizing" strategy parameters to maximize profit. This is the wrong approach. You should really be measuring the parameter sensitivity of your profitability. If profits change a lot when a parameter value changes, you are in trouble! (i.e you will be overfitting by optimizing the parameter)...
Good luck
I also agree with the comment that profits of a strategy will often decline over time. So it is best to have multiple **profitable** strategies running. If a strategy loses enough times, you should have a mechanism to stop you trading it further. Remember Betfair has data scientists who can easily look into profitable accounts and reverse engineer your logic.
I saw someone mention "optimizing" strategy parameters to maximize profit. This is the wrong approach. You should really be measuring the parameter sensitivity of your profitability. If profits change a lot when a parameter value changes, you are in trouble! (i.e you will be overfitting by optimizing the parameter)...
Good luck