How to discover profitable automation strategies?

[jimibt]

Without staring an entire book on the subject the #1 sin with any data is backfitting. If you give me data I guarantee I can find you a strategy that would have won. What I can't guarantee is that it'll win ever again. Keep some data to one side for out of sample tests.

#2 would be an understanding of sample size with regard to statistical significance.

#3 Do as much testing and modeling as possible offline. Trialing or refining using money is a really ineffective way to do it and the final stage not the first one.

#4. Be realistic. If manual trading isn't easy then building something that will make you rich while you sit in the garden is even harder.

#5 Fail fast. Only 1 idea in (10?) will have legs so don't get wedded to ideas.

#6 Strategies that have a higher frequency will give you a faster yes/no than infrequent ones. If you trade 200 dogs a day you'll get am answer faster than say trading premiership footy with 11 games a week.

#7 for longevity and scale consider the fundamentals of any market and the things least likely to change. Things like supply and demand, or the tendency for markets to overshoot on price corrections amongst othees. That way it stands a chance of working everywhere dogs/horses/win/place/football or anything. 10p on 1000 trades a day is clearly better than 10p on 100 trades. That's also going to insulate you from the problem of killing all the fish. A strategy might start to fail on one market type but could still work elsewhere. Picking trap 6 on Wednesdays certainly won't do that.

#8 same again. If you're going to spend a year working on the perfect strategy, you want it to last otherwise the return never exceeds the investment.

#9 Expect a cost. For any idea you need to think of it as a raffle ticket. How much you spend depends on how much chance you think you have and the size of the prize on offer. (see #5)

That'll do for now.

Truth is though that there's as many types of automated trader as there are manual traders. Some automate what they've learned manually and focus on niche situations a bot can sit and look for (and you might not even need data for that) and some are hft quants placing 100s of 000s of offers a day for whom tick data and an accurate simulator are a necessity.

Shaun -nice summary. depending on the reader, that is a very encouraging list or a door slammer. i particularly resonate with trap 6 on a wet wednesday afternoon

[Archery1969]

To the OP...

One thing you could look at is the following:

- Looking at the best 3 back and lay prices.
- Ratio of unmatched back money / unmatched lay money.
- Ratio of matched back money / matched lay money.
- Top, middle and bottom of a selections range.

If you combine the above then you will soon start to spot things and therefore be able to automate ideas etc.

Although I am probably giving a bit too much information away.

Have fun,

I don't think you've given anything away. There's only about 10 or 15 market metrics so naming 4 and saying 'combine them and you'll see something' is what everyone has been trying to do for donkey's years. If you can just 'see something" then that's a rare gift you have that not many people could hope to replicate.

Give us a clue archery, what weighing are you giving to each of those signals? what timeframe do you monitor? is the close formulaic too or neutral eg 0s or bsp? Do you flip your position if the signals flip? Offers or taking? The devil is in the detail.

Hi Shaun,

The detail.

With regard to 1st and 2nd favourite:

- 10s, 30s, 1m, 2m and 3m.
- Ratio of unmatched back / lay money
- Ratio of matched back / lay money
- Price range
- Is the money waiting on one side or the other increasing or decreasing.

Normally I offer prices unless I see large sums of money being matched or withdrawn on one side. If it is then I will take what’s left but only if something else is happening on the other side.

Finally, I never actually trade the favourite but it does feature highly in my calculations.

Cheers,

[Crazyskier]

Personally I'm finding that I'm becoming increasingly restricted with off-the-shelf bots that are much more limited than the custom-made coded stuff that only enters a bet if your current bets aren't first in the queue. And the ability to cancel existing bets if XYZ happens, ie prices shrinking / increasing. I see a lot of automated stuff on horses and dogs in particular that are surely very profitable long term, but impossible to replicate using standard software such as BA, etc.

Stop-losses, fill or kills, all have their uses but when trying to be front of the queue as much as possible with a very fluid market, I'm finding myself frustrated...

CS

[wearthefoxhat]

Personally I'm finding that I'm becoming increasingly restricted with off-the-shelf bots that are much more limited than the custom-made coded stuff that only enters a bet if your current bets aren't first in the queue. And the ability to cancel existing bets if XYZ happens, ie prices shrinking / increasing. I see a lot of automated stuff on horses and dogs in particular that are surely very profitable long term, but impossible to replicate using standard software such as BA, etc.

Stop-losses, fill or kills, all have their uses but when trying to be front of the queue as much as possible with a very fluid market, I'm finding myself frustrated...

CS

One bot posted by Dallas over 4 years ago might be worth exploring! (the man's a legend)

viewtopic.php?f=53&t=11785

The WOM setting can be tweaked to suit, but in general, it's a fair starting point to get in and out of a position. Getting in front of the queue is a tough one. I sense the best trades are more momentum based and getting on the tail of a good one makes up for the smaller losses.

[ShaunWhite]

The detail.

Didn't feel the need to stop there If I'd known you'd elaborate I'd have framed my questions better!

[spreadbetting]

Personally I'm finding that I'm becoming increasingly restricted with off-the-shelf bots that are much more limited than the custom-made coded stuff that only enters a bet if your current bets aren't first in the queue. And the ability to cancel existing bets if XYZ happens, ie prices shrinking / increasing. I see a lot of automated stuff on horses and dogs in particular that are surely very profitable long term, but impossible to replicate using standard software such as BA, etc.

Stop-losses, fill or kills, all have their uses but when trying to be front of the queue as much as possible with a very fluid market, I'm finding myself frustrated...

CS

Time to get out those coding books and write your own, plenty of help on the developers site and Liam even released his python framework for free. At the end of the day it'll always be more efficient to use your own bespoke software rather than trying to fudge your strategies within someone else's framework. But it all boils down to exactly what you're doing and how much effort you want to put in. Off the shelf software may well do all you need if not it's time to get out the textbooks or chequebook and get someone else to code it.

[Archery1969]

The detail.

Didn't feel the need to stop there If I'd known you'd elaborate I'd have framed my questions better!

Too late now Shaun, you had one chance to ask all and you messed it up.

But I am confident your doing far better than me.

[ShaunWhite]

Shaun -nice summary. depending on the reader, that is a very encouraging list or a door slammer. i particularly resonate with trap 6 on a wet wednesday afternoon

Cheers Jimi,
Available in all good bookshops soon, and shortly after in all good charity shops and skips.

Saying automation is straightfoward is like saying manual trading is straightforward. Some might get jammy and find something fast just like some people nail manual trading quickly. But for the rest it takes a shed load of effort, money, endless false dawns, and learning how cruel randomness can be; before then going away and learning some basic stats and analysis, without which you'll probably spend months on a backfitted, statistically insignificant, doomed to reversion, filtered to barely 4 trades a day, hair-brained scheme.

I know people do stumble across something reasonably easily sometimes, but I'll have a tenner on the second strategy not being so easy to find. And you'll need more then two if you don't want to be stressing about your only child keeling over and your income dropping to zero.

[ShaunWhite]

that is a very encouraging list or a door slammer.

Automation is as easy as manual trading.
Automation is as difficult as manual trading.

Delete as applicable.

[ShaunWhite]

Normally ....

"Normally" implies data driven but manually executed ?
All the benefits of data and all the benefits of subjectivity, a Cyborg trader? Half rice half chips.

With tight margins 'you' and your grey-i probably contribute a pretty big slice of your return, stripping that way and replacing it with something formulaic can be really tricky. Anyway being semi-automated is cheating, it's too easy, 'proper' full-autos like a bit more of a challenge

btw I loved the bit about "- Is the money waiting on one side or the other increasing or decreasing."

errr yes....now what?

[gazuty]

that is a very encouraging list or a door slammer.

Automation is as easy as manual trading.
Automation is as difficult as manual trading.

Delete as applicable.

Directed at the OP - old hands already know what follows

Actually I think the key to automation is all captured in these lyrics (played forward not backwards)

And if you listen very hard
the tune will come to you at last
when all is one and one is all
to be a rock and not to roll

////

And if you listen very hard - poetic licence, listen, watch observe the markets
the tune will come to you at last - you will notice things that you can exploit
when all is one and one is all - they might be quite simple things, you only need one thing (I have one main idea that accounts for 99% of my profits)
to be a rock and not to roll - don't think you need to copy other people

[wearthefoxhat]

Shaun -nice summary. depending on the reader, that is a very encouraging list or a door slammer. i particularly resonate with trap 6 on a wet wednesday afternoon

Cheers Jimi,
Available in all good bookshops soon, and shortly after in all good charity shops and skips.

You forgot car boot sales...(and an Ebay auction)....

[jimibt]

that is a very encouraging list or a door slammer.

Automation is as easy as manual trading.
Automation is as difficult as manual trading.

Delete as applicable.

Directed at the OP - old hands already know what follows

Actually I think the key to automation is all captured in these lyrics (played forward not backwards)

And if you listen very hard
the tune will come to you at last
when all is one and one is all
to be a rock and not to roll

////

And if you listen very hard - poetic licence, listen, watch observe the markets
the tune will come to you at last - you will notice things that you can exploit
when all is one and one is all - they might be quite simple things, you only need one thing (I have one main idea that accounts for 99% of my profits)
to be a rock and not to roll - don't think you need to copy other people

i presume the geeter solo that follows would be the kerching of checking ones healthy betfair account balance

[abgespaced]

Thanks everyone for your generous contributions.

I'll admit there's more in this thread than I can digest all at once.

One thing that's really pertinent in my mind after reading all of these posts is the idea that automation can actually be an excellent tool for testing manual trading strategies that would otherwise be clouded by emotional decision making.

I've been obsessed with creating a bot over the last couple of days and one thing I've realised is that if the bot doesn't work over 100 trades, then statistically speaking there's no way I could any better if I were trying manually.

One glaringly obvious reason is the speed at which a bot can operate. This is especially true in-play, since I simply cannot react quick enough to the markets. As an example, the first time I used an offset while trading the cricket was the quickest profit I have ever made. Things were getting placed and matched quicker than I knew (or could even see) what was going on, while my green just going up and up. It was magical.

Another reason is that if a bot doesn't work over a large number of trades, then the underlying assumptions are wrong. I therefore fail to see how a human could do any better given that they have an emotional handicap. If something doesn't produce a profit over an infinite number of iterations then how would manual trading be any better?

I also really like the idea of fundamentals. Those core variables that make up the functions of a market. I've tried to define markets mathematically, but being a visual guy the best I can come up with is a graph in ms paint. It's like, I understand the underlying concepts but just don't know have terminology or technical skill (yet) to create solid mathematics or automation around it.

But lots to think about. My first bot went live today after a few hours testing. It's doing well... just needs some tweaks to iron out the quirks produced by the BA software. It seems like there are so many opportunities just waiting to be discovered!

[ShaunWhite]

i therefore fail to see how a human could do any better given that they have an emotional handicap. If something doesn't produce a profit over an infinite number of iterations then how would manual trading be any better!

Because your presence is supposed to be a asset not a liability. Experienced traders don't have an emotional handicap because they know what they're doing and emotion isn't even an issue. You underestimate the value of subjectivity.

You also mention speed, speed is an advantage but you're never going to be the fastest. And unless your strategy is sound you'll just be the first to grab a bad price as easily as trying to be the first to a good price.

Automation is for things you can't do, and if you could do it yourself manually then that's going to outperform an automated version......if you know what you're doing