Betfair API : Streaming Data API

Post Reply
xitian
Posts: 441
Joined: Fri Jul 08, 2011 2:08 pm

Fri Mar 10, 2017 11:00 am

Question for the folks who write their own programming code and connect to the Streaming Data API:

Do you use data from ExAllOffers, ExBestOffers or ExBestOffersDisp. I'm currently relying on ExBestOffersDisp because I'm assuming that it's worth having cross-matching data on to see the "fullest" market possible, but now I'm wondering if I should use ExAllOffers after all. I haven't backtested against the other data streams yet, but I was just interested if anyone else has spent any time thinking/testing this. Of course which you pick might only be relevant to your strategy, but I'm still interested. Here's a few points I'm thinking:

1. All streams seem to update at the same time, so there's no speed advantage based on when you get the data.
2. ExBestOffersDisp shows cross-matching money, which perhaps for strategies which take liquidity might be an advantage to see?
3. I used to think using only a few levels for ExBestOffersDisp would require much less data than ExAllOffers, but that's not the case! Since, when level 0 (front of the book) changes in price that's a single value updated for ExAllOffers, but for ExBestOffersDisp you get an update for all the levels because level 0 is now level 1, level 1 is now level 2, etc... So ExAllOffers receives less data, but also requires less processing time.
4. Based on the reasoning in 3, I see absolutely no reason to ever use ExBestOffers - unless for some reason you only want 1 or 2 levels of the book.

In the past I've been quite careful not to subscribe to any more data than I absolutely require, but in future I'll probably gather a bit more so that I can backtest and see if it makes any difference which one I use for my strategies. Perhaps the difference will be negligible, but who knows.

LinusP
Posts: 1430
Joined: Mon Jul 02, 2012 10:45 pm

Fri Mar 10, 2017 1:54 pm

Point 3 is interesting although I think you will get a lot more data to process if selecting all offers as the book is constantly updating due to offsets etc.

I request best offers (ladder levels 3), I am not interested in display as I don't trust the data returned, not sure what happens when you request it and xm is off? There are times when xm can hide opportunities and I am not completely sure that it offers betters the best price every time, especially when the market is volatile.

Would be interested to hear of the results if you do compare.

xitian
Posts: 441
Joined: Fri Jul 08, 2011 2:08 pm

Fri Mar 10, 2017 2:21 pm

Thanks for your input. I definitely agree cross-matching is a bit of a black box. Not sure if it's something that's beneficial or detrimental to me until I run a backtest.

As an initial measure of the bytes of data for maintaining ExAllOffers vs ExBestOffersDisp/ExBestOffers for the first race today for the favourite - I measured about 500kb length of data for the ExAllOffers stream against just under 2,000kb for ExBestOffersDisp with 10 levels. I think restating of all of the levels if just one price at the front changes is working out as a lot more data than things changing lower down in the book.

Not sure any of this has any noticeable impact on running strategies, but does make me inclined to use ExAllOffers given it's less data and includes all the levels should I ever need it in future. However if ExBestOffersDisp turns out to be more beneficial to me, then that would be a bigger trade-off.

LinusP
Posts: 1430
Joined: Mon Jul 02, 2012 10:45 pm

Fri Mar 10, 2017 3:06 pm

That's very interesting and tbh I find it hard to believe! Out of interest how did you measure this / what was the time frame, pre-play?

xitian
Posts: 441
Joined: Fri Jul 08, 2011 2:08 pm

Fri Mar 10, 2017 3:54 pm

Yes, sorry, that was pre-race for that first race. There would probably be less of a difference in-play.

I didn't do anything very technical. I write all stream updates to a database, and I just pulled them out for one runner and measured the length.

I see a lot of updates like the following for ExAllOffers:
1.96,0

With an equivalent for ExBestOffersDisp:
0,1.94,14.07;1,1.93,19;2,1.92,56.31;3,1.91,33.93;4,1.9,59.26;5,1.89,6;6,1.88,133;7,1.87,31.83;8,1.86,7.45;9,1.85,4

I'm sure you do get changes further down the book for ExAllOffers (beyond the front 10 levels), but those don't seem to outweigh the data to maintain level information in BestOffers.

LinusP
Posts: 1430
Joined: Mon Jul 02, 2012 10:45 pm

Sat Mar 11, 2017 11:20 am

I guess the amount of levels requested is going to impact this as 10 is quite a lot. I have setup two ec2 nano instances for today's racing, one with ALL_OFFERS and another with just 3 ladder levels, will post a screenshot of the metrics this evening.

It would be nice if betfair offer a bit more information on this subject but then its gonna be market dependent.

xitian
Posts: 441
Joined: Fri Jul 08, 2011 2:08 pm

Sat Mar 11, 2017 11:52 am

Ok, that will be interesting. I expect they'll be roughly the same amount of data. Like you say, probably varies by sport. I would have thought something like football will constantly have the front of the book changing due to the way prices decay.

It's interesting that Betfair changed the default for the number of levels from 6 to 10 (which is apparently the max anyway).

LinusP
Posts: 1430
Joined: Mon Jul 02, 2012 10:45 pm

Sat Mar 11, 2017 5:32 pm

Ok the test didn't go to plan, I ran out of CPU credit in less than 30 minutes when requesting the following for all of todays racing (blue):

Code: Select all

fields=['EX_ALL_OFFERS', 'EX_TRADED', 'EX_TRADED_VOL', 'EX_LTP', 'EX_MARKET_DEF']
Base (orange):

Code: Select all

fields=['EX_BEST_OFFERS', 'EX_TRADED', 'EX_TRADED_VOL', 'EX_LTP', 'EX_MARKET_DEF'],
ladder_levels=3
screenshot.png
Maybe its my framework and the number of updates that come through which then have to be processed that is causing the high CPU but then the 'network in' is double that of the base.
You do not have the required permissions to view the files attached to this post.

xitian
Posts: 441
Joined: Fri Jul 08, 2011 2:08 pm

Sat Mar 11, 2017 5:44 pm

Hmm.. interesting. Thanks for the plot. Was it actually running for the whole of that graph then, or did the blue ALL_OFFERS stream just stop when that line dropped down?

Also, any idea what all that Network Out is for the orange line?

LinusP
Posts: 1430
Joined: Mon Jul 02, 2012 10:45 pm

Sat Mar 11, 2017 5:50 pm

I let it continue but once it was out of CPU credits it dropped down to 5% for the rest of the afternoon, this caused a major backlog of events to be processed. By the time I stopped the program I was getting latency of around 300s (current time - betfair publish time)

I would ignore the network out as that is the program sending zipped market book csv's to S3 after each race, so the test wasn't in 'ideal' conditions but gives some guidance

Post Reply

Return to “Betfair API”

  • Information
  • Who is online

    Users browsing this forum: No registered users and 2 guests