NBA: The State of Stats (315 views)

[eric]

The State of Stats

There are so many stats these days I thought it would be helpful to do a brief overview. I won't go into the explicit formula or statistical derivation of any of them, to my mind the more important thing is whether they work.

.

But How Do They Work?

All composite stats have the same goal: how much does Player X help or hurt their team's prospects of winning? They assign various and usually interrelated weights to their inputs and spit out an answer in Wins. (Some metrics say Wins and mean Wins Above Replacement, but it's the same principle.) To accomplish this there are two basic types of inputs: box score and plus minus.

The advantage to stats that only use box score as input is that they have a much larger sample size to work with, going back at least to 1978 when the NBA first started measuring turnovers. It is also usually possible to extend such a stat to at least 1974 (offensive and defensive rebounds, blocks and steals) and even as far as 1952 (minutes played), though of course these necessarily become more like estimates and less like measurements.

The disadvantage is that the box score does not measure everything; floor spacing, hockey assists, boxing out, screen setting, will to win, etc. all have no entry in your local news-paper, so many stats instead rely on plus minus. The disadvantage there is that such records are only available back to about 2001 depending on whom you ask, and cannot possibly be extended prior.

One solution that has gained a lot of currency recently is to use both sets of inputs. We shall see if that solution is effective.

.

Alright, So Which are Which?

Anything that has "plus minus" in its name will use plus minus, and anything that doesn't won't. Unfortunately, all of the ones that do use plus minus do not have publicly available formula. This is a red flag for lots of reasons: there's no way to apply the formula outside the publicly available results, there's no way to consider the formula a priori, if the publicly available database goes down or is terminated there's no way to reproduce it, the methodology could be being misstated by whomever is publishing the results, they could be making honest mistakes in the calculations, they could just be making it up as they go. These concerns are best illustrated with a quick gander at Jeremias Engelmann's Real Plus Minus as published by ESPN.

As stated earlier, plus minus goes back to 2001, so it is theoretically possible to calculate RPM back about that far. Some stats use a season or three of past data to help smooth out year to year variations, but even with that in mind it's very odd that RPM starts only in the 2014 season. The pages of past years mysteriously keep getting updated well into the next season, although as near as I can tell the values themselves aren't actually changed. There's no linked explanation of how the stat works or if/how/why it's better than any other, to find that you have to search the ESPN site, good luck with that. Most critically, they have without announcement or comment whatsoever fundamentally changed how the formula works.

.

Wait a Minute Smart Guy, I Thought You Said the Formula Wasn't Public?

That's true.

So How Do You Know It's Changed?

By an ancient and obscure mathematical technique known as "addition". There are 30 teams in the NBA and they all (usually) play 82 games that must end in exactly one win and exactly one loss, therefore there are 1230 wins to go around every year. If a stat outputs Wins, it should therefore add up to about 1230. If a stat instead outputs Wins Above Replacement, it should output a number smaller than 1230, with the precise amount depending on how that stat defines replacement level. Conceptually a team of replacement level players should be near the worst in most any NBA season but it would nevertheless generate more than literally zero wins. I'll go into this in much greater depth later, but for now let's just look at the past three years of RPM, Justin Kubatko's Win Shares as published by basketball-reference.com, and Daniel Myers' Box Plus Minus as published by the same. The latter two are explicitly described as being flat wins and wins over replacement player respectively, and you'll see what I'm talking about.

year Real PM WS Box PM
2014 868 1256 813
2015 870 1256 810
2016 1172 1255 811
WS and Box PM are right around the same spot every year, and Box PM is built around a team of replacement level players generating about 14 wins. The first two years of Real PM look a lot like Box PM except they have a different replacement level of about 12 wins. That's not a major concern. What's a major concern is that the NBA abruptly jumps 300 wins in 2016 according to Real PM. This change is so drastic and it occurring without comment is so bizarre that I even counted up how many minutes played were in each sample to make sure the website itself wasn't somehow horribly broken, but in all years there were about 590,000 MP which is about right. Note that this doesn't mean the website isn't somehow horribly broken! Just that it's not horribly broken in a way that would for example list every Atlantic Division player twice or include the playoffs or something else like that.

.

How Can Mirrors Be Real if Our Stats Aren't Real?

It's a real conundrum. I will say though that I've been looking at these composite stats for a long time and I've never seen anything even close to like this. I've been in contact with the stat's creator and he describes the year over year change as "very slight", so either he's not privy to what ESPN's doing with his methodology or we have wildly divergent definitions of the word slight. In any event, over the coming days I'm going more in depth on various stats. First of all we'll look at the interestingly distinct assumptions they make, then we'll look at the results. Getting the right number of wins in an NBA season is a pretty low bar, let's see who gets the (most) correct number of wins for each team.

[SPL]

my head hurts

[Lazy Pete]

keep pulling that thread, Eric

[MJ]

What

[ANK1990]

This was a good one

[eric]

A Rundown of Assumptions

All composite stats have the same method: weight your inputs so that the output best matches the data, in the hope that it will continue to do so outside of the sample (i.e. in the future). There is of course enormous diversity in the specifics of that method, and I will go over some of them here.

.

Player Efficiency Rating

The composite stat that everyone knows, created by John Hollinger and posted just about everywhere, though for reasons that will be made clear shortly it's useful to link to ESPN. The formula itself has two main pieces: scoring and other. The other section straightforwardly weights box score stats by the league average Value of a Possession, so a turnover is -1 VOP, a steal is +1 VOP, and so on. Scoring is essentially that a point is worth a point, with some credit taken away based on how often the player's team generates assists and some given back for how often the player records assists. The net result of all this is a rate efficiency stat that is pace- and league- neutral by construction.

Except it's not.

If you construct two players on identical teams in identical leagues and themselves identical but for the number of shots they take, the player that takes more shots will have a higher PER. That's not how efficiency works. That's not how any of this works.

.

Let's put aside the terminology issues and return to the ESPN page, where you may have noticed a column on the far right labeled EWA, for Estimated Wins Added. This is a much less well known stat than PER though it is derived from it; specifically, you subtract a position-specific replacement level from a given player's PER, multiply the result by their minutes played, then divide by 2010 because that's what makes the numbers work.

I want to stop for a moment to stress the consequences of the bolded assumption. For example, is LeBron really a small forward? or really a power forward? or how about a point guard or a point center? How would we even begin to define that in an objective way? You want to go by height? Okay Magic Johnson is a small forward. Assists? Okay Magic and Chris Paul are directly comparable. No matter which way you go you end up in absurdity. The only winning move is not to play: there can't be explicit position adjustments because there's no such thing as positions. Luckily the replacement levels for PER vary by at most 10%, so it doesn't make that much difference, but still.

.

.

Wins Produced

Developed by David Berri, Martin Schmidt, and Stacey Brook, this is a much more intricate approach based on derivatives and statistical modeling. That's nice. The real difference between this and PER is the range of inputs included; instead of looking almost exclusively at the player's box score, we also take information from the team's box score. For example, we include points allowed and team rebounds. Wins Produced follows the general PER idea that a steal is great and a block is okay, but in recognition of the fact that there's a heck of a lot more to defense than that it then adds a team based adjustment to every member of that team. Unfortunately the peak to peak difference in the value of these adjustments is only about .05 WP/48, but at least it's something. The point of the team rebounds term is that players primarily take defensive rebounds from their teammates, so the formula smooths out the difference between high and low defensive rebounders.

They're interesting ideas and we'll see later on how well it all works, but the main thing I want to bring up with Wins Produced is, again, position adjustments. Ugh! And it's far worse than the PER situation: the relative difference in positions is as high as 187%(!) and the original methodology differs from boxscoregeeks, the only publicly available tables. These are huge differences resulting from completely arbitrary disagreements, which is profoundly troubling.

.

Win Shares

Developed by Justin Kubatko (with a heavy and acknowledged debt to Dean Oliver) and posted on basketball-reference. Like Wins Produced it goes into much greater detail, especially on the defensive side. Unlike Wins Produced, it starts with team defense and then jimmies it based on the player's steals, blocks, and defensive rebounds, then throws in a term for opponent missed shots and turnovers in general.

Again, why not. Sounds good. We'll see the proof in the pudding soon enough. The great thing about Win Shares though is that there's no positional adjustment. A rebound is a rebound. A pass is a pass. And look at the top twelve Win Share havers last year: four bigs, four wings, four point guards. Same positional breakdown as the top twelve Wins Produced last year, and in fact eleven of the same twelve on each list, the only difference being #10 WS Andre Drummond instead of #12 WS LaMarcus Aldridge. Not bad, Star Fox!

.

.

Box Score Plus Minus

I said way back in the first post how some stats use box scores as input and some use plus minus, the problem with plus minus being that it's a much more limited data set. Several bright people have had the bright idea to use box score as the input for plus minus output instead of wins output. The idea here is forget wins, let's take our box scores and model plus minus instead. We'll then figure out how to convert our plus minus to wins, and with that we can apply our box scores to plus minus transmogrifier to the additional twenty some odd years where box scores were recorded and we'll have the same sample size as any other pure box score stat.

It's a bright idea, and the best version of it is Daniel Myers' Box Plus Minus over at b-r. It even has a very simple formula! But it also takes quite a naughty step. You know the team's true plus minus for sure, that's easy. If all the players' BPMs don't add up to it, you just fudge them by the difference. This is very naughty. It would be like Win Shares saying "okay add up all the Win Shares on the team and if they don't match the team's wins then yeah they do, 100% accuracy, wow, what a stat." Because the conversion from plus minus into wins is imprecise especially on the player level we'll still see variance for BPM, I just want to stress how none of the other stats take this step and it's a critical difference.

.

.

True Plus Minuses

The stats that use plus minus as input are a lot more straightforward at the top level. When a player's on the court his team scores points and the other team scores points, take the difference, plus is good and minus is bad. What they all struggle with is that basketball teams always have five players on the court, no more, no less. On top of that, their playing time is not randomly assigned at all. There are starters who get more minutes, starting lineups who disproportionately play more together, backups who tend not to play with the relevant starter, it's a real mess.

The main wrinkles in plus minus stats are regularization and priors. Regularization means that the sample sizes get so tiny they're bound to be lying, so we'll assume a +20 result is more like a +10, a +2 more like a +1, and so on, with the specific mapping depending on the stat. This includes the choice of prior, or what we know the stat ought to spit out. This can be based on the box score, or based on previous years of the stat itself, because like any good American it can pull itself up by its bootstraps.

The bottom line is that the only publicly available version of this stat is ESPN's Real Plus Minus. There have been loads of others that just went kaput: 82games, basketballvalue, Individual Player Value, Adjusted Plus Minus, poof. All gone, often because the creators got hired by NBA teams who presumably preferred to keep the info to themselves. It's on the one hand sad because it's such an interesting premise and I'd love to see more people take a crack at it, but on the other hand it's nice to see stat heads making bank, so kudos to them.

.

.

Tomorrow I'll get down to the results: which stat best describes how many wins the teams had when you sum up all the players? There are so many differences between all the stats that I don't intend this as a referendum against any particular assumption made by any of them except positional adjustments are SO BAD, the main point of all this is to know which of these stats we should even be looking at and which we should disregard.

[eric]

Now it's time for the finale. Which stat does the job? I took the years of 2014, 2015, and 2016 because that's as far back as RPM goes and because all the stats were created before those years, so we're out of sample for all stats. I summed each stat across all players on each team, subtracted that amount from the actual team's wins, squared it, averaged it across all teams in all years, and took the square root. This is called Root Mean Squared Error. It's cool. Trust me.



foh PER

For Real Plus Minus I used replacement level for the first two years and didn't for 2016, so I'm not even penalizing them for their methodology switch, and they're still neck and neck with Box Plus Minus for third place. foh Real Plus Minus.

Box Plus Minus naughtiness notwithstanding is still clearly behind the big two. Even its best year can't compete with their worst years.

Incidentally I also found a very old reference to Adjusted Plus Minus for the 2008 season. This is a stat that takes plus minus input like Real Plus Minus. Like all other plus minus based stats they don't go into detail on how they calculate it, and it technically doesn't say how it is intended to be converted to wins, but using the most generous assumptions I got it down to about a four RMSE for that single season. That's good, definitely better than RPM, but still way behind the big two, and as the stat is long since defunct the point is less to evaluate whether you should use it or not, but whether plus minus input stats in general live up to the hype. They don't. Box score stats are not just as good as plus minus stats, they're better.

I mentioned in a previous post how boxscoregeeks uses different positional adjustments. I acquired the data they used to have posted for 2014 with the old positional adjustments, and the RMSE for that year alone was 3.98 for the old way and 4.02 for the new way so at the team level it doesn't change much. An individual player can change by five relative Wins Produced a year, though, which is kind of a lot when even the best players barely hit twenty.

It's really close between Win Shares and Wins Produced, and the year to year variations are such that in a couple years they could easily flip flop. If you want to model how many wins a team generated, either is fine. If you want to model how many wins an individual player generated, I advise you to go with the metric that won't say Player A did 20 and Player B did 10 under one set of positional designations and 15 each under another set. That doesn't seem super reliable to me.

If you want to see how good a player was and you only want to look at one number, look at Win Shares.

.

Alright, so sugar is tastier than flour. But if you combine sugar with flour, you get something tastier than both: a cake. (I don't know how cooking works.) It's reasonable to suppose that combining these stats into a super stat will give an even better model, and as it turns out that is the case.

The best RMSE was Win Shares' 3.17.
The best RMSE for all combo compo stats is 3.03:
-twelve parts Win Shares
-two(!) parts Wins Produced
-two parts Box Plus Minus
-two parts Real Plus Minus
-one part PER's Estimated Wins Added(!)

Yes! Even PER, that mongrel idiot, will (very slightly) improve our RMSE! But come on, having to look up stats in five different places? Can't we get most of the way there with just two?

Yes! A recipe of four parts WS and one part RPM will get you down to 3.05. (No other two will even match Win Shares by itself.) (In fact, even all four put together can't do that.) (Win Shares Rules #1.) This is a really interesting result to me because for all its many glaring faults, Real Plus Minus is (probably) a plus minus input stat, and by dint of being the only such stat we have it's the best one, which suggests that if we got an actually good one we could combine it with the best box score input stat to get something much stronger than both.