Let's Kick RPM Around A Little More (275 views)

[eric]

h/t ANK1990

ESPN's RPM is a composite stat based (mostly) on plus-minus, and then doing super secret math on it that plebes like you and me don't deserve to know so that you split out which player is responsible for which pluses and minuses in each lineup. Great. Sounds great.

But hoooooo boy does it have issues.

For starters let's look at the minutes played and total wins for RPM and Win Shares as a control over the past five years:

Year WS MP Wins RPM MP Wins
2014 595193 1256.4 588314 868.1
2015 595203 1256.9 591784 870.0
2016 594863 1255.6 578523 1170.3
2017 594404 1253.4 592964 1179.7
2018 302034 636.5 301614 593.6
Note there have been 595 wins so far this season, so Win Shares is a little ahead of the game, but you can see how consistent it is in full years. 3 wins of variation out of 1255ish is fantastic precision. Good job Win Shares. RPM, go to your room.

RPM, get back out here. What the f*** happened in 2016? How do you miss 16,000 minutes played? The answer is simple - 53 players just aren't listed for 2016 RPM.



We talked before about how they abruptly and silently changed from doing Wins Over Replacement to just plain Wins in 2016, and we see the evidence again up there. That's pretty bad, but it turns out they've been monkeying with it in other ways too. Let's take a look at the max, min, and standard deviation of RPM and WS/48 over the last five seasons, and we'll only look at players with 1000+ MP (500+ MP for 2018 since we're only halfway through) to keep things kosher.

. . WS . . RPM .
Year Max Min SD Max Min SD
2014 .295 -.036 .052 9.08 -8.44 2.98
2015 .288 -.026 .053 9.34 -6.87 2.94
2016 .318 -.049 .056 9.79 -6.27 2.77
2017 .278 -.023 .055 8.42 -5.69 2.57
2018 .302 -.057 .058 6.52 -5.4 2.19
Now, we expect the max and min to change year over year. Not every best player is exactly as good as each other. Fine. But look at how the overall gap fluctuates relative to the average:

Year WS RPM
2014 99% 116%
2015 94% 107%
2016 110% 106%
2017 90% 93%
2018 107% 79%
Win Shares goes up, goes down, it's random. Real Plus Minus on the other hand is clearly shrinking over time, and dramatically so this year. That's what we in the biz call systematic (as opposed to random) error - we're seeing an actual phenomenon, and this is further reflected when you look at the standard deviation changes by year. Win Shares' barely changes, Real Plus Minus' is dramatically lower this year.

Well, Maybe It'll Get Better as the Year Goes On!

Maybe! But here's the thing, according to what ESPN has told us about RPM it should do the opposite. One of the things these plus-minus stats do is use a prior, which means "okay we've got some data measured this year, but it's small, so we'll tell our formula that the real answer is probably the prior until we hear otherwise." Which is fine! It's a totally fine way of doing things.

So long as you don't change your priors midstream.

And ESPN's (original?) prior was to use box score stats to model out a plus minus, so guys with great box score stats effectively get a bump to the relevant part of plus minus early in the season. I don't think I'm going out too far on a limb when I say guys like Steph Curry and James Harden have pretty good box score stats, so they shouldn't have any problem putting up great RPM numbers from the start...

...unless ESPN changed the prior for everyone to 0. In which case you might have guys with comparatively middling box score stats like Spencer Dinwiddie and Tyus Jones in the top ten of RPM.

Hey look! You do!


real plus minus

.

In memoriam
The Forgotten Boys of 2016

Andrea Bargnani
Anthony Bennett
Beno Udrih
Bryce Cotton
Bryce Dejean-Jones
Cameron Bairstow
Chris Copeland
Chris Johnson
Christian Wood
Chuck Hayes
Cory Jefferson
Coty Clarke
DeJuan Blair
Elijah Millsap
Elliot Williams
Erick Green
Gary Neal
Ian Clark
J.J. Hickson
J.J. O'Brien
James Ennis
Jared Cunningham
Jarnell Stokes
Jason Thompson
Jeff Ayres
Jimmer Fredette
Joe Harris
John Jenkins
Jordan Farmar
Jordan McRae
Jorge Gutierrez
Justin Harper
Keith Appling
Kendall Marshall
Kevin Martin
Kostas Papanikolaou
Lorenzo Brown
Mario Chalmers
Nate Robinson
Orlando Johnson
Phil Pressey
Rasual Butler
Ray McCallum
Russ Smith
Ryan Hollins
Sean Kilpatrick
Sonny Weems
Spencer Dinwiddie
Steve Novak
Thanasis Antetokounmpo
Tim Frazier
Toney Douglas
Tony Wroten
Hey Spencer Dinwiddie! From not even mentioned by the stat to top ten in two years. What a long spence dinwiddie it's been.

I have no idea why RPM missed these specific guys.

It kind of looks like minutes played plays into it:
every one of the 208 players with 1400+ are in
only 3 of the 66 players with 1000-1399 minutes are missing
28 of the 65 players with 200- minutes are missing
...but Rakeem Christmas and Sam Dekker both made it with 6 minutes each.

It also kind of looks like midseason trades play into it:
21 of the 50 guys who were are missing.
But the G-League All-Star (yes this is a real thing) Alex Stepheson and Briante Weber (easily a top ten VCU Ram in NBA history) made it after being on multiple teams for under 200 minutes each.

I don't know.

It's bad.

[Heebs]

When I played RPM it was all about cycling the DR timers with silence and mind control. I wish I could have been on opponents teamspeak when their healer went from poly/poly/poly to getting kicked and then mind controlled out of range.

[eric]

We have demonstrated previously that regular on/off even at the end of February was still ± 6 compared to where it would end up, which is incredibly bad (and it would get worse for regular on/of from there). I was interested to see whether RPM improved on that, so I decided to look back, way back, at RPM values throughout the season. (It's not possible to simulate RPM values because the calculation is TOP SECRET.)

The first and most inexplicable thing I found is that "where it would end up" turns out to be pretty murky. With every other stat I've ever seen, you get to the end of the season and that's that. You scored X points, you accrued Y Win Shares per 48 minutes played, if you're Andrew Wiggins you tallied 0.1 assists per fortnight. RPM doesn't work that way.

The ESPN site maintains a record of values for previous seasons back to 2014, with an odd message that says each was last updated in November or so of the calendar year, which of course is many months after the season ended and encroaches on the next. I remember seeing this the first season RPM was posted and not seeing any difference after the values were "updated" in November. Whether that was true or not then, it's definitely not now. Check out the ESPN 2017 values as of November 13, 2017 and compare to the Wayback Machine's as of June 13, 2017 (i.e. after not only the regular season but the playoffs were finished). Somehow in the offseason LeBron's RPM improved by 0.26, and that's not all! Of the 40 players listed on the front page (the only one in the archive) 24 stay the same, 10 worsen, and 6 improve...

...without a single more minute of basketball being played in the season!



Not only that, 6 players see position changes. "Aha," you may be thinking, "that's why they improved!" Alas, no. In fact five of the six who changed position by doing literally nothing saw no RPM change at all, and Nikola Jokic saw a mere -0.01 change moving from center to power forward.

This.
Is.
Bananas.

Nikola Jokic did not change position in the 2016-17 season during friggin' August 2017, okay? The season already happened, nothing about it changed. Stop, RPM, you're traumatizing me.

There are a lot of potential reasons for numbers to keep changing after the playoffs end:
maybe they use future data to stabilize the numbers the same way they use past data,
maybe they change the methodology in the offseason (we already know they do this to some extent),
maybe they just don't process data that quickly and they're still working regular season data into the machine,
but if any of those were the answer almost everyone would change, instead of most people staying the same.

My best guess is that they take playoff data into account, and do so throughout the playoffs but not in a super timely manner. The only players to change more than .01 in either direction are Cavaliers and Warriors, who played in the Finals in 2017, and all the Cavs and Dubs on the front page did so. However, this does not explain the .01s, who include players who didn't play in the playoffs at all like Nikola Jokic, or why players who went quite deep in the playoffs like Jae Crowder saw no change.

.

Bottom line, it's a mystery.

Stats aren't supposed to change mysteriously.

That's bad.

.

.

We can still see how much a player's RPM changes during the season, though, and since it appears to be the final final answer and since it's the only full dataset I have anyway I went with the numbers as of November 2017 as the comparison. With the Wayback Machine I was able to pull individual pages from Feb 14 8 AM, Feb 14 2 PM (yes this is relevant, yes it's ridiculous), Feb 27 2 PM, and Mar 16 2 PM.

Let's talk about February 14th. On that morning an upload occurred with guys like C.J. Miles, J.J. Redick, and J.R. Smith. That afternoon another upload occurred with guys like C.J. Watson and D.J. Augustin, who hasn't missed a sim league free throw in decades. This is all well and good, except in the final upload the names are spelled CJ Miles, JJ Redick, JR Smith, C.J. Watson, and D.J. Augustin.



Now to be fair this is how each of those men actually spells their name. But a) why are you changing it in the middle of a day let alone the middle of a year and b) why aren't you publishing that you're changing around your database. Stats aren't supposed to change mysteriously! Go to your room, RPM!

Anyway let's look at the ± compared to the November 2017 numbers for each date, along with the % of minutes played:

%mp ± date
71% 1.49 Feb. 14
74% 1.55 Feb. 27
84% 1.45 Mar. 16
The ± is quite small! That's good!

But it definitely, definitely should vary with how many minutes played there are! That it doesn't kind of makes it look like RPM isn't taking the additional time into account at all! That would be a big problemo! They're asking us to trust them that their black box is taking the inputs and turning them into outputs, if the inputs change and the outputs don't that's what we in the stats biz call a big ol' red flag.

.

.

.

It could be a coincidence. We're also dealing with subsets. So what I'm doing is archiving every page of the espn page every month starting today, making sure to hit April 12th (the day after the regular season ends) and each round of the playoffs. Of course I expect them to secretly change their methodology again in a couple years so this will only tell us so much, but if it tells us something bad I see no reason not to continue believing it until they deign to prove otherwise.

Stay tuned, sports fans!