On Off Plus or Minus (251 views)

[eric]

Prompted by a recent shout discussion vis a vis Marcus Smart, I wondered about on/off confidence intervals over time. When we talk about small sample size effects, what we're talking about is signal versus noise. Noise decreases as sample size increases, but how big specifically does a sample size have to be for a certain change in on/off to be statistically significant?

Thanks to the indispensable basketball-reference, I took LeBron James' monthly splits and in so doing computed his on/off at the end of each month, then graphed it against his on/off in that regular season, and did this for all years from 2011-2017 except 2012. (Note that there's no a priori reason to believe a full regular season isn't a small sample size.) The numbers marking each point are the cumulative minutes played by LeBron James to that point in each season.

Alright boys let's take some pictures.



october, -76 on/off having motherf***er



Definitely better than October but still trash (note that we are pretty much at the end of November now IRL)



Now we're getting somewhere. By the end of December we've got some decent correlation and our root mean squared error is below double digits. It's still seven so any December reading of on/off should be taken as value ± 14, and even then the 2014 year end value +7 only barely falls into the confidence interval of -6 ± 14. Let's drive on.



Mixed bag here. Our R^2 is down but so is our RMSE. At least we don't have any negative on/offs as of month-end any more, but consider this: our confidence interval of 7.4 means that even if Marcus Smart continues to be +5.1 on offense through January we couldn't be confident for just that season he would improve Boston's offense to a statistically significant degree. (To convert overall uncertainty to just one side of the ball, divide by the square root of two.)



February is named after the ancient Roman feast of purification, and boy did we need it. New max for R^2, our RMSE is down to 3, we're cooking with gas now.



And it's no surprise the picture is even better in March, but even at the end of March with 93% of the season's minutes played we can still be 2 off in either direction. Also worth pointing out is that while 2014 has been the furthest off throughout the year, by March it was closer than average and fully three years were further off than it.

.

Summary:

95% CI Just ORtg Month End
89 63 October
23 16 November
14 10 December
7 5 January
6 4 February
2 2 March
A season is an important unit. There is one MVP, one champion, one Rising Stars Skills Challenge winner per season, no more, no less.

But that doesn't mean a season is a statistically significant unit.

And this evidently applies just as well to a bloc of time that seems like it ought to be.

Be careful out there, folks.

[TimPig]

I don't get it.

Can you just tell me if Marcus Smart sucks?

[eric]

Dec 7, 2017 12:57:37 GMT -6 TimPig said:

I don't get it.

Can you just tell me if Marcus Smart sucks?

so imagine you're rolling dice

what color are they

YOU DIDN'T IMAGINE IT AT ALL

.

so imagine you're rolling dice

and you want to know if they're fair, which is to say there's an even chance of any side coming up

you measure 6 times, but you don't get exactly 1 per side - that's probably not evidence the dice are fixed
you measure another 6 times, and you're not exactly at 2 per side - same
and so on, and so on

at what point is it probably evidence that the dice are fixed? how many times do you have to roll?

one way of figuring it out is to look at records of that process with multiple sets of dice you know aren't fixed, and see what they looked like when rolled 6 times, 12 times, 216 times, etc. if the dice you're checking look like those, they're probably not fixed either. (for example, having 4 extra threes out of 216 rolls happens with non fixed dice, so if the investigated dice have 4 extra threes at the same point that's not evidence they're fixed.) but if the dice you're checking have results way beyond what any of those dice had, they're probably fixed. (for example, having 40 extra threes is way beyond what any fair dice will do.)

in this case, we don't know exactly what the random noise looks like in midseason on/off measurements. so we look at them in multiple cases where we know what the final value is and see how far off they were at various points in the season, because we know that was just noise.

we don't know where Marcus Smart's ORtg on/off will end up this year.
we know his ORtg on/off has been negative the past two years.
and we know that at the same point of the year where we are now, the ORtg on/off has been as much as 16 off from where it ended up.
since his current ORtg on/off is only +5, there's no reason to believe he won't end up negative this year too, because 5 - 16 is well below zero.

you follow?

[TimPig]

Dec 7, 2017 13:36:25 GMT -6 eric said:

Dec 7, 2017 12:57:37 GMT -6 TimPig said:

I don't get it.

Can you just tell me if Marcus Smart sucks?

so imagine you're rolling dice

what color are they

YOU DIDN'T IMAGINE IT AT ALL

.

so imagine you're rolling dice

and you want to know if they're fair, which is to say there's an even chance of any side coming up

you measure 6 times, but you don't get exactly 1 per side - that's probably not evidence the dice are fixed
you measure another 6 times, and you're not exactly at 2 per side - same
and so on, and so on

at what point is it probably evidence that the dice are fixed? how many times do you have to roll?

one way of figuring it out is to look at records of that process with multiple sets of dice you know aren't fixed, and see what they looked like when rolled 6 times, 12 times, 216 times, etc. if the dice you're checking look like those, they're probably not fixed either. (for example, having 4 extra threes out of 216 rolls happens with non fixed dice, so if the investigated dice have 4 extra threes at the same point that's not evidence they're fixed.) but if the dice you're checking have results way beyond what any of those dice had, they're probably fixed. (for example, having 40 extra threes is way beyond what any fair dice will do.)

in this case, we don't know exactly what the random noise looks like in midseason on/off measurements. so we look at them in multiple cases where we know what the final value is and see how far off they were at various points in the season, because we know that was just noise.

we don't know where Marcus Smart's ORtg on/off will end up this year.
we know his ORtg on/off has been negative the past two years.
and we know that at the same point of the year where we are now, the ORtg on/off has been as much as 16 off from where it ended up.
since his current ORtg on/off is only +5, there's no reason to believe he won't end up negative this year too, because 5 - 16 is well below zero.

you follow?

Yes thank you <3