Central Winger: Can "luck" and skill be separated in soccer?

Lately, I've been captivated by a sports analytics concept called PDO. As James Grayson explains in his fantastic blog (where he explores PDO extensively), PDO is originally an ice hockey metric which takes its name from the person that developed it. In soccer, PDO is calculated by combining a team's shooting percentage and save percentage and multiplying by 1,000. Since league-wide shooting percentages and save percentages are reflexive, every league's average PDO must be 1,000.

Dutch analytics blog 11tegen11 does a fabulous job demonstrating that a team's current PDO is a very good proxy for luck. It is very common for soccer teams to have outstandingly good or poor PDO scores at the beginning of the season only to see these values regress toward the mean of 1000. PDO has proven to be a very good indication whether a team is currently under- or over-preforming and an academic example of how exceptional sports statistics tend to regress toward the mean.

However, no matter how drastically each team moves toward the 1000 mark, PDO is still a very good predictor of end of season results. In fact, for the last two years in MLS, PDO still accounts for 58 percent of the variance in a team's total league points at the end of the season.

Using every shot on target from the 2011 and 2012 MLS season, I constructed a model that predicts the likelihood of a goal (or a save) based on multiple shot attributes such as distance, angle and if the shot resulted from a header or penalty kick. Using this model, we can construct a similar metric called Expected PDO. Instead of using actual goals and saves, Expected PDO looks at the number of goals and saves a team would have normally been expected to accumulate based on the types of shots they experienced over the season.

From there, I hoped to compare the Expected PDO to the PDO and consider whatever the difference was between the measurements to be "luck." Since good teams are increasingly likely to create more (and higher quality) shooting situations than an average team, the Expected PDO should be a relatively good indicator of team skill (and it is). Therefore, it seemed safe to assume the difference between the actual and the predicted was a product of un-modeled bias.

But, I don't think that un-modeled bias is luck. Or, at least solely luck. In fact, this table looks a lot more like a measurement of team skill.

And this made me realize just how difficult it is to quantitatively separate skill from luck. In a sense, skill could merely be the ability to get lucky consistently – and we would never be able to tell the difference in this context. At what point does luck become skill? If you "over-preform" for many consecutive games, are you really over-performing? Has your skill level temporarily changed?

Or are good teams simply better at getting lucky?