Part Two: What If Money Doesn’t Matter?

Editor’s Note: This article is the second in a series exploring the effect of money in Major League Soccer.

Key information related to this piece can be found in the first article. Links to the other articles can be found at that bottom as they are released.

If American soccer were like English soccer, MLS would look drastically different.

There are certainly aspects of the game — we’re talking about soccer as a system, not as the literal sport here — that have transferred across the pond, some that are only present in name (like the USL’s re-brand to create three leagues with the same confusing names as England’s second-fourth divisions), and other uniquely American details that are baffling to foreign fans.

Pushing aside from (the lack of) promotion/relegation, ignoring college soccer and the draft system, and moving past allocation money, one not-so-obvious difference arises: how money affects performance.

Simon Kuper and Stefan Szymanski’s Soccernomics laid out the extent to which team payrolls are correlated with performance in the English system, finding the relationship to be very strong.

Looking into MLS is a little bit different, though. For one, there simply isn’t as much data. With only one division to look at, which only has easily-found figures dating back to 2007, we can’t enjoy the statistical accuracy that a larger population size gives. But with 218 different “campaigns” — that being one season for one club — to examine, we can get a reasonable idea.

Inside the Stats: In order to standardize results and make them easier to understand, we went about measuring both spending and performance by ranking all MLS teams from 1 (largest payroll or highest Supporters Shield finish) to bottom, which varied as the league expanded and added clubs. If the model performed statistically perfectly, the highest spender would also be the top finisher, and the numbers would match-up down the chart. You’ll see how that worked out soon...

Performance was easy to measure. The Supporters Shield trophy is awarded to the top-finishing MLS team each year and ignores the conference alignment, allowing us to see a simple ranking of teams.

Payroll measuring was a little trickier. The statistical norm would be to rank teams by their median salaries, but we chose to rank by total payroll for a math-y reason: With the median, each player carries the same “weight,” regardless of how much they’re being paid. With that model, a rookie making the league minimum carries the same value as a Designated Player. But if a team is paying a player more, that player should perform at a higher level, meriting that bigger paycheck. So if a club is going to spend a lot of money for one particularly good player (like the LA Galaxy with David Beckham), that player should have more of an effect on their performance, and should have the same for their payroll.

In short, spending doesn’t matter — at all. Not even a little bit.

Here’s a graph of spending rankings (on the horizontal axis) compared to performance rankings (on the vertical axis). If payroll really affected performance, you’d see a clear upward trend across the graph. Instead, there’s no noticeable trend at all.

A club that spent the most and finished first would sit at (1, 1) on the graph (this was done three times: Toronto FC in 2017, New York Red Bulls in 2013 and LA Galaxy in 2011), and the slope — if the relationship was perfect — would rise steadily from there, as show below:

As you can see, the points do not follow the blue line. The overall correlation from 2007-2018 yields an r-squared value of 0.039, showing no statistical significance.

Inside the Stats: R-squared is a useful tool in statistics for measuring the correlation, or strength of the relationship, between two variables. The number is always between 0 (absolutely no correlation) and 1 (perfect correlation), because it is the square of the “r” value, which falls between -1 (perfectly negative correlation) and 1 (perfectly positive correlation).

R-squared allows us to express the model’s relationship in terms of a percentage. If the value had been, for example, 0.54, we could assert that the model accounts for 54% of the variation in variable Y.

We cannot use r-squared to say we’re 3.9% accurate, or can predict the future with 3.9% accuracy — though we really wouldn’t want to brag about that anyway.

We can further see the lack of a relationship by looking at a different type of graph. Below, you can see what is called a box-and-whisker plot.

The top of each bar shows the worst finish by a club that fell at that payroll ranking and the bottom shows the best. The shaded box shows the spread of the middle 50% of clubs that fell at that payroll ranking. The bar within that box marks the median finish for that payroll rank. (Ignore what’s happening on the right side of the graph — this format doesn’t work well with the small number of data points)

Again, if there was a perfect relationship, we would see a clear upward trend. And again, there isn’t one. The third quartile (top of the shaded box) for the second payroll rank is lower than that of the first rank, and the median rank for the eighth highest-spending team is lower than ranks 3-7.

Apparently, money doesn’t matter.

There are some years, though, where payroll does seem to have an effect — maybe.

Believe it or not, this was as positive correlation gets in the years studied.

This graph is of the 2017 season, which yielded an r-squared value of 0.192. You can see the general upward trend, but there are also some data points that stick out. The LA Galaxy, who spent the 5th-most but finished 22nd, are noticeable, as are the Columbus Crew, whose payroll ranked 15th but performance was good enough for 5th.

Most years aren’t as clear as 2017, though. Take 2008, for example:

This mess of a year gave us the strongest r-squared of any given season (0.309). The catch? The correlation is negative, suggesting that the more a team spent on its players, the worse it performed. This is a good lesson in why it’s important to look at data in a broader sense, not just one weird, wacky season.

But what about MLS Cup? The knockout playoffs at the end of the season are generally considered to be the league’s championship, even though the smaller sample size is more likely to be influenced by a non-champion-determining factor like luck.

Still, money doesn’t matter all that much.

The playoffs don’t behave much differently from the regular season. In fact, the top spending teams have won more Supporters’ Shields (3) than MLS Cups (2).

There are certainly teams that are able to perform better than their payrolls suggest, and there certainly some who can only be called flops. We’ll analyze both of those categories in the next two articles of this series.

Stay tuned for further analysis in the coming parts of this series, all of which can be found in the storystream below.