It is that time of the year, when the Premier League season is approaching its halfway point, and fans have seen enough of their clubs over the course of the season so far to have very pointed opinions about who needs to stay or go.
This essay spawned from one of said discussions – but the topic was not one of the eleven on the field. No, this time we were here to discuss the fate of the man on the sidelines, none other than Ruben Amorim, the latest in a long line of United managers that have failed to impress.
I actually spent a lot of time that day defending Amorim – or, to be more specific, pointing out that he was not the problem. For one, the case presented to me that day was nothing I had not heard before: stale tactics, player misprofiling, benching a fan favorite, and the likes. I recognized it because it is the same argument used against every manager in peril of losing his job, a way to say that the team’s bad form is the manager’s fault. The problem with that though, is that if the manager survives long enough to steer the ship around, the same sackable offenses get hand-waved away as the fans begin to dream of European nights. The issue was never really the tactics, but the losses, and reversing those is usually enough to right any wrongs for the foreseeable future.
I also believe the theory that managers have relatively little impact on club performance generally, and sacking them generally does not provide much impact on winning. I got this idea from reading Soccernomics, where they go through a plethora of arguments as to why this is the case, ranging from the manager market being inefficient, to performing statistical analyses showing that only 10% of managers regularly outperform the expectations of their team. He even uses this theory to explain the continued lack of black managers at the top level, though going into detail on that could be its own article. What I can say is that following from this theory, sacking a manager is a waste of resources that could be better spent improving the squad (which I think needs work anyway).
Suffice to say, I didn’t really give the thought of sacking Amorim much credence. He was just another manager drinking from the poisoned chalice.
However, after thinking about the question a bit more, I realized two things: one being that I found my own answer unsatisfactory (I make many general arguments, but none apply to Amorim specifically), and two being that this was a very tractable problem with the potential to bring a lot of insights, not on Amorim but on the impact of club managers in general.
That brings us to the objective of this essay, in which I intend to empirically measure the influence Premier League managers have on their clubs, and use this information to better understand what Amorim’s tenure so far can tell us about his future.
To do this, however, we need to start with some definitions.
Defining Impact
What is a manager’s job?
If I asked you that, we would probably end up with a list of things that a coach is expected to perform in his day-to-day. A manager should lead the training and hiring of coaching staff, provide insights on recruitment, handle the media, and maybe most importantly, put out the best squad possible on a weekly basis.
I’d agree with you. And to put it succinctly, I’d say that a manager’s job is to make his team better – through all the avenues available to him.
How can we measure a manager’s impact?
That might take you a bit longer to think about, but the answer needs to be just as clear to have any hope of answering our research question. In my workplace, a manager is evaluated through a yearly review process, where they present all the actions they carried out throughout the year to support my team to a jury of senior staff, who deliberate on their yearly rating.
This method is intuitive: it takes into account everything the manager did, and provides a human aspect to what sounds like a robotic task. But it only works if you know what the manager does behind the scenes – without it, we need something else.
Most soccer fans resort to evaluating a manager through the manifestations of his team’s performances – in other words, results. We may not know exactly how the manager goes about his team talks, but if the club is sat in 10th by Christmas, then he probably is no Erwin from AOT.
This is a step in the right direction, but it still has some issues. The first is that short-term results are a very noisy sample. Things like strength of schedule, injuries, and plain variance can all make the league table look very deceptive within the first 20 games. And beyond that, it still takes about a full season’s worth of games to know a team’s true attacking and defensive strength, but head coaches enter the hot seat way before then. We need a better measure of team performance if we want to perform any empirical study.
Another concern is one of expectations. Going back to the previous example, 10th for Chelsea might be a sackable offense, but 10th for Burnley would be a dream. Expectations are relative – not just to the club, but to each individual fan, and deciding which expectations are “reasonable” is an arduous task.
I’m going to try and sidestep these problems.
For the latter, we can return to our definition at the beginning of this chapter: What is a manager’s job? To make his team better. From the moment we get a new man in the seat, the same rules apply: a good manager is one that can leave his new squad in better shape than he found it. That means that we can use the strength of the team that a new coach inherits as his new baseline expectation, however we measure team strength.
As for the problem of measuring team performance, I am lucky to live in a world where that is already a heavily researched question, as there exists plenty of methods that I could apply for this use case. I ended up choosing a Dixon-Coles model to represent team strength, mostly because the theory is sound, the results are intuitive, and I had already been using it in previous projects.
What is a Dixon-Coles model?
For those who are not aware, this is a statistical model that tries to assign attack and defence ratings to teams based on the number of goals they score and concede in their previous games, respectively, while also weighting recent games more than older games. We can use these ratings to predict the expected result, as well as the probability of less likely scores, between two teams. For a more detailed explanation, you can head here.
Now that the question has been defined, we can finally begin our analysis.
The Methodology
So now we have a model, and we have a reasonable way to measure expectations, but we are still a far cry from measuring manager impact. What are the next steps?
First, I gathered a dataset containing all Premier League and Championship results from the 2021/22 season from FBRef using the soccerdata package. This is required for the Dixon-Coles model to generate team ratings.
I included Championship data so that any promoted team still has a history of match information. This method ensures that no special work needs to be done for promoted teams to have reasonable team strength estimates.
Then, I had to generate team strength estimates for Premier League teams as they changed over time from the 2021/22 season until now. I did this by fitting our data to the Dixon-Coles model in sequential batches, and cumulatively writing every team’s attack and defence coefficients to one dataset.
To go into more detail, I sorted every game in the dataset by their date. Then, starting from all matches on the first match day, I fit the model to the data and got team ratings for every team, which I stored. Next, I fit the model to the first and second match day, then the first 3 match days, etc. and stored each iteration in a separate data structure until the current day.
For those familiar with the Dixon-Coles model, you know that it does not actually spit out reasonably-scaled values, like “expected goals for/against an average team”. Instead, it gives you coefficients that can be plugged into a separate formula that then gives you an expected goals value for that particular fixture. So, in order to make the numbers more interpretable, for every weekly iteration, I scaled the attack and defence coefficients by simulating a full Premier League season given those coefficients and finding every team’s average goals scored and conceded in that simulation. From this point, I could calculate a team’s expected goal difference, and this is the primary measure of team strength that I will be using for the rest of the article.
Now that we have every team’s rating history from the 21/22 season until now, the final step involves associating each team with their manager at the time. I did this by scraping Wikipedia for a list of Premier League managers and the time they spent at PL clubs, and performing a simple join with our team rating history dataset.
This is the final dataset that we can query to get all the answers we need about manager impact, but I made some additional changes to make analysis easier.
Firstly, I added an expectation column. This column contains the expected goal difference of the team in the match before the new manager took over, and acts as the baseline expectation for the new coach. There is a debate to be had on whether this number is even a good measure of a club’s expectations, but as we will see soon, it is a decent empirical approximation.
Also notice that the data in our table actually starts from the 2022/23 season. I decided not to include the 21/22 season in our final dataframe for two reasons:
- I’ve seen evidence that it takes about a season’s worth of data to be reasonably confident about a team’s strength, and so it made more sense to fit our model with at least a season’s worth of results.
- In our dataset, there was no way for our model to compare Championship and Premier League teams until the 22/23 season, when some teams got promoted and relegated. This meant that the Championship team ratings did not mean anything in relation to the ratings of Premier League teams, but more importantly it broke my code, so I started the first iteration of data with the whole 21/22 season and the first week of 22/23 to avoid the problem.
I also filtered the dataset to only include matches from managers that started after the 1st of May, 2022. For managers that started before that date (like Pep and Arteta), I did not have the data to compute their team’s strengths before their start date, which meant that I did not have a baseline expectation to compare them to. I also excluded managers with less than 31 days spent at a club, as such managers did not have much chance to make any impact.
With that in mind, we now have everything we need to answer our questions on the impact of managers on team performance.
Interpreting the Results
Let us start with a number: 0.48. This is the average increase in xGD associated with Unai Emery’s tenure at Aston Villa. In other words, since Emery joined Villa, their xGD per game has increased by 0.48 on average.
That is a massive jump: it took Aston Villa from a lower mid-table side in 2022/23 to a “best of the rest” team in 25/26 alongside Newcastle, and unsurprisingly it is also the largest average increase in xGD associated with a manager’s tenure in the dataset, beating out Iraola at Bournemouth and ten Hag’s spell at United.
However, as we can see in this histogram, managers are rarely that impactful, with most clustered around having no impact on xGD. About 40% of the 45 managers in the dataset actually have negative impacts: meaning that the team became worse under them than the previous manager. Notable tenures on this end of the spectrum include Potter at Chelsea, Kompany at Burnley, and van Nistelrooy at Leicester.
So far, evidence supports the theory that managers usually have no impact on team performance. But what about long-term managers? The median duration of a head coach term is just under a year, and it is reasonable to expect that managers that keep their job for longer than that actually make the team better. Conversely, managers that underperform expectations usually get culled quickly to make space for a new manager that can potentially save the season.
I say a year in the histogram, but the true cutoff point is 325 days, the median duration of a coach tenure in the PL during this time period.
The data confirms our suspicions here: a disproportionate amount of underperforming managers do not last the year, and the managers that do are associated with a slight uplift in team performance, with a median increase of 0.1 xGD per game relative to expectation. However, even then this increase is not particularly noticeable in terms of improved league finishes, and 20% of long-lasting managers are still considered underperformers by this metric.
Let’s answer one more question: what does a successful manager’s tenure look like? Do they hit the ground running, or do they require time and patience? Is the improvement linear, or are there bumps along the way? There are probably statistical ways to answer that question, but I’d prefer to keep things visual with another graphic.
These are all 12 successful manager tenures in our dataset (where successful means lasting more than 325 days and having a mean impact > 0.1 xGD per game), with the trendlines showing the club’s rating history while the coach was with the squad.
At first glance, it seems like manager performance is unpredictable – very nonlinear with lots of hills and valleys. Notably, we also see some cases coaches who overperformed relative to the previous manager, but in doing so set a new standard that they could not maintain. I suspect this is what led to the early demise of managers like ten Hag and Edwards.
While there are not too many similarities, I think we can highlight the fact that most successful managers do not make the team worse in the short term – besides Iraola, every manager here was around or above expectations by their 8th game. While this is no causal analysis, it lends credence to the idea that good managers do not need too much time to “instill their values” and get the squad performing at the required level.
The Verdict for Amorim
I would be remiss if I did not finish this by revisiting my opinion on Amorim with all this new information. TL;DR: It’s not looking good.
Just over a year into Amorim’s reign at United, he has a manager rating of -0.11. That means that since Amorim joined United, their xGD has dropped by 0.11 per game relative to the baseline. That’s good enough for 16th percentile amongst all manager tenures, and 5th(!) percentile amongst managers that have lasted more than a year. For reference, only Vincent Kompany’s Burnley spell did worse by this metric.
That is pretty appalling. And it probably stings even more when put in a side-by-side comparison with former United manager ten Hag.
Ironically, it seems the case that even with injuries and poor recruitment, United were still better just before ten Hag’s sacking than they have been for most of Amorim’s stay.
Conclusion
Stepping back, there have been two headline findings in this analysis. One is that the data does suggest that managers have a limited impact on winning, and sacking them may not be the best use of resources. That remains true, even for Amorim: United’s squad has been hopping around mid table for a while now, and the boost from an average manager would not change that.
However, I do have to concede that in the case of Amorim in particular , my friends were in the right: everything points to him being part of the problem at United right now. I can understand why the United hierarchy would want to avoid pulling the trigger in the hopes that their man can rebuild a Champions League squad in his image, Arteta-style. The idea is solid; however, I’m not sure that I can still recommend Amorim as the man to execute anymore.