Note: I wrote this a couple of months ago, and have done a lot of work surrounding this since. Will follow up with new architectures but wanted to finally get this out.
TL;DR: I built a neural n-gram model to learn from sequences of football events with the goal of next-event prediction. The model significantly outperforms the baseline for almost all event fields in teacher-forced and autoregressive prediction tasks, but struggles with longer-horizon event prediction. In this article, I discuss the intuition, architecture, results, and learnings over the course of this project.
Context
I’ve recently spent a lot of my free time studying Deep Learning, and I wanted to take a break from the lessons and use what I had learned to build a project. In a discussion with someone in the industry, I got the idea to build a small foundation model for football that is able to predict event sequences, akin to the current large language models that predict the next word. To get a sense of the idea, imagine how a game like Football Manager works under the hood: events are simulated based on player attributes, tactical setup, morale, and several other relevant data points. A foundation model would perform the same function, but with player, manager, and contextual representations learned from data, as opposed to being hand-tuned by experts. If a tool like this were accurate enough, you could use it for a variety of downstream tasks, for example:
- Counterfactual player simulation: how would a player perform at a new club?
- Tactical optimization: what is the impact of tactical instructions on the match?
- Match state analysis: given all thats happened in the game so far, what are all the possible outcomes?
This is particularly powerful in the game of football because it is an observational game. You cannot run controlled experiments where you, for example, play one of your best players in only half the games and bench him for the other half to see how he performs. Mostly because you would lose your job, but also because the game state changes so often, not just game to game, but minute to minute, that it's hard to generalize across all these various situations anyway. The idea is that with a model like this, you can do all the experimentation by configuring the model and simulating games. And the closer the model is to real life, the more confident you can be that findings discovered in the simulation can be applied to real life.
Data
To train the model used for this project, I used event data from the 2015/16 Premier League season, one of StatsBomb’s many open datasets. It contained data from all 380 Premier League games, totalling about 750,000 events. I then converted the events into the SPADL data format and derived the following dataset from it:
380 games · 740,510 events (median 1,938 per game) · split 304 / 38 / 38 games (train / val / test)
Token | Field | Description | Values |
|---|---|---|---|
h | is_home | Which team performed the action | home / away (+ boundary) |
e | event | SPADL action type | 21 actions + 3 boundary markers (most common: pass 43%, dribble 37%) |
x | start_x | Start location along the pitch, 1 token per metre | 0 – 105 m |
y | start_y | Start location across the pitch, 1 token per metre | 0 – 68 m |
t | time_elapsed | Seconds since the previous event | 0 – 105 s (median 1 s) |
d | duration | How long the action lasted | 0 – 68 s (median 1.2 s) |
a | result | Outcome of the action | 6 outcomes (success 83%, fail 17%) |
| match state | Period, minute, second, goal / yellow / red counts per team | context only, never predicted |
Then, to account for the fact that an event is actually a grouping of information, I defined an event as a 7-tuple of the home/away, event, x, y, time elapsed, duration, and result tokens.
To ensure that the model learns to predict all the tokens, each row was duplicated 7 times, where the target variable was one of the 7 tokens making up the event. In other words, the model predicts the same event 7 times, once for each token making up said event. In order to prevent leakage when performing this task, I mask unpredicted tokens of the event by replacing them with -1 as a sentinel value.
So, when the model is predicting h, all the columns corresponding to that event are passed in to the input layer as -1. When predicting e, h has been predicted already in the previous row so it becomes unmasked, while all other columns are masked, etc.
Finally, I added boundary rows to indicate the start, mid-point (half time) and end of a game. This gave me a standardized start/end token I could look for when trying to generate match sequences.
Architecture and Training
To generate event predictions, I built a multilayer perceptron (MLP) model that takes in the current masked event tuple plus the previous three event tuples and contextual game information at that point in the match. The input data was processed as follows:
The home, event, and result columns were treated as categorical and passed through their own distinct three-dimensional embedding spaces. The x, y, time elapsed, and duration columns were normalized and passed in as scalars.
These inputs were then concatenated and fed through three layers of 96 neurons with ReLU activations, before being routed to one of 7 output heads, one for each event field. This ended up totalling approximately 59 thousand trainable parameters.
To train the model, I set aside events from 304 randomly selected games as the training set. I used plain stochastic gradient descent for the optimizer over 1 million steps, and the batch loss summed the seven output heads’ cross-entropy losses.
Results
After training the model, I calculated certain result metrics and compared them to the baseline, which corresponds to either the mode or the median of certain columns. I present the results below.
Teacher-forced: the model is given the true values of the event's earlier fields.
field | metric | baseline | model |
|---|---|---|---|
is_home | accuracy | 51.6% | 94.9% |
event | accuracy | 43.3% | 75.3% |
start_x | MAE (m) | 20.80 | 6.48 |
start_y | MAE (m) | 17.57 | 6.87 |
time_elapsed | MAE (s) | 2.23 | 0.93 |
duration | MAE (s) | 0.90 | 0.75 |
result | accuracy | 82.4% | 88.1% |
Autoregressive: the model fills in the event one field at a time, feeding its own predictions forward.
field | metric | baseline | model |
|---|---|---|---|
is_home | accuracy | 51.6% | 94.9% |
event | accuracy | 43.3% | 73.4% |
start_x | MAE (m) | 20.80 | 6.73 |
start_y | MAE (m) | 17.57 | 7.41 |
time_elapsed | MAE (s) | 2.23 | 1.43 |
duration | MAE (s) | 0.90 | 0.89 |
result | accuracy | 82.4% | 82.1% |
I was quite pleased with the results: the model achieved better results than the baseline for all event tokens in the teacher-forced setting, and for all but one token in the autoregressive setting. Even though the baseline was simple, this was also achieved with a pretty small model and dataset - the paper I used as a reference point had up to 3 million parameters and several seasons’ worth of event data. The results showed that the neural network was capable of learning rules from the data that could be used in prediction contexts, even though there is still some ways to go in terms of the model being a faithful replication of reality.
As a demonstration of the network's capabilities, I had the model perform next-event prediction on the events leading up to Jamie Vardy's late goal in the Southampton vs Leicester game in the 15/16 season.
Anecdotally, the model is right in broad strokes (the predicted events do usually occur in the general area of the actual event over short distances, and it correctly predicted that Leicester would keep the ball through the whole sequence), but it struggles with predicting unlikely events, like long line-breaking passes or shots.
I also took my model and tried to apply it to long-horizon event generation tasks, but that was less successful. I first attempted to get the model to generate several full games, to check if the sampled frequencies matched the distribution in the data. Yet I did not get to complete this experiment, as about 50% of the games never terminated - the model failed to emit the sentinel value indicating full time. I dug deeper into the failure mode and learned that it is a well documented phenomenon known as exposure bias. While there are some mitigations I could have implemented, this was more of a project than a full product, and so I decided to keep that out of scope.
Learnings and Next Steps
There were a few interesting lessons I learned over the course of this project.
- The importance of experimenting with different architectures. While I started this as a project to build my intuition on building neural networks, I ended up going through several notably distinct architectures to see how they impacted model performance. Eventually, some of the decisions, like the final model having separate output heads per predicted token, turned out far better than I expected. An example of this came from using a disjoint output vocabulary. The earliest MLP architectures had one shared output head because my reference paper seemed to use it, but swapping to disjoint heads in the final experiment contributed to the single biggest improvement in teacher-forced next token prediction. While there were some hints in the literature that pointed me in that direction, I only would have seen the effects by testing it myself.
- The importance of properly evaluating a model for the task it is built to perform. Coming into this project, I was not familiar with exposure bias, and I expected a neural net optimized for next-event prediction would see similar performance in long-horizon match generation tasks. As a result, testing the model’s performance at these tasks was an afterthought, and I could have easily misrepresented the network’s abilities at the very job I built it for. After I saw the performance drop when it came to long-horizon match generation tasks, I looked at NLP videos/papers/textbooks to see if this was a novel issue. Luckily, it was not, and there are actually several ways to train and evaluate a model to do better at these open-ended generation tasks. I kept them out of scope for this project, but it was also good to run into the issue myself and learn a lot more about the intricacies of model building.
I am also aware that this is a very small model, and this whole project is more of a proof of concept + learning activity. I plan on looking into more developed architectures and scaling this type of model up significantly, and will continue to write up on my work as I go.
References
- Bengio, Y., Ducharme, R., Vincent, P. & Jauvin, C. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3, 1137–1155. The architecture: learned embeddings for the previous few tokens, fed through an MLP to predict the next one.
- Mendes-Neves, T., Meireles, L. & Mendes-Moreira, J. (2025). A Scalable Approach for Unified Large Events Models in Soccer. ECML PKDD 2025. Applying that idea to soccer event data: each event predicted one field at a time, with the event's later fields masked.