Building a Pokemon Showdown Bot Detection System

Motivation

When I was younger, I got really into Pokemon: I had a Switch to play the games, watched Wolfe's VGC content right as he was starting out, and played VGC on Showdown in high school (where it was explicitly banned). In college I played much less, but I still followed the big game announcements, Worlds, etc. while taking my Computer Science classes. Fun fact: my college capstone project was building Pokemon bots.

So naturally, seeing bots pop up in the Pokemon community, and the community's reaction to them, pulled me back in.

That reaction has been less than positive, for a variety of reasons. One of the biggest is the worry that bots will spread through the scene until there are no human vs. human games left - a sort of dead internet theory for Pokemon.

I think this is a valid concern, but I don't think it's an unavoidable outcome. I do believe that PS can provide an environment for both humans and agents, while also weeding out bad actors that try to break the site's norms by stealthily entering human-only ecosystems.

However, in discussions with community members, I found that a lot of people doubted how well bot accounts could be detected from human accounts, for a variety of reasons. So, this project is my attempt to build a simple bot detection model that can tell bots and humans apart, and to see for myself where it breaks down.

Data

Pokemon Showdown has a lot of public replay data thanks to replay.pokemonshowdown.com. I needed to collect 2 types of replays: replays involving bot accounts and replays involving human ones.

For the bot accounts, I only used bots that were either publicly announced by their creators (mikumiku, Jaxcalibur, Nessie and Foul Play) or officially registered as bots on Pokemon Showdown (ZeroBot, MysteriousWind, porygoned, botmonchan, emifan420, and bbiddy and ryanvcgtob): 10 bots in total across 12 accounts, and scraped their games.

For the human accounts, this was a bit tougher as people generally do not go announcing their human Pokemon Showdown accounts. So to be specific, this classifier predicts whether an account is a bot or a presumed human, where a presumed human is one that:

1. Was registered before Jan 1, 2026,
2. Does not have an allusion to bots in their name, and
3. Is not a known bot already.

I then got human replay data by either:

1. Randomly selecting some of the top 500 players in different formats, or
2. Randomly selecting the opponents of the bot accounts

After doing this, I ended up with ~15 thousand player replays - of which about 3 thousand were bot replays and the rest were presumed human replays.

After collecting this data, I went through each player replay and built game-level behavioral signals that I thought could be used to discriminate between bots and humans. I then aggregated these signals into an account-level profile and fed them into a logistic regression model that assigns a 'suspicion score', a value indicating whether this account has a lot of bot-like behaviors (values closer to 1) or human-like behaviors (scores closer to 0).

Evaluation

After fitting the model, I evaluated it by having it predict unseen accounts (both bot and human), and having it estimate their suspicion scores. If the model gave an account a suspicion score greater than 0.5, I considered it as the model flagging the account as a bot, and naturally a score of 0.5 or less meant the model considered the account human. I then took the model's predictions and calculated these metrics:

1. Recall: What proportion of bots are flagged? The higher this is, the better.
2. False-positive rate: What proportion of humans are flagged? The lower this is, the better.
3. Time until detection: How many games does it take for the median bot to get flagged? The lower this is, the better.

Naturally, 1 and 2 are the meat of any detection system, and we want to identify as many bots as possible while flagging as few humans as possible. 3 is also valuable to know as the faster bot accounts can be flagged and further investigated, the quicker bot accounts violating ToS can be removed from the player pool.

Results

1. Recall and False Positive Rate

By setting the threshold at 0.5, this model correctly identifies 83% of bot accounts without mistakenly flagging any humans. The threshold can also be adjusted depending on how many human accounts the mods are comfortable with flagging: for example, if we set the suspicion threshold at 0.2, we can flag 92% of the bots while flagging only 1% of humans.

Notably, most humans do not come close to being suspicious, and similarly most bots do not come close to not being suspicious - Foul Play and ZeroBot are the only bot families with a suspicion score lower than 0.5 (and ZeroBot had a lot of bot signals, but a small sample size did not allow my model confidently flag it as suspicious ). This suggests that there are clear differences between how the current generation of bots and humans behave.

2. Time until detection

Using this detection system, the median bot got flagged within six games, and every bot except Foul Play (which was never flagged) was first flagged by its 20th game at the latest. For most bots, this would be about an hour or two of play.

Again, Foul Play proves specifically hard to detect, but outside of that every other bot family would have been flagged at some point well within a typical day's worth of bot games.

Some Closing Thoughts

I want to be clear about this: this is not meant to be a full-fledged bot detection system - instead, it is more of a proof of concept. I had talked with a lot of people who doubted that bots could be detected in a game like Pokemon, and wanted to see it for myself.

Since this is more of a proof of concept, I can go into detail about what I knew already/did learn while working on this:

1. Building bot detection itself is not particularly difficult. Not significantly more difficult than building the bot itself, anyway. However, the difficulty of bot detection scales with the complexity with which bots try to evade detection - a bit of a catch-22. The bots I used for this project did not really do much in terms of masking (and even then Foul Play was particularly hard to detect for reasons I still do not know), but the intentions of their creators were never to hide them, and so it is possible that my model is biased, in the sense that it was not trained on detecting bots that are built to be stealthy.

• This is more of a fun detail that popped up during this, but another thing about the battle between bots and bot detection is that knowledge about one feeds into the other. For example, by publicizing how this system detects bots, someone else could build a bot specifically to fool my system, making it obsolete. More powerful bots can then serve as detectors for weaker bots, because the stronger bot can properly evaluate the actions taken by both the weaker bot and humans, and flag any disparities. Because of this, though originally I intended to either share the codebase or have the model hosted somewhere for public use, I decided against doing both, at least for now.

2. Even with good bot detection, some worries from the Pokemon community are not obviously resolved. I chatted with someone who was worried about the future of online tournaments if bots could not reliably be detected from a small sample of games. Now, while 6 games is fairly quick, it is admittedly not instantaneous, so this project did not do much to prove that one-game detection was possible.

• Though there are still protections that could be put in place: a better agent detection system could theoretically detect bots faster, and account checks could be implemented (say, accounts in tournaments need x amount of game history), and of course in chess you have some sort of human-behind-the-screen validation. But I'm not here to provide Smogon with policy suggestions (unless they ask me, would love to do that).

3. However, I do believe that this is a good demonstration of bot detection being a tractable problem. I was motivated to do this project seeing the negative reaction around bots, which surprised me because, at least in chess, bots/engines have a lot of positive impact. Through discussions, I did come to understand some worries (mostly around playing against bots against their will), and I hope that with projects like this combined with some of the work Smogon is doing already to provide controlled bot environments, the public's concerns can be eased, and the benefits of these tools can be reaped.