Good bots identify themselves, respect robots.txt, and crawl at a steady rate. Bad bots spoof user agents, ignore robots.txt, and hit login and checkout endpoints in bursts. The tell is behavior, not the user agent string, because any bot can claim to be Chrome.
The traffic you actually want
Search engine crawlers are the good bots every site needs. Googlebot and Bingbot fetch pages, follow links, and index content so the site shows up in search results. Monitoring bots check uptime, SEO tools audit pages, and social platforms fetch link previews. These bots identify themselves honestly, publish their IP ranges, and respect robots.txt. Block them and the site disappears from search within weeks.
The new arrivals are AI crawlers, which copy content for training and for AI search answers. They are a third category: not search crawlers and not attackers, but also not obviously useful to the site owner. Cloudflare caught one major AI crawler ignoring robots.txt in 2025 and removed it from its verified bot list. The right move is a deliberate policy per crawler, allow, throttle, or block, written down and enforced, not a blanket rule that treats all crawlers the same.
Related reading on this site: Home and Queue Dodging Is a Matchmaking Problem, Not a Player Problem cover the neighbouring problems in more depth.
The traffic that costs you money
Bad bots come in four flavors that matter to a game or store operator. Credential stuffing bots test stolen username and password pairs against login forms. Scrapers steal prices, content, and inventory data. Spam bots flood forms. Click fraud bots burn ad budgets on fake clicks. Thales counted bots at 53 percent of all web traffic in 2025, with bad bots alone at 40 percent. That is not background noise. It is nearly half the internet.
The expensive mistake is judging bots by their user agent. Thales found 41 percent of bot attacks use Chrome to look legitimate, and any script can claim any user agent string it likes. What a bot cannot fake as easily is behavior: the IP address, the request pattern, the timing between requests, and whether it ever loads images or runs JavaScript. A visitor that posts to the login endpoint fifty times in a minute without ever loading a stylesheet is not a player with a fast connection.
How to sort them without breaking the site
Start with robots.txt, which good bots obey and bad bots ignore. That alone splits the traffic into two piles: the bots that follow the rules and the bots that just announced they will not. Then verify the good ones. Real search crawlers can be confirmed by reverse DNS against their published ranges, and Cloudflare maintains a verified bot list that does the check for you.
For the rest, score behavior. Cloudflare scores every request from 1 to 99, with 1 to 29 likely automated and 66 to 99 likely human. Challenge the middle band with a CAPTCHA or a proof-of-work test, block the bottom band from sensitive endpoints, and leave the top band alone. The goal is not zero bots. It is a site where the bots that remain are the ones you invited.
Frequently asked questions
How much web traffic is bots?
Thales counted bots at 53 percent of all web traffic in 2025, with bad bots alone at 40 percent. More than a third of the traffic Cloudflare sees is bad bots.
Can I just block all bots?
No. Blocking search crawlers removes the site from search results, and blocking monitoring bots blinds you to outages. Allow verified good bots, and block by behavior instead of by user agent.
How do I verify a bot is really Googlebot?
Check reverse DNS against Google's published IP ranges. A bot that claims to be Googlebot from an address Google does not own is spoofed. Cloudflare's verified bot list automates this check.
Do AI crawlers have to obey robots.txt?
They are supposed to, but enforcement is uneven. Cloudflare documented a major AI crawler ignoring robots.txt in 2025. Set an explicit policy per AI crawler and enforce it at the firewall.