The split is behavioural, not moral. A good bot is one whose work benefits the site it visits and which can be verified as what it claims to be. Search crawlers like Googlebot and Bingbot index pages so the site can be found. Link preview bots fetch a page when someone shares it. Monitoring bots check uptime. Cloudflare maintains a verified bot directory for exactly this class: bots whose operator, IP ranges, and reverse DNS can be checked, so a site can allow them by identity instead of by a user agent string anyone can copy. A bad bot fails one or both tests. It scrapes content against the site's terms, stuffs stolen credentials into login forms, posts spam, or hammers a release page. It also usually lies about what it is, rotating user agents and residential proxies to look like ordinary visitors. Any rule that treats the two classes the same will either block the crawlers a site needs or wave through the traffic it is trying to stop.
Verification is the line, not the user agent
A user agent is a claim, not a credential. Any scraper can send a Googlebot user agent string, and many do. Verified bot programs solve this by checking the request against published IP ranges and reverse DNS under the operator's domain, so the identity is confirmed by infrastructure the impostor does not control. That is why bot management fields distinguish a verified bot flag from a bot score: one is an identity check, the other is a behavioural likelihood, scored from 1 for likely automated to 99 for likely human in Cloudflare's system.
Scores handle the grey zone. A request that is not a verified bot and not obviously human gets challenged or rate limited rather than blocked outright, because the cost of a false block is a real customer, and the cost of a false pass on a login page is an account takeover attempt that one challenge would have stopped.
Related reading on this site: Home and Ready Checks Exist Because Lobbies Leak Players. Dodge Penalties Should Admit That. cover the neighbouring problems in more depth.
Good bots can still be the wrong bots for your site
Verified does not mean wanted. AI training crawlers are identifiable and increasingly listed in bot directories by category, search, aggregator, AI crawler, page preview, advertising, feed fetcher, and more, precisely so a site can allow search crawlers and refuse training crawlers in the same rule set. A publisher who blocks every automated request to stop training scrapers will also block the search indexing that sends them readers. Category level control is the fix: decide per job, not per technology.
Rate matters too. A legitimate price comparison crawler hitting a small store at full crawl speed is a load problem even though its purpose is legitimate. Good bot status earns a crawler a rate limit and a robots.txt conversation, not unlimited access.
What bad looks like in the logs
Bad bot traffic announces itself in patterns a single request hides. Login attempts in the thousands from rotating addresses, with stolen username and password pairs, are credential stuffing. Product pages fetched in catalogue order at machine speed are scraping. A release page whose sessions arrive in a single burst at the drop second, from accounts created that morning, is a scalping queue. None of these is visible in a page view count, which is why sites discover bad bots in their conversion rate first: traffic up, sales flat, support tickets about sold out inventory.
Start by splitting your own traffic report three ways: verified bots, scored likely bots, and the rest. Most sites have never looked at the middle band, and it is where the scraping and stuffing volume lives.
Frequently asked questions
How can you tell a real search crawler from a fake one?
Check the request against the operator's published IP ranges and reverse DNS. A genuine Googlebot request resolves back to a googlebot.com hostname from an address Google publishes. A scraper copying the user agent string cannot pass that infrastructure check.
Should a site block all bots to be safe?
No. Blocking all automation blocks search indexing, link previews, and monitoring along with the abuse, and the site loses the traffic the good bots deliver. The workable policy is allow verified bots by category, challenge or rate limit unknown automation, and block on behaviour.
What is a bot score?
A bot score is a behavioural estimate of how likely a request is automated. In Cloudflare's system it runs from 1, likely automated, to 99, likely human, and rules typically block or challenge low scores on sensitive paths while letting higher scores through.
Do good bots follow robots.txt?
Reputable crawlers do, and operators of verified bot programs commit to it as part of being listed. Robots.txt is a voluntary protocol though, so it filters the honest bots and does nothing against an operator who chooses to ignore it.