> it's not actually obvious who's running these bots
This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!
Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option
[−]tmp10423288442 · 2026-08-30 Sun 16:26 UTC ·
link
ChatGPT uses a distinct bot type for ChatGPT-initiated web search versus model training:
GPTBot → model improvement / training crawl
OAI-SearchBot → search index / ChatGPT Search
ChatGPT-User → fetch initiated by a user's ChatGPT interaction
This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!
Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option
GPTBot → model improvement / training crawl
OAI-SearchBot → search index / ChatGPT Search
ChatGPT-User → fetch initiated by a user's ChatGPT interaction
OAI-AdsBot → advertising landing-page crawl