The kernel folks likely have a good profile on what page people trigger Anubis on (i.e. what page people hit first). From that they could make heuristics about what pages are likely to be useful deep links.
Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.
Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.