Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
That was my first thought, varnish (vinyl these days) in front of the website should dramatically reduce this as the git repo should be practically static for most of the content.
But there are "cubic bazillions" of possible URLs that are being requested. Even if they boil down to "only" some millions actual commits, their rendered HTMLs are all different.
Yes, this is very difficult to solve for sites with many URL variations, like git repos and heavily threaded forums. Your cache is always full, but the hit ratio is abysmal.
Or, html generation could be very, very fast in the first place. For example, serving static files is usually just as fast as a reverse-proxy cache.
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
The issue is there's far more pages than could reasonably be stored: cgit is an old-school server-side renderer and there's a combinatorial explosion of possible pages it could render. The article calculates there's about a quadrillion different pages that could be 'crawled'
i think its only 7gb of data total, therefore there should be a strategy to fully optimize for this deterministic scenario, but instead of caching instead with maybe compiled streaming renderer(s)... but he said its ok for now.. and anyway why should they need to fix anything, the slurp is what sucks. pun intended
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
[0] https://git.zx2c4.com/cgit/about/