cgit/robots.txt, branch lf/filter🍴 My patches to cgit
https://git.ache.one/cgit/atom?h=lf%2Ffilter2013-08-12T19:14:10Zrobots.txt: disallow access to snapshots2013-08-12T19:14:10ZJason A. DonenfeldJason@zx2c4.com2013-05-28T12:17:00Zurn:sha1:23debef62104c70600be2b745ec3957538eeac6e
My dmesg is filled with the oom killer bringing down processes while the
Bingbot downloads every snapshot for every commit of the Linux kernel in
tar.xz format. Sure, I should be running with memory limits, and now I'm
using cgroups, but a more general solution is to prevent crawlers from
wasting resources like that in the first place.
Suggested-by: Natanael Copa <ncopa@alpinelinux.org>
Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>