# Search engines are OK - they help people find stuff # The Internet Archive is Very OK - they preserve stuff # SEO bots are not OK - they provide no value to this site or its users. # They'll be blocked as they're encountered. # Block the useless SEO bots User-agent: BLEXBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: Seekport Crawler Disallow: / User-agent: barkrowler Disallow: / User-agent: AhrefsBot Disallow: / User-agent: dotbot Disallow: / User-agent: DataForSeoBot Disalow: / # A CGI script in here tends to cause problems for crawlers (ngm-list3). There # is a limit to the number of unique URLs it will generate but that limit is # quite high and bots tend to get trapped. User-agent: * Disallow: /~david/cgi-bin/ # This stuff isn't public and shouldn't be included in search engines # Most of this stuff is pretty ancient - the ~ftp proxy was turned off well # over a decade ago. I expect most of these endpoints are just 404s at this # point. User-agent: * Disallow: /restricted/ Disallow: /~ftp/ Disallow: /svn/ Disallow: /trac/ Disallow: /pmvadmin/ Disallow: /bucket/ Disallow: /lan/ Disallow: /old/ Disallow: /sites/ Disallow: /projects/ Disallow: /archives/comics.shtml Disallow: /archives/comics/ Disallow: /phpmyadmin/ Disallow: /cgi-bin/ Disallow: /~david/temp/ # Temporary - until we fix gallery2 Disallow: /gallery2/