User-agent: SemrushBot Disallow: / User-agent: DotBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: PetalBot Disallow: / User-agent: Googlebot-Image Disallow: / # # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / # # Doesn't follow robots.txt anyway, but... # User-agent: k2spider Disallow: / # A capture bot, downloads gazillions of pages with no public benefit # http://www.webreaper.net/ User-agent: WebReaper Disallow: / #----------------------------------------------------------# # All rules set here are active for every user-agent. User-agent: * Disallow: /cgi-bin/ Disallow: /scgi-bin/ Disallow: /entropybanner/ Disallow: /js/ Disallow: /x.htm Disallow: /x2.htm Disallow: /x3.htm Disallow: /x4.htm Disallow: /id68.htm Disallow: /id72.htm Disallow: /id83.htm Disallow: /csbdiscontinued.htm Disallow: /attachments/ Disallow: /avatars/ Disallow: /backgrounds/ Disallow: /bobshamo/ Disallow: /bobshamofile/ Disallow: /colortest/ Disallow: /download/ Disallow: /frameredirect/ Disallow: /fredhen/ Disallow: /fredhenley/ Disallow: /ftp/ Disallow: /house/ Disallow: /images/ Disallow: /kwikkerb/ Disallow: /marketing/ Disallow: /publish/thanks.htm Disallow: /publish/id86.htm Disallow: /publish/full.htm Disallow: /qgimages/ Disallow: /resolredirect/ Disallow: /resource/ Disallow: /Resource-107/ Disallow: /richardtest/ Disallow: /samples/ Disallow: /sanna/ Disallow: /searchtest/ Disallow: /Smiley/ Disallow: /sound/ Disallow: /sources/ Disallow: /temp/ Disallow: /Themes/ Disallow: /trash/ Disallow: /upload/ Disallow: /uploads/ Disallow: /vmm/ Disallow: /photo/id125.htm Disallow: /photo/id100.htm Disallow: /photo/id110.htm Disallow: /photo/id102.htm Disallow: /photo/id30.htm Disallow: /photo/id48.htm Disallow: /photo/id53.htm Disallow: /photo/id54.htm Disallow: /photo/id85.htm Disallow: /photo/if1.htm Disallow: /photo/if1_1.htm Disallow: /photo/if1_2.htm Disallow: /photo/if1_3.htm Disallow: /photo/if1_4.htm Disallow: /photo/if1_5.htm Disallow: /test-csb2nf/noframes.htm Disallow: /test-csb2nf/noframeanchor.htm Disallow: /test-csb2nf/thanks.htm Disallow: /test-csb2nf/id101.htm Disallow: /test-csb2nf/id103.htm Disallow: /test-csb2nf/id51.htm Disallow: /test-csb2nf/textstart.htm Disallow: /test-csb2nf/textj1.htm Disallow: /test-csb2nf/textj2.htm Disallow: /test-csb2nf/textf1.htm Disallow: /test-csb2nf/textf2.htm Disallow: /test-csb2nf/texts1.htm Disallow: /test-csb2nf/texts2.htm Disallow: /test-csb2nf/frame3link.htm Disallow: /test-csb2nf/frame4blank.htm Disallow: /test-csb2nf/frame4spaz.htm Disallow: /test-csb2nf/frame4si.htm Disallow: /test-csb2nf/frame4oscar.htm Disallow: /test-csb2nf/frame4silky.htm Disallow: /test-csb2nf/cb.htm Disallow: /test-csb2nf/77.htm Disallow: /test-csb2nf/67.htm Disallow: /test-csb2nf/58.htm Disallow: /test-csb2nf/62.htm Disallow: /test-csb2nf/73.htm Disallow: /test-csb2nf/60.htm Disallow: /test-csb2nf/72.htm Disallow: /test-csb2nf/external1.htm Disallow: /test-csb2nf/external2.htm Disallow: /test-csb2nf/external3.htm Disallow: /test-csb2nf/static2.htm Disallow: /test-csb2nf/special1.htm Disallow: /test-csb2nf/special2.htm Disallow: /test-csb2nf/bradburg.htm Disallow: /test-csb2nf/asx.htm Disallow: /test-csb2nf/Resource-978/