Control bots, spiders and crawlers

Bots, spiders, and other crawlers browsing your pages can cause extensive resource (memory and CPU) usage on server. This can lead to high load on the server and slow down your site overall performance..

One option to reduce server load from bots, spiders, and other crawlers is to create a "robots.txt" file at the root of your website. This guide search engines what content on your site they should and should not index.

To block this Googlebot, use the following in your robots.txt file:

# go away Googlebot
User-agent: Googlebot
Disallow: /

User-agent: AhrefsBot
User-agent: Baiduspider
User-agent: EasouSpider
User-agent: Ezooms
User-agent: YandexBot
User-agent: VelenPublicWebCrawler
User-agent: MJ12bot
User-agent: SiteSucker
User-agent: HTTrack
User-agent: SeznamBot
User-agent: serpstatbot
User-agent: CloudFlare-AlwaysOnline
User-agent: special_archiver
User-agent: Nimbostratus-Bot
User-agent: SemrushBot
User-agent: ZoominfoBot
User-agent: Jooblebot
User-agent: NetcraftSurveyAgent
User-agent: MegaIndex
User-agent: NetcraftSurveyAgent
User-agent: DotBot
User-agent: abot
User-agent: dbot
User-agent: ebot
User-agent: hbot
User-agent: kbot
User-agent: lbot
User-agent: mbot
User-agent: nbot
User-agent: obot
User-agent: pbot
User-agent: rbot
User-agent: sbot
User-agent: tbot
User-agent: vbot
User-agent: ybot
User-agent: zbot
User-agent: bot.
User-agent: bot/
User-agent: _bot
User-agent: .bot
User-agent: /bot
User-agent: -bot
User-agent: :bot
User-agent: (bot
User-agent: crawl
User-agent: slurp
User-agent: spider
User-agent: seek
User-agent: accoona
User-agent: acoon
User-agent: adressendeutschland
User-agent: ah-ha.com
User-agent: ahoy
User-agent: altavista
User-agent: ananzi
User-agent: anthill
User-agent: appie
User-agent: arachnophilia
User-agent: arale
User-agent: araneo
User-agent: aranha
User-agent: architext
User-agent: aretha
User-agent: arks
User-agent: asterias
User-agent: atlocal
User-agent: atn
User-agent: atomz
User-agent: augurfind
User-agent: backrub
User-agent: bannana_bot
User-agent: baypup
User-agent: bdfetch
User-agent: big brother
User-agent: biglotron
User-agent: bjaaland
User-agent: blackwidow
User-agent: blaiz
User-agent: blog
User-agent: blo.
User-agent: bloodhound
User-agent: boitho
User-agent: booch
User-agent: bradley
User-agent: butterfly
User-agent: calif
User-agent: cassandra
User-agent: ccubee
User-agent: cfetch
User-agent: charlotte
User-agent: churl
User-agent: cienciaficcion
User-agent: cmc
User-agent: collective
User-agent: comagent
User-agent: combine
User-agent: computingsite
User-agent: csci
User-agent: curl
User-agent: cusco
User-agent: daumoa
User-agent: deepindex
User-agent: delorie
User-agent: depspid
User-agent: deweb
User-agent: die blinde kuh
User-agent: digger
User-agent: ditto
User-agent: dmoz
User-agent: docomo
User-agent: download express
User-agent: dtaagent
User-agent: dwcp
User-agent: ebiness
User-agent: ebingbong
User-agent: e-collector
User-agent: ejupiter
User-agent: emacs-w3 search engine
User-agent: esther
User-agent: evliya celebi
User-agent: ezresult
User-agent: falcon
User-agent: felix ide
User-agent: ferret
User-agent: fetchrover
User-agent: fido
User-agent: findlinks
User-agent: fireball
User-agent: fish search
User-agent: fouineur
User-agent: funnelweb
User-agent: gazz
User-agent: gcreep
User-agent: genieknows
User-agent: getterroboplus
User-agent: geturl
User-agent: glx
User-agent: goforit
User-agent: golem
User-agent: grabber
User-agent: grapnel
User-agent: gralon
User-agent: griffon
User-agent: gromit
User-agent: grub
User-agent: gulliver
User-agent: hamahakki
User-agent: harvest
User-agent: havindex
User-agent: helix
User-agent: heritrix
User-agent: hku www octopus
User-agent: homerweb
User-agent: htdig
User-agent: html index
User-agent: html_analyzer
User-agent: htmlgobble
User-agent: hubater
User-agent: hyper-decontextualizer
User-agent: ia_archiver
User-agent: ibm_planetwide
User-agent: ichiro
User-agent: iconsurf
User-agent: iltrovatore
User-agent: image.kapsi.net
User-agent: imagelock
User-agent: incywincy
User-agent: indexer
User-agent: infobee
User-agent: informant
User-agent: ingrid
User-agent: inktomisearch.com
User-agent: inspector web
User-agent: intelliagent
User-agent: internet shinchakubin
User-agent: ip3000
User-agent: iron33
User-agent: israeli-search
User-agent: ivia
User-agent: jack
User-agent: jakarta
User-agent: javabee
User-agent: jetbot
User-agent: jumpstation
User-agent: katipo
User-agent: kdd-explorer
User-agent: kilroy
User-agent: knowledge
User-agent: kototoi
User-agent: kretrieve
User-agent: labelgrabber
User-agent: lachesis
User-agent: larbin
User-agent: legs
User-agent: libwww
User-agent: linkalarm
User-agent: link validator
User-agent: linkscan
User-agent: lockon
User-agent: lwp
User-agent: lycos
User-agent: magpie
User-agent: mantraagent
User-agent: mapoftheinternet
User-agent: marvin/
User-agent: mattie
User-agent: mediafox
User-agent: mediapartners
User-agent: mercator
User-agent: merzscope
User-agent: microsoft url control
User-agent: minirank
User-agent: miva
User-agent: mj12
User-agent: mnogosearch
User-agent: moget
User-agent: monster
User-agent: moose
User-agent: motor
User-agent: multitext
User-agent: muncher
User-agent: muscatferret
User-agent: mwd.search
User-agent: myweb
User-agent: najdi
User-agent: nameprotect
User-agent: nationaldirectory
User-agent: nazilla
User-agent: ncsa beta
User-agent: nec-meshexplorer
User-agent: nederland.zoek
User-agent: netcarta webmap engine
User-agent: netmechanic
User-agent: netresearchserver
User-agent: netscoop
User-agent: newscan-online
User-agent: nhse
User-agent: nokia6682/
User-agent: nomad
User-agent: noyona
User-agent: nutch
User-agent: nzexplorer
User-agent: objectssearch
User-agent: occam
User-agent: omni
User-agent: open text
User-agent: openfind
User-agent: openintelligencedata
User-agent: orb search
User-agent: osis-project
User-agent: pack rat
User-agent: pageboy
User-agent: pagebull
User-agent: page_verifier
User-agent: panscient
User-agent: parasite
User-agent: partnersite
User-agent: patric
User-agent: pear.
User-agent: pegasus
User-agent: peregrinator
User-agent: pgp key agent
User-agent: phantom
User-agent: phpdig
User-agent: picosearch
User-agent: piltdownman
User-agent: pimptrain
User-agent: pinpoint
User-agent: pioneer
User-agent: piranha
User-agent: plumtreewebaccessor
User-agent: pogodak
User-agent: poirot
User-agent: pompos
User-agent: poppelsdorf
User-agent: poppi
User-agent: popular iconoclast
User-agent: psycheclone
User-agent: publisher
User-agent: python
User-agent: rambler
User-agent: raven search
User-agent: roach
User-agent: road runner
User-agent: roadhouse
User-agent: robbie
User-agent: robofox
User-agent: robozilla
User-agent: rules
User-agent: salty
User-agent: sbider
User-agent: scooter
User-agent: scoutjet
User-agent: scrubby
User-agent: search.
User-agent: searchprocess
User-agent: semanticdiscovery
User-agent: senrigan
User-agent: sg-scout
User-agent: shai'hulud
User-agent: shark
User-agent: shopwiki
User-agent: sidewinder
User-agent: sift
User-agent: silk
User-agent: simmany
User-agent: site searcher
User-agent: site valet
User-agent: sitetech-rover
User-agent: skymob.com
User-agent: sleek
User-agent: smartwit
User-agent: sna-
User-agent: snappy
User-agent: snooper
User-agent: sohu
User-agent: speedfind
User-agent: sphere
User-agent: sphider
User-agent: spinner
User-agent: spyder
User-agent: steeler/
User-agent: suke
User-agent: suntek
User-agent: supersnooper
User-agent: surfnomore
User-agent: sven
User-agent: sygol
User-agent: szukacz
User-agent: tach black widow
User-agent: tarantula
User-agent: templeton
User-agent: /teoma
User-agent: t-h-u-n-d-e-r-s-t-o-n-e
User-agent: theophrastus
User-agent: titan
User-agent: titin
User-agent: tkwww
User-agent: toutatis
User-agent: t-rex
User-agent: tutorgig
User-agent: twiceler
User-agent: twisted
User-agent: ucsd
User-agent: udmsearch
User-agent: url check
User-agent: updated
User-agent: Uptimebot
User-agent: vagabondo
User-agent: valkyrie
User-agent: verticrawl
User-agent: victoria
User-agent: vision-search
User-agent: volcano
User-agent: voyager/
User-agent: voyager-hc
User-agent: w3c_validator
User-agent: w3m2
User-agent: w3mir
User-agent: walker
User-agent: wallpaper
User-agent: wanderer
User-agent: wauuu
User-agent: wavefire
User-agent: web core
User-agent: web hopper
User-agent: web wombat
User-agent: webbandit
User-agent: webcatcher
User-agent: webcopy
User-agent: webfoot
User-agent: weblayers
User-agent: weblinker
User-agent: weblog monitor
User-agent: webmirror
User-agent: webmonkey
User-agent: webquest
User-agent: webreaper
User-agent: websitepulse
User-agent: websnarf
User-agent: webstolperer
User-agent: webvac
User-agent: webwalk
User-agent: webwatch
User-agent: webwombat
User-agent: webzinger
User-agent: whizbang
User-agent: whowhere
User-agent: wild ferret
User-agent: worldlight
User-agent: wwwc
User-agent: wwwster
User-agent: xenu
User-agent: xget
User-agent: xift
User-agent: xirq
User-agent: yandex
User-agent: yanga
User-agent: yeti
User-agent: yodao
User-agent: zao
User-agent: zippp
User-agent: zyborg

Disallow: /