Skip to content

Instantly share code, notes, and snippets.

@Burshyn
Created April 15, 2022 06:48
Show Gist options
  • Save Burshyn/96d0d1dddee1d9ebf9963e7c67310d33 to your computer and use it in GitHub Desktop.
Save Burshyn/96d0d1dddee1d9ebf9963e7c67310d33 to your computer and use it in GitHub Desktop.
Code to close the site from unwanted bots.
# off spiders
RewriteCond %{HTTP_USER_AGENT} (Baiduspider|YandexBot|Aboundex|legs|360Spider|Java|Cogentbot|Alexibot|asterias|attach|BackDoorBot|BackWeb|Bandit|BatchFTP|Bigfoot|Black.Hole|BlackWidow|BlowFish|BotALot|Buddy|BuiltBotTough|Bullseye|BunnySlippers|Cegbfeieh|CheeseBot|CherryPicker|ChinaClaw|Collector|Copier|CopyRightCheck|cosmos|Crescent|Custo|AIBOT|DISCo|DIIbot|DittoSpyder|Demon|Devil|Wonder|dragonfly|Drip|eCatch|EasyDL|ebingbong|EirGrabber|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|Exabot|WebPictures|Extractor|EyeNetIE|Foobot|flunky|FrontPage|Go-Ahead-Got-It|gotit|GrabNet|Grafula|Harvest|hloader|HMView|HTTrack|humanlinks|IlseBot|Stripper|Sucker|Library|InfoNaviRobot|InfoTekies|Intelliseek|InterGET|Ninja|Iria|Jakarta|JennyBot|JetCar|JOC|JustView|Jyxobot|Kenjin.Spider|Keyword.Density|larbin|LexiBot|lftp|likse|LinkextractorPro|LNSpiderguy|LinkWalker|lwp-trivial|Simple|Magnet|Mag-Net|MarkWatch|Downloader|Mata.Hari|Memo|Microsoft.URL|Control|MIIxpc|Mirror|Locator|PiX|moget|NAMEPROTECT|Navroad|NearSite|NetAnts|Netcraft|NetMechanic|NetSpider|Vampire|NetZIP|NextGenSearchBot|NICErsPRO|niki-bot|NimbleCrawler|Ninja|NPbot|Octopus|Explorer|Navigator|Openfind|OutfoxBot|PageGrabber|Foto|pavuk|pcBrowser|Pockey|ProPowerBot|ProWebWalker|psbot|Pump|QueryN.Metasearch|RealDownload|Reaper|Recorder|ReGet|RepoMonkey|RMA|Siphon|SiteSnagger|SlySearch|SmartDownload|Snake|Snapbot|Snoopy|sogou|SpaceBison|SpankBot|spanner|Sqworm|Stripper|Sucker|SuperBot|SuperHTTP|Surfbot|suzuran|Szukacz|tAkeOut|Teleport|Telesoft|TurnitinBot|The.Intraformant|TheNomad|TightTwatBot|Titan|True_Robot|turingos|TurnitinBot|URLy.Warning|Vacuum|VCI|VoidEYE|Collector|Sucker|WebAuto|WebBandit|Webclipping.com|WebCopier|WebEMailExtrac|WebFetch|IS|Web.Image.Collector|WebLeacher|WebmasterWorldForumBot|WebReaper|WebSauger|eXtractor|Quester|Webster|WebStripper|WebWhacker|Whacker|Widow|WISENutbot|WWWOFFLE|WWW-Collector-E|Xaldon|Xenu|Zeus|ZmEu|Zyborg|sc_bot|Clarabot|DuckDuckGo-Favicons-Bot|DuckDuckBot-Https|oBot|TinEye-bot|TwitterBot|SemrushBot-SA|aiHitBot|SemrushBot-SI|SemrushBot-BA|Nimbostratus-Bot|coccocbot-web|SeznamBot|coccocbot-image|SemrushBot|YandexAccessibilityBot|Pinterestbot|DuckDuckBot-Https|MJ12bot|SeznamBot|LinkpadBot|DotBot|AhrefsBot|AhrefsSiteAudit|UptimeRobot|BLEXBot) [NC]
RewriteRule .* - [R=403,L]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment