ботов надо отстреливать ещё на подлете, то есть средствами nginx :

PHP код:

if ($http_user_agent ~ "Python-urllib|Spinn3r|SISTRIX|Morfeus|Toata|MJ12bot|QuerySeekerSpider|JikeSpider|proximic|Sosospider|Sogou web spider|aesop_com_spiderman|alexibot|backweb|batchftp|bigfoot|black.hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye|cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck|cosmos|crescent|custo|da|diibot|disco|dittospyder|dragonfly|drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon|emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky|frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet|grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot|infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar|joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb|likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch|mata.hari|memo|microsoft.url|midown.tool|miixpc|mirror|missigua|mister.pix|moget|mozilla.newt|nameprotect|navroad|backdoorbot|nearsite|net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot|attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer|offline.navigator|openfind|outfoxbot|pagegrabber|papa|pavuk|pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker|psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot|repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload|snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot|superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport|telesoft|the.intraformant|thenomad|tighttwatbot|titan|urldispatcher|turingos|turnitinbot|urly.warning|vacuum|vci|voideye|whacker|libwww-perl|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse|zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|sauger|site.quester|whack|craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|grabber|webpictures|Jakarta|User-Agent|libwww|lwp-trivial|PHPCrawl|WEP Search|Missigua Locator|ISC Systems iRc|Aboundex|360Spider|Java|Cogentbot|BunnySlippers|Cegbfeieh|AIBOT|Demon|Devil|Wonder|Foobot|Kenjin Spider|Density|LinkextractorPro|LWP::Simple|MassDownloader|Mass Downloader|NetZIP|NG|Metasearch|WebFetch|WebCopier|Webclipping|WebBandit|WebAuto|WebGo|Web.Image.Collector|WebLeacher|WebmasterWorldForumBot|WebReaper|WebSauger|eXtractor|Webster|WebStripper|WebWhacker|WebZIP|Catall Spider|AcoiRobot|rogerbot|mj12bot|Ahrefsbot|Zite|TweetmemeBot|woriobot|topsy|js-kit|NING") {return 400;}