Hướng dẫn chặn bot tìm kiếm không có ích bằng robots.txt

[AdminAdmin is verified member.]

Tạo file robots.txt với nội dung sau

#Baiduspider
User-agent: Baiduspider
Disallow: /
User-agent: Baiduspider+
Disallow: /
User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm)
Disallow: /
User-agent: Baiduspider/2.0; +http://www.baidu.com/search/spider.html
Disallow: /
User-agent: Baiduspider/2.0
Disallow: /
User-agent: +Baiduspider
Disallow: /
User-agent: +Baiduspider/2.0
Disallow: /
User-agent: +Baiduspider/2.0;++http://www.baidu.com/search/spider.html
Disallow: /
User-agent: Mozilla/5.0(compatible;Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Disallow: /
#Yandex
User-agent: spider
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: Yandex.com
Disallow: /
User-agent: YandexBot/3.0
Disallow: /
#Sosospider
User-agent: Sosospider+
Disallow: /
#Ezooms
User-agent: ezooms
Disallow: /
User-agent: Ezooms/1.0
Disallow: /
User-agent: Mozilla/5.0(compatible;Ezooms/1.0;ezooms.bot[at]gmail[dot]com)
Disallow: /
#Sogou
User-agent: sogou spider
Disallow: /

Chúc bạn thành công
Nguồn:
[Hướng dẫn] chặn bot tìm kiếm không có ích bằng robots.txt
http://tuoitreit.vn