This page provide information of Baiduspider Robot FAQ in English. Original FAQ is in Chinese Language (+http://www.baidu.com/search/spider.htm)
What is Baiduspider, Why it is visiting my Site?
Baiduspider is baidu.com's web crawling robot. It collect information around the internet to build a most updated index for search engine on baidu.com. This is an automatic procedure. Its role is to visit the html page on the Internet and create a index database so that users can find your site on Baidu Search.
Why baiduspider of non-stop crawling my site?
You have a new Web site or the continuous updating of the page, baiduspider will continue to crawl. In addition, you can visit the site inspection log baiduspider visit is normal in order to prevent malicious baiduspider pretending to frequent crawling your site. If you find abnormal baiduspider crawled your site, please feedback to webmaster@baidu.com, and please try to give a snapsot of Baidspider visit from your webserver log so that we can understand how Baiduspider visiting your site with abnormal behabier.
I Do Not Want My Site to Be Visited by Baiduspider, How Can I do It ?
Baiduspider read, understand and follow the Robots Exclusion Protocol. You can use the robots.txt file on your site to block the access of Baiduspider Robot/Spider on your website. You can either place a Total ban on your site or prohibit any part of your Web site from Baiduspider Spider.
Note: By prohibiting access of baiduspider your Web site will make your web site unavailable on Baidu search engine and as well as all the search engines who use our data to to provide web search services. To modify your robots.txt file to ban Biduspier from your site, robots.txt writing method
Baidu Spider's 'User-Agent' I Will Use In My robots.txt File?
"Baiduspider" the first capital letter B, for the rest of the lower case.
Why Baiduspider Visiting My Site When There is a robots.txt file in Place, Seems Your Baiduspider Don't Care About It All!
Baiduspider Do Follow the Robot Exclusion Protocol. If you have recently changed your robots.txt file but still seeing Baiduspider is visiting your site, the resaone may be an old outdated copy of robots.txt file in Baiduspider's Cache. Baiduspider is an Incremental Spider. It Checked The robots.txt file and restored it locally. It will not find your robots.txt changed until the next checking (two days later usually). So it like breaking the protocol if the robots.txt changed before the next checking. Baidu.Com is working on this issue to reduce the time of featching robots.txt file from any website.
I have updated my robots.txt file week ago but still Baidu.Com search result shwoing my site, Why ?
Because the search engine's index database update take time. Although Baiduspider has stopped visiting your web site, It may take 2 to 4 weeks to drop your site from our search engine database. Also Please also check your robots configuration is correct.
Baiduspider How Long Baiduspider Take to Re-crawl a Page on My Website?
Baidu search engine database updated weekly, depending on the importance of any web pages, Baiduspider have a different update rate, the frequency is from a day to week.
Baiduspider crawling caused by bandwidth congestion?
When Baiduspider normaly crawling your site, it will not cause bandwidth congestion. Baiduspider use 'if-Modified-Since HTTP Protocol Command' it will not fatch any page if it is not modifyed since it's last visit. We have receive some complain that few bad people using User-Agent 'Baiduspider' on malicious spider crawl. If you found Baiduspider known as the agent of crawling and cause bandwidth congestion, and as soon as possible, please contact us. You can feedback to webmaster@baidu.com, if you can provide the time to visit the Web site log will be more conducive to our analysis. We are also working for a site with the currect information of IP address, from where Baiduspider visit your website.
I Have Other Query Regarding your Baiduspider and It's Crawling, Where I Will Find Information?
You are alwayes welcom to contact us on 'webmaster@baidu.com' for your query. Other then that there is a lot of Internet Forum, where you can find a lot of Information people are talking about Baiduspider. Do a Google Search. You will find a lot of information on our Spider. Please don't forget to verify any information found on an unauthorative website.
Thank you for your interest on Baiduspider, Web Spider of Baidu.Com
Posts
0 comments:
Post a Comment