Robots are very important nowadays to handle bots from crawling your website or pages, Hence it is important to add robots.txt file in your website’s root directory.
One of the main reason of using robots.txt is hiding your private pages from search results like for .e.g. your-website.com/wp-admin
Generally, If you don’t use robots.txt file, bots (spiders) will crawl your entire webpages, some are not necessary or it can also be duplicate contents.
I would better suggest you to use this file to guide bots to crawl your websites/blogs properly.
There is no need to create robots.txt nowadays because while installing WordPress, It automatically creates a virtual robots file with default coding. (Example given below)
How should I verify If I have default/virtual robots.txt file ?
Just you need to verify is by typing this url in your browser:
http://your-website.com/robots.txt
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Still there are many threads on WordPress support forums related to editing this virtual robots.txt, In my opinion editing this virtual robots file is difficult.
This is what I have faced with my blog; I tried every possible ways to solve but failed to edit this virtual file, Hence after doing a good research.
I found out a nice trick or you can call it plugin which worked for me like a charm.
Go to your Dashboard > Plugins > Add New
Type:Â WP Robots Txt
Or Use this link to download zip file directly: http://wordpress.org/extend/plugins/wp-robots-txt/
Install and activate it.
Go to Dashboard > Settings > Privacy
how to edit robots.txt in wordpress
Now you are ready to modify your virtual robots.txt with your desired syntax you wish to add on it.
Add your desired code in the box and update. Check your link (http://your-website.com/robots.txt), If it is updated or not.
You can also view my previous post on : Adding yoast breadcrumbs in every post
Please share If you like this article
How to edit robots.txt in wordpress
No comments:
Post a Comment