What are robots?
For those who aren't as tech savvy or on the top of their internet jargon game (as most of us aren’t), robots.txt is a text file that webmasters use to instruct robots. In this particular case the robots are search engine robots and they are being instructed on how to crawl and index pages on their site. There are two ways these robots are typically managed; robots.txt and meta robots. The meta tag instructs bots on which individual files they should not index, while the robots.txt can be used to restrict access to entire directories. So to put it in layman’s terms, if there are certain pages of your site that you prefer to not be shared in search engine results, then using either of these codes will make that happen for you, but there is definitely a method to make it easier and more secure.
How do Meta Robots and Robots.txt works?
In order for the robots.txt file to work it has to be placed in the in the top-level directory of a web server, so the text would read, www.nameofsite.com/robots.txt. The meta tag on the other hand can be applied to an URI using REP tags such as “noindex or nofollow.” You have to also keep in mind that robots.txt are not private. Anyone can see what you have blocked from the search engines if they expend the time and effort to look. This is problematic because the pages (that aren’t being crawled by any search engines) can still show up in search engine results and they can accumulate rankings. Of course you want to accumulate rankings for your site but you don’t want it to be for hidden pages; it would just be a waste.
Which is more secure Meta Robots or
Robots.txt?
If you are using robots.txt and you want your blocked information completely private, it is recommended that you find a more secure way to keep visitors from viewing the pages that you don’t want indexed, like instituting password protection. Also malicious crawlers will likely ignore a robots.txt code which makes it a less secure method as well. Lastly, file name for robots.txt is case sensitive so a simple misstep with the text will render the code useless.What should I use?
Now comes the question of which one is better to use. Well there are really no short and sweet answers but it boils down to deciding what best meets your needs.If you really want to block a page from showing up in search engine results then you may want to seriously consider the meta robots tag. It will tell the search engine they are welcome to visit the page but they are not allowed to display your URL in their results. It also offers you more privacy because the pages you wish to be hidden can’t be found as is the case with robots.txt. It is a much more direct method of executing this task and most people find it superior to robots.txt.
I found this is an informative and interesting post and i think its useful and knowledgeable too..Thanks..-atlanta it support
ReplyDelete