What is the robots.txt? |
| |
 |
Reputation Power: 1
| | New Poster Posts: 10 Topics: 2 |
|
 What is the robots.txt?
August 14, 2011, 12:28:19 AM
|
Robots.txt file is a file that is not really an HTML file, but it is placed on your site and instructs the robot's search engine page. In general, search engines follow the instructions to the file, but it is not mandatory, they do not obey instructions. You cannot avoid, the search engine robots will file on your website, you should never forget. On the other hand, this is a great opportunity to direct the robot in the right direction. You have more input?
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
| |
|
Reputation Power: 1
| | New Coder Posts: 33 Topics: 0  |
|
Re: What is the robots.txt?
August 14, 2011, 12:59:10 AM
|
It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
 |
Reputation Power: 1
| | New Coder Posts: 29 Topics: 1 |
|
Re: What is the robots.txt?
August 14, 2011, 11:45:30 PM
|
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots. txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
|
Reputation Power: 2
| | Regular Coder Posts: 93 Topics: 34 |
|
Re: What is the robots.txt?
August 25, 2011, 10:36:15 PM
|
Hi,
"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth.
Thanks...........
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
 |
A Web Developern from India
Reputation Power: 1
| | New Coder Real name: Maulik shah
Location: India
Gender:  Posts: 46 Topics: 1 
|
|
Re: What is the robots.txt?
September 23, 2011, 10:10:38 PM
|
In simple words, robots.txt is artlessly a plain-text file that a Web publisher should put in the root directory of their website. It’s job is to tell search engine robots what content on your website what not to visit.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
|
Reputation Power: 1
| | New Coder Posts: 39 Topics: 7 |
|
Re: What is the robots.txt?
September 24, 2011, 02:12:10 AM
|
Robots. txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots. txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots. txt is not a way from preventing search engines from crawling your site (i. e. it is not a firewall, or a kind of password protection) and the fact that you put a robots. txt file is something like putting a note “Please, do not enter” on an unlocked door – e. g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots. txt to protect it from being indexed and displayed in search results.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
|
Reputation Power: 1
| | New Poster Posts: 4 Topics: 1 |
|
Re: What is the robots.txt?
May 08, 2012, 03:50:54 AM
|
Robot. txt files tell your instructions to a search engine robot. it depends on what you what it to do. if you dont have the robot. txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. these files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. it is also important to make a note that it is not a method that prevents search engines from crawling your site. so it is important to create a robot. txt file. if you need this file to be created you can get it from TheWebPole. com
A robots. txt is a simple text file that can be created with Notepad. each robot. txt file should contain two parts
1. User agent: * means all search bots should use the instruction to crawl the website. unless your site is a complex one you need not seperate instructions for seperate spiders.
2. disallow : contains the files which should be avoided from crawling.
Eg.
User-agent: *
Disallow: /wp-
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
 |
Reputation Power: 1
| | New Coder Real name: Sir Timan
Location: 7figurementor.com/goto/50aday
Posts: 16 Topics: 1  
|
|
Re: What is the robots.txt?
May 08, 2012, 09:44:29 AM
|
Is it possible the robots are wrong about the crawling?
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
|
Reputation Power: 3
| | Professional Coder Location: India, Punjab/Ludhiana
Posts: 200 Topics: 2 |
|
Re: What is the robots.txt?
May 11, 2012, 04:11:42 AM
|
robot.txt file is used for giving instruction to search engine robots regarding what pages on your website should be crawled and consequently indexed.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
 |
Reputation Power: 3
| | Professional Coder Posts: 215 Topics: 1 |
|
Re: What is the robots.txt?
May 12, 2012, 06:32:07 AM
|
Robots. txt file is the file which is used for hiding some privacy or policy page of a website from spider. So that spider will not will not reach this page.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
|
Reputation Power: 2
| | Skilled Coder Posts: 133 Topics: 7 |
|
Re: What is the robots.txt?
May 15, 2012, 10:11:56 PM
|
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. It is a set of instructions.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
|
Reputation Power: 1
| | New Coder Posts: 48 Topics: 0 |
|
Re: What is the robots.txt?
August 13, 2012, 05:08:24 AM
|
Robots. txt file is the file which is used for hiding some privacy or policy page of a website from spider. So that spider will not will not reach this page.
nice information about robots.txt file.Now i get the clear idea why use the robots.txt file.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
|
Reputation Power: 1
| | Regular Coder Posts: 63 Topics: 0 |
|
Re: What is the robots.txt?
November 30, 2012, 05:17:22 AM
|
Robots.txt= A file on your website that can either allow robots or restrict them. Robots files can be useful when you want duplicate pages to be ignored, or search engines are crawling unnecessary pages.
« Last Edit: February 26, 2013, 01:42:23 AM by Abhi »
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
|
Reputation Power: 1
| | New Coder Location: Delhi
Gender:
Posts: 37 Topics: 3 |
|
Re: What is the robots.txt?
December 04, 2012, 05:14:38 AM
|
Robots.txt is a very useful file that is used to exclude content from the crawling process of search engine bots.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
|
Reputation Power: 2
| | Skilled Coder Posts: 120 Topics: 0 |
|
Re: What is the robots.txt?
December 13, 2012, 09:33:14 PM
|
Hello,
Robots. txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots. txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
|
Reputation Power: 1
| | New Coder Posts: 22 Topics: 1 |
|
Re: What is the robots.txt?
May 10, 2013, 03:30:35 AM
|
Robot.txt tells search engine crawlers about which pages of your website to track and which not.
Sorry, but you are not allowed to view signatures , please Register or Login
|
|
| |
TechnicalTalk » SEO and Marketing Success Strategy » SEO - Search Engine Optimization » What is the robots.txt?
