What do the robots.txt file analysis results mean?

When you test a URL against a robots.txt file, you will see one of the following results:

* Allowed— Googlebot will crawl the URL.
* Blocked— Googlebot will not crawl the URL.
* Not in domain— This URL is not on the same domain as the robots.txt file and therefore, you cannot block it.
* Syntax not understood— Googlebot does not recognize this as a valid URL.

Additionally you may see the following message:

* Detected as a directory; specific files may have different restrictions— Although this directory is blocked or allowed, there may be other, more specific rules in the file that block or allow URLs in the directory, so you will want to check those as well.

If Googlebot has difficulty understanding parts of your robots.txt file, you will see one of the following parsing results, which you will want to fix:

* Accepted, but should be Disallow— You misspelled "Disallow."
* Accepted, but should be User-agent— You misspelled "user-agent."
* Accepted, but correct syntax includes a colon (Rule: path)— You forgot to put a colon between "Allow" or "Disallow" and the path.
* Rule ignored by Googlebot— This is not a rule that Googlebot follows (for example, "Crawl-delay").
* No user-agent specified— You have rules that aren't associated with a user-agent.
* Syntax not understood— Googlebot does not understand this line.
* robots.txt file does not appear to be valid— Googlebot doesn't understand any parts of this file and therefore, doesn't recognize it as a valid a robots.txt file.