|
Robots.txt Validator
|
|
Robots.txt files are used by web owners and webmasters to signify to spider-bots and search engines not to crawl or index certain files and directories. The Robots.txt file is a command file that most creditable search-bots will obey and check for on entry to your website URL.
|
|
This robots.txt validator will analyze the syntax of your robots.txt file to see if its format is valid. If there are any errors in your robots.txt file they will be marked in red to point out which sections of your robots.txt file need attention.
|
How to check your robots.txt file?
To check your robots.txt file, simply put the full URL. Example: http://www.yoursite.com/robots.txt
|
What will the robots.txt validator look for?
The robots.txt validator will examine your file and look for syntax errors, mistyped words, improper use of the asterisk * and the disallow line. It will mark any errors found in red to indicate what sections need correction.
|
|
|
|
Support future development
|
If you enjoy our services and online resources consider making a donation to support our efforts.
|
