Robots.txt Generator

Link Directory
Want Exposure? Looking for one way Backlinks? Get Text Link Advertising for $1.00.

link directory

main menu

Free Templates

Free Template Stats
Total Templates: 335
Total Categories: 15
Total Downloads: 4232

Nobody Loves Me Blogger
Christmas Joomla
MobileMagic 1.0 WordPress
tem_asteri_v2 Joomla
Puppy Madness! Wordpress
Hardware Flash Template
CLB Business Joomla
The Working Girl 2.0 Wordpress
Computer Blogger Template
The Girl (Blue) Blogger

Feature Resources

translation services
translation

free resources

Total Links: 1039
Total Categories: 40

1: FontFile
2: eZ Publish
3: CGI - Works
4: Photoshop Contest
5: PHP Coder

Robots.txt Generator Tool

Robots.txt files are used by web owners and webmasters to signify to spider-bots and search engines not to crawl or index certain files and directories. The Robots.txt file is a command file that most creditable search-bots will obey and check for on entry to your website URL. To create a robots.txt file, use the following form and select the fields to either disallow certain bots, disallow directories or deny access from bad-bots. If you have any questions please use our support forums.

Support future development

If you enjoy our services and online resources consider making a donation to support our efforts.

Menu

| Creating A Robots.txt File | List of Bad-Bots |

Default Robots.txt file setting

Default - All Robots are:

Crawl-Delay:

Specific Search Robots

Search Engine	Choose Setting	Bot-Name
Google		googlebot
MSN Search		msnbot
Yahoo		yahoo-slurp
Alexa/Wayback		ia_archiver
Ask/Teoma		teoma
DMOZ		robozilla
GigaBlast		gigabot
Scrub The Web		scrubby

Specific Special Bots:

Search Engine	Choose Setting	Bot-Name
Google Image		googlebot-image
Yahoo MM		yahoo-mmcrawler
MSN Pic-Search		psbot
Yahoo Blogs		yahoo-blogs/v3.9

Extra Exclusion features:

Search Engine	Choose Setting	Bot-Name
Google		googlebot
MSN		msnbot
Yahoo		yahoo-slurp

Disallow Bad Bots:
Select this option to block the most common "Bad-Bots," also known for spam and email harvesting from accessing your domain.
Disallow Bad-Bots:	User-agent: Black Hole User-agent: Titan User-agent: WebStripper User-agent: NetMechanic User-agent: CherryPicker User-agent: EmailCollector User-agent: EmailSiphon User-agent: WebBandit User-agent: EmailWolf User-agent: ExtractorPro User-agent: CopyRightCheck User-agent: Crescent User-agent: Wget User-agent: SiteSnagger User-agent: ProWebWalker User-agent: CheeseBot User-agent: Teleport User-agent: TeleportPro User-agent: MIIxpc User-agent: Telesoft User-agent: Website Quester User-agent: WebZip User-agent: moget/2.1 User-agent: WebZip/4.0 User-agent: WebSauger User-agent: WebCopier User-agent: NetAnts User-agent: Mister PiX User-agent: WebAuto User-agent: TheNomad User-agent: WWW-Collector-E User-agent: RMA User-agent: libWeb/clsHTTP User-agent: asterias User-agent: httplib User-agent: turingos User-agent: spanner User-agent: InfoNaviRobot User-agent: Harvest/1.5 User-agent: Bullseye/1.0 User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 User-agent: CherryPickerSE/1.0 User-agent: CherryPicker /1.0 User-agent: WebBandit/3.50 User-agent: NICErsPRO User-agent: Microsoft URL Control - 5.01.4511 User-agent: DittoSpyder User-agent: Foobot User-agent: WebmasterWorldForumBot User-agent: SpankBot User-agent: BotALot User-agent: lwp-trivial/1.34 User-agent: lwp-trivial User-agent: Wget/1.6 User-agent: BunnySlippers User-agent: Microsoft URL Control - 6.00.8169 User-agent: URLy Warning User-agent: Wget/1.5.3 User-agent: LinkWalker User-agent: cosmos User-agent: moget User-agent: hloader User-agent: humanlinks User-agent: LinkextractorPro User-agent: Offline Explorer User-agent: Mata Hari User-agent: LexiBot User-agent: Web Image Collector User-agent: The Intraformant User-agent: True_Robot/1.0 User-agent: True_Robot User-agent: BlowFish/1.0 User-agent: JennyBot User-agent: MIIxpc/4.2 User-agent: BuiltBotTough User-agent: ProPowerBot/2.14 User-agent: BackDoorBot/1.0 User-agent: toCrawl/UrlDispatcher User-agent: WebEnhancer User-agent: TightTwatBot User-agent: suzuran User-agent: VCI WebViewer VCI WebViewer Win32 User-agent: VCI User-agent: Szukacz/1.4 User-agent: QueryN Metasearch User-agent: Openfind data gathere User-agent: Openfind User-agent: Xenu's Link Sleuth 1.1c User-agent: Xenu's User-agent: Zeus User-agent: RepoMonkey Bait & Tackle/v1.01 User-agent: RepoMonkey User-agent: Zeus 32297 Webster Pro V2.9 Win32 User-agent: Webster Pro User-agent: EroCrawler User-agent: LinkScan/8.1a Unix User-agent: Keyword Density/0.9 User-agent: Kenjin Spider User-agent: Cegbfeieh

Restricted Directories:

The path is relative to root and must contain a trailing "/"

Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"

Create The Robots.txt

Now, copy and paste this text above into a blank text file called "robots.txt" and put it in the root directory of your website. Make sure it's permissions are set so that visitors and search engines can read it.

Example: http://www.yoursite.com/robots.txt

Car Insurance | Free Ringtones | Zune | Debt Consolidation | Magazine Subscriptions

Get Your Ad Here

Click Here to Advertise

Affordable Hosting! http://www.invision-graphics.com/images/banners/468X60_VISIONHOSTING.gif

Today: 22,745 Yesterday: 49,846 Total Hits: 18,197,149
Page Rendered in: 0.23s - Total Queries: 25 - MySQL DB: 17.1 mb's - Pages served in past 5 minutes: 161

Invision-Graphics