Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Robots. txt is a file that tells search engine spiders to not crawl certain pages or sections of a website. Most major search engines (including Google, Bing and Yahoo) recognize and honor Robots. txt requests.

txt file controls which pages are accessed. The robots Meta tag controls whether a page is indexed, but to see this tag the page needs to be crawled. If crawling a page is problematic (for example, if the page causes a high load on the server), you should use the robots. txt file.

Robots. txt is one of the simplest files on a website, but it's also one of the easiest to mess up. Just one character out of place can wreak havoc on your SEO and prevent search engines from accessing important content on your site.

By setting up your robots. txt file the right way, you're not just enhancing your own SEO. You're also helping out your visitors. If search engine bots can spend their crawl budgets wisely, they'll organize and display your content in the SERPs in the best way, which means you'll be more visible.

The robots. txt file must be located at the root of the website host to which it applies. For instance, to control crawling on all URLs below http://www.example.com/ , the robots. txt file must be located at http://www.example.com/robots.txt .

You can easily create a new or edit an existing robots.txt file for your site with a robots.txt generator. To upload an existing file and pre-populate the robots.txt file generator tool, type or paste the root domain URL in the top text box and click Upload. Use the robots.txt generator tool to create directives with either Allow or Disallow directives (Allow is default, click to change) for User Agents (use * for all or click to select just one) for specified content on your site. Click Add directive to add the new directive to the list. To edit an existing directive, click Remove directive, and then create a new one.

Our Robots.txt Generator tool is designed to help webmasters, SEOs, and marketers generate their robots.txt files without a lot of technical knowledge. Please be careful though, as creating your robots.txt file can have a significant impact on Google being able to access your website, whether it is built on Word Press or another CMS.