Introduce a Robots.txt file validation tool online

2010-03-16  来源:本站原创  分类:Internet  人气:214 

http://tool.motoricerca.info/robots-checker.phtml

The robots.txt analysis is an Italian by the Italian non-profit web site optimization and search engine Motoricerca's,

Note, however, use this tool, robots.txt should be clear in the character notes, it does not recognize Chinese characters; If character is not considered a valid robots.txt file.

If the Robots file verification passed, usually show the following results:

Analyzing file http://www.XXX.com/robots.txt
No errors found in this robots.txt file
Hide empty and comments lines:  

The following block of code DISALLOWS the crawling of the following files and directories: /inc/ to all spiders/robots.
Line 1 # robots.txt for www.XXX.com
Line 2 User-agent: *
Line 3 Disallow: /inc/

In addition, Google Webmaster Tools verification robots can file online, Google is supporting the Chinese comments.
Log in to Google webmaster tools, in the "Control Panel", click the required web site, click on Tools, then click on the line of robots.txt.

相关文章
  • Introduce a Robots.txt file validation tool online 2010-03-16

    http://tool.motoricerca.info/robots-checker.phtml The robots.txt analysis is an Italian by the Italian non-profit web site optimization and search engine Motoricerca's, Note, however, use this tool, robots.txt should be clear in the character notes,

  • Site Help robots.txt file 2010-02-26

    robots.txt is a very simple text file, you simply indicate the "who can not access which links" can be. Write the first line in the file: User-Agent: YodaoBot This tells the following description of the reptiles called YodaoBot for reptiles. You

  • Site robots.txt file description (2) 2010-02-26

    Today, one day, accurately, less than three hours apart, there are two friends in the QQ and let me help him look website, why does not your search engine. From my personal point of view, they stand to do quite well, apart from some places (if the ti

  • the use of robots.txt file 2010-05-04

    We often Google Webmaster Tools, or My applications of the log file to see the name robots.txt. This file is in the end what is the use? Judging from the name, it seems like search engine crawlers have been, yes, you guessed it. This document is a re

  • Interpretation of the robots.txt file 2011-01-21

    robots.txt Search engine web root directory, or write web crawler, should have learned that a robots.txt file. So what is the use of this file? Here we give its definition and purpose: Quote from: Wikipedia, the free encyclopedia robots.txt (Unified

  • robots.txt and Meta tags in SEO role. how to submit a site map (transfer) 2011-01-04

    Transfer from: http://hi.baidu.com/wzk717/blog/item/ab854ddfb8350a19495403d8.html 1, robots.txt is what? robots.txt is a text file, used to tell the spider (not limited to search engine spiders) to crawl this site which allows the location, which loc

  • robots.txt and Meta tags in SEO role. how to submit your site map (transfer) 2011-01-04

    Transfer: http://hi.baidu.com/wzk717/blog/item/ab854ddfb8350a19495403d8.html 1, robots.txt is what? robots.txt is a text file that tells web spiders (search engine spiders are not limited to) what position allowed to crawl the site, which location is

  • Understand the use of robots.txt search engines crawl and index optimization 2010-05-29

    seo robots.txt usage and the role of -Googlebot/Baiduspider By setting the appropriate robots.txt to the site on Google and Baidu seo optimization effect is obvious. WordPress blog site, too. We first look at robots.txt what for? robots.txt What is t

  • Heritrix cancel robots.txt 2010-09-12

    Robots.txt is a file used to limit Web crawler, if the building site, placed at the site of a Robots.txt file, which can be declared in the search engines do not want to part. But it is also Heritrix crawler to crawl pages in spending too much time t

  • seo robots.txt usage and the role of -Googlebot/Baiduspider 2010-11-15

    By setting the appropriate robots.txt to the site on Google and Baidu optimization seo role is obvious. WordPress blog site, too. We take a look at what is robots.txt, what role? What is robots.txt? We all know that the file is a plain txt suffix tex

  • Robots.txt protocol standards introduced 2011-05-03

    Robots.txt is located in the site root directory of a plain text file. Although its setting is simple, but the role is very powerful. It can specify the search engine spiders crawl only specific content, or the prohibition of the search engine spider

  • [Turn] How to write a robots.txt? 2011-03-15

    In China, the site manager did not seem to cause much attention to robots.txt, you should request of some friends today want to talk about this article to a simple robots.txt writing. robots.txt basic introduction robots.txt is a text file, in this f

  • robots.txt 2012-06-02

    robots.txt是一个协议,而不是一个命令.robots.txt是搜索引擎中访问网站的时候要查看的第一个文件.robots.txt文件告诉蜘蛛程序在服务器上什么文件是可以被查看的. 当一个搜索蜘蛛访问一个站点时,它会首先检查该站点根目录下是否存在robots.txt,如果存在,搜索机器人就会按照该文件中的内容来确定访问的范围:如果该文件不存在,所有的搜索蜘蛛将能够访问网站上所有没有被口令保护的页面.百度官方建议,仅当您的网站包含不希望被搜索引擎收录的内容时,才需要使用robots.txt文件

  • File comparison tool (to do the software version) 2011-03-30

    1 When the needs analysis, detailed design and development done later. We will begin to do for software version upgrade in the original basis, or to increase module development Choose a good file comparison tool is required, this tool support files,

  • Txt file reading and writing VB 2010-03-08

    Each record to a line that each record field has "@" symbol distinction. The length of each field has its maximum limit. And the field is defined as segment symbol position calculation, if the record in After a field to place line breaks, then t

  • CuteRegexTool: publish a regular expression validation tool 2010-03-20

    About <br /> regular expression validation tool, based on. Net framework 2.0, used to validate regular expressions. Immediately verify the regular expression; Support for multiple matching model; Highlight matches. Effect Picture Download

  • How to write CString ANSI, Unicode-encoded txt file 2010-03-01

    1. In VS2005 in, CString type in the characters as wide character. 2. To write ANSI encoded string CString txt file: CString szChar = L ""; szChar = setlocale (LC_CTYPE, ("chs")); CStdioFile * file = new CStdioFile (aTxtFileName, CFile

  • Android Study Notes - mdb file into txt file 2010-04-27

    27 Apr 2010 The question is: When I copy files to xls mdb file and then copy the txt file, appeared in the black box. Tried it last night, if you just want to delete all of the black block, just use wordpad.exe (notepad) to open a TXT file and then c

  • Pdf file how to convert txt file 2010-08-29

    Here, I used two packages one is PDFBox-0.3.7.jar one FontBox-0.1.0-dev.jar front to the back of a package use the package so do not forget the introduction of a second package, two packages I put the following Functions to achieve the code below imp

  • Txt file to export data to Oracle 2010-10-08

    Oracle Export to TXT: 1, with the PL / SQL DEV open the CMD window. 2, spool d: / output.txt; 3, set heading off; - removed table header 4, select * from usergroup; 5, spool off; TXT into Oracle: 1, with the PL / SQL DEV TOOLS tools under the "Text I