Heritrix源码分析(六) Heritrix的文件结构分析

   本博客属原创文章,欢迎转载！转载请务必注明出处:http://guoyunsky.javaeye.com/blog/642618

       欢迎加入Heritrix群(QQ)： 109148319 , 10447185（已满）  , Lucene/Solr群(QQ) :  11897272

       每通过Heritrix运行一次抓取后,发现在该Job目录下就会有很多文件。这里说明下每个文件的作用，同时更主要介绍它的日志文件，因为我们可以通过日志文件发现Heritrix的抓取情况。首先贴个图:
 
        以上就是Heritrix完成的文件结构，现在一一说明

        现在重点说明一下Heritrix的所有日志文件：

             1.抓取汇报文件:crawl-report.txt 
                    抓取名字:Crawl Name
                    抓取状态:Crawl Status
                    抓取用时:Duration Time
                    成功抓取的种子数:Total Seeds Crawled    
                    没有抓取的种子数:Total Seeds not Crawled 
                    共抓取的host个数:Total Hosts Crawled
                    共抓取的文档数(URL数):Total Documents Crawled
                    处理速度(文档/秒):Processed docs/sec
                    宽带速率(KB/秒):Bandwidth in Kbytes/sec
                    抓取的字节数:Total Raw Data Size in Bytes
                    共抓取的字节数:Novel Bytes: 11644599984 (11 GB)

           2.调度器汇报:frontier-report.txt 
                   1)队列描述：
                             处理中队列:IN-PROCESS QUEUES
                             准备队列:READY QUEUES
                             延迟队列:SNOOZED QUEUES
                             不在活动状态队列:INACTIVE QUEUES
                             退休队列:RETIRED QUEUES
                   2)属性描述:
                              queue:队列ID,为class key
                              currentSize:当前队列包含URL个数
                              totalEnqueues:进入该队列的URL个数
                              sessionBalance:预算值
                              lastCost:上一个URL所花费成本
                              (averageCost):平均成本
                              lastDequeueTime:上一个URL出队列的时间 
                              wakeTime:醒来的时间 
                              totalSpend/totalBudget:总花费 
                              errorCount:出现错误的URL个数 
                              lastPeekUri:上一个获取的URL 
                             lastQueuedUri:上一个进入队列的URL

          3.抓取host汇报(按urls个数从大到小排序):hosts-report.txt 
                  [#urls]:该host下URL个数
                  [#bytes]:该host下所抓取的字节数
                  [host]:host 
                  [#robots]:被爬虫协议拒绝的url个数
                  [#remaining]:剩下还未处理的URL个数

         4.网页类型(一般为text/dns和text/html)抓取汇报:mimetype-report.txt 
                  [#urls]:该网页类型的url个数
                  [#bytes]:该网页类型一共处理的字节数
                  [mime-types]:网页类型(text/dns和text/html)

         5.处理器汇报:processors-report.txt 
                 1)总体汇报:Processors report
                          Job being crawled:该JOB来源的JOB
                          Number of Processors:处理器个数,包括无效的处理器
                          NOTE:处理器说明
                 2)单个Processor汇报:
                          Processor:处理器名
                          Function:处理器功能
                          CrawlURIs handled:该处理器下处理的URL个数
                          Recovery retries:
                          Links extracted:抽取出来的URL个数

          6.Http response汇报:responsecode-report.txt 
                         [rescode]:Http response code
                                     1:请求http
                                     200:http成功相应
                                     302:暂时性重定向
                                     500:http服务器内部错误
                          [#urls]:是该code的URL个数