首页 \ 问答 \ 在整合nutch 2.3，Hbase和Solr时，在索引方面花费了太多时间(Taking too much time in indexing while integrating nutch 2.3, Hbase and Solr)

在整合nutch 2.3，Hbase和Solr时，在索引方面花费了太多时间(Taking too much time in indexing while integrating nutch 2.3, Hbase and Solr)

 我正在整合Nutch，Hbase和Solr 。 
 

 我配置了Nutch，Hbase和Solr，并且还进行了对网站进行爬网的操作，但是在通过遵循这个集成Nutch 2.3，HBase和Solr将Nutch与Solr 集成之后 ，我在/opt/solr-4.8中执行了命令java jar start.jar。 1 /例子 。 
 

 这个过程已经开始，但是执行大约10天仍需花费很多时间，现在它还在运行。 
 

 我无法找出它出了什么问题。 任何人都可以建议问题是什么以及如何解决。 
 

 以下是日志文件的一些细节。 
 

INFO  - 2016-05-18 15:58:00.286; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  - 2016-05-18 15:58:00.287; org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes. Skipping IW.commit.
INFO  - 2016-05-18 15:58:00.287; org.apache.solr.core.SolrCore; SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
INFO  - 2016-05-18 15:58:00.288; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
INFO  - 2016-05-18 15:58:00.288; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update params={waitFlush=true&optimize=true&wt=json&_=1463567280272} {optimize=} 0 2
INFO  - 2016-05-18 15:58:01.976; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  - 2016-05-18 15:58:01.976; org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes. Skipping IW.commit.
INFO  - 2016-05-18 15:58:01.977; org.apache.solr.core.SolrCore; SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
INFO  - 2016-05-18 15:58:01.977; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
INFO  - 2016-05-18 15:58:01.978; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update params={waitFlush=true&optimize=true&wt=json&_=1463567281965} {optimize=} 0 2
INFO  - 2016-05-18 15:58:05.799; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/threads params={wt=json&_=1463567285780} status=0 QTime=8 
INFO  - 2016-05-18 15:58:09.267; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/properties params={wt=json&_=1463567289183} status=0 QTime=0 
INFO  - 2016-05-18 15:58:11.225; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={wt=json&_=1463567291213} status=0 QTime=1 
INFO  - 2016-05-18 15:58:11.260; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={wt=json&_=1463567291242} status=0 QTime=1 
INFO  - 2016-05-18 15:58:13.808; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/luke params={show=index&numTerms=0&wt=json&_=1463567293791} status=0 QTime=1 
INFO  - 2016-05-18 15:58:13.821; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/replication params={wt=json&command=details&_=1463567293794} status=0 QTime=1 
INFO  - 2016-05-18 15:58:13.837; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/system params={wt=json&_=1463567293796} status=0 QTime=4 
INFO  - 2016-05-18 15:58:13.845; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/file/ params={file=admin-extra.html&_=1463567293798} status=0 QTime=0 
INFO  - 2016-05-18 15:58:13.856; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/ping params={action=status&wt=json&_=1463567293801} status=503 QTime=1 
INFO  - 2016-05-18 16:54:35.235; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/logging params={wt=json&since=0&_=1463570675193} status=0 QTime=1 
INFO  - 2016-05-18 16:54:38.820; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/replication params={wt=json&command=details&_=1463570678769} status=0 QTime=0 
INFO  - 2016-05-18 16:54:38.821; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/luke params={show=index&numTerms=0&wt=json&_=1463570678764} status=0 QTime=2 
INFO  - 2016-05-18 16:54:38.823; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/ping params={action=status&wt=json&_=1463570678776} status=503 QTime=0 
INFO  - 2016-05-18 16:54:38.829; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/file/ params={file=admin-extra.html&_=1463570678774} status=0 QTime=1 
INFO  - 2016-05-18 16:54:38.831; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/system params={wt=json&_=1463570678772} status=0 QTime=11 
INFO  - 2016-05-18 16:54:46.728; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/mbeans params={stats=true&wt=json&_=1463570686705} status=0 QTime=5 
INFO  - 2016-05-18 16:54:49.533; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/mbeans params={stats=true&wt=json&_=1463570689477} status=0 QTime=3 
INFO  - 2016-05-18 16:54:52.762; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/replication params={wt=json&command=details&_=1463570692692} status=0 QTime=0 
INFO  - 2016-05-18 16:56:33.180; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/logging params={wt=json&since=0&_=1463570793166} status=0 QTime=0 
INFO  - 2016-05-18 16:56:38.195; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/luke params={show=index&numTerms=0&wt=json&_=1463570798128} status=0 QTime=0 
INFO  - 2016-05-18 16:56:38.198; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/replication params={wt=json&command=details&_=1463570798132} status=0 QTime=0 
INFO  - 2016-05-18 16:56:38.199; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/ping params={action=status&wt=json&_=1463570798137} status=503 QTime=0 
INFO  - 2016-05-18 16:56:38.201; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/file/ params={file=admin-extra.html&_=1463570798135} status=0 QTime=0 
INFO  - 2016-05-18 16:56:38.211; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/system params={wt=json&_=1463570798133} status=0 QTime=12 


I am integrating Nutch,Hbase and Solr .
 

I configured Nutch, Hbase and Solr and also did operation for Crawling the Websites but while integrating the Nutch with Solr by following this Integrating Nutch 2.3, HBase and Solr, I executed the command java jar start.jar in /opt/solr-4.8.1/examples. 
 

The process is started but it is taking so much time for execution about 10 days and still now it's running. 
 

I am unable to find out what is going wrong with it. Can anyone suggest what's the problem and how to solve. 
 

Below are few details of logs file.
 

INFO  - 2016-05-18 15:58:00.286; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  - 2016-05-18 15:58:00.287; org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes. Skipping IW.commit.
INFO  - 2016-05-18 15:58:00.287; org.apache.solr.core.SolrCore; SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
INFO  - 2016-05-18 15:58:00.288; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
INFO  - 2016-05-18 15:58:00.288; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update params={waitFlush=true&optimize=true&wt=json&_=1463567280272} {optimize=} 0 2
INFO  - 2016-05-18 15:58:01.976; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  - 2016-05-18 15:58:01.976; org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes. Skipping IW.commit.
INFO  - 2016-05-18 15:58:01.977; org.apache.solr.core.SolrCore; SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
INFO  - 2016-05-18 15:58:01.977; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
INFO  - 2016-05-18 15:58:01.978; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update params={waitFlush=true&optimize=true&wt=json&_=1463567281965} {optimize=} 0 2
INFO  - 2016-05-18 15:58:05.799; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/threads params={wt=json&_=1463567285780} status=0 QTime=8 
INFO  - 2016-05-18 15:58:09.267; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/properties params={wt=json&_=1463567289183} status=0 QTime=0 
INFO  - 2016-05-18 15:58:11.225; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={wt=json&_=1463567291213} status=0 QTime=1 
INFO  - 2016-05-18 15:58:11.260; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={wt=json&_=1463567291242} status=0 QTime=1 
INFO  - 2016-05-18 15:58:13.808; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/luke params={show=index&numTerms=0&wt=json&_=1463567293791} status=0 QTime=1 
INFO  - 2016-05-18 15:58:13.821; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/replication params={wt=json&command=details&_=1463567293794} status=0 QTime=1 
INFO  - 2016-05-18 15:58:13.837; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/system params={wt=json&_=1463567293796} status=0 QTime=4 
INFO  - 2016-05-18 15:58:13.845; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/file/ params={file=admin-extra.html&_=1463567293798} status=0 QTime=0 
INFO  - 2016-05-18 15:58:13.856; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/ping params={action=status&wt=json&_=1463567293801} status=503 QTime=1 
INFO  - 2016-05-18 16:54:35.235; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/logging params={wt=json&since=0&_=1463570675193} status=0 QTime=1 
INFO  - 2016-05-18 16:54:38.820; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/replication params={wt=json&command=details&_=1463570678769} status=0 QTime=0 
INFO  - 2016-05-18 16:54:38.821; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/luke params={show=index&numTerms=0&wt=json&_=1463570678764} status=0 QTime=2 
INFO  - 2016-05-18 16:54:38.823; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/ping params={action=status&wt=json&_=1463570678776} status=503 QTime=0 
INFO  - 2016-05-18 16:54:38.829; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/file/ params={file=admin-extra.html&_=1463570678774} status=0 QTime=1 
INFO  - 2016-05-18 16:54:38.831; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/system params={wt=json&_=1463570678772} status=0 QTime=11 
INFO  - 2016-05-18 16:54:46.728; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/mbeans params={stats=true&wt=json&_=1463570686705} status=0 QTime=5 
INFO  - 2016-05-18 16:54:49.533; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/mbeans params={stats=true&wt=json&_=1463570689477} status=0 QTime=3 
INFO  - 2016-05-18 16:54:52.762; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/replication params={wt=json&command=details&_=1463570692692} status=0 QTime=0 
INFO  - 2016-05-18 16:56:33.180; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/info/logging params={wt=json&since=0&_=1463570793166} status=0 QTime=0 
INFO  - 2016-05-18 16:56:38.195; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/luke params={show=index&numTerms=0&wt=json&_=1463570798128} status=0 QTime=0 
INFO  - 2016-05-18 16:56:38.198; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/replication params={wt=json&command=details&_=1463570798132} status=0 QTime=0 
INFO  - 2016-05-18 16:56:38.199; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/ping params={action=status&wt=json&_=1463570798137} status=503 QTime=0 
INFO  - 2016-05-18 16:56:38.201; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/file/ params={file=admin-extra.html&_=1463570798135} status=0 QTime=0 
INFO  - 2016-05-18 16:56:38.211; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/admin/system params={wt=json&_=1463570798133} status=0 QTime=12 


原文：https://stackoverflow.com/questions/37297902
更新时间：2022-01-27 06:01

最满意答案

 你不能！ 
 

 这实际上是一个安全功能。 您绝不应以纯文本格式存储密码。 
 

 好消息是，您不需要自己进行比较： 
 

 而是使用类似ValidateUser东西让成员资格提供程序验证提供的密码。 在幕后，此方法将散列密码并将其与数据库中包含的散列版本进行比较。 
 

 编辑： 
 

 另请注意，由于您使用的是WebSecurity类，因此有一个接受当前密码的方法ChangePassword 。 似乎该方法将检查当前密码是否与指定的currentPassword参数匹配。 也许你应该使用这个而不是ResetPassword 

You can't !
 

That's actually a security feature. You should never store a password in plain text.
 

The good thing is, you don't need to do the comparison yourself:
 

Instead, use something like ValidateUser to let the Membership Provider validate the provided password. Behind the scenes, this method will hash the password and compare it with the hashed version contained in the database.
 

EDIT: 
 

Also, note that since you are using the WebSecurity class, there is a method, ChangePassword that accepts the current password. It seems that method will check the current password matches the specified currentPassword parameter. Maybe you should use this one instead of ResetPassword

相关问答

更多

  • LINUX 如何查看JPG文件[2022-06-13]

    find

  • 使用Expect Module更改密码(ldap password change with Expect Module)[2023-09-11]

    你知道Net :: Ldap模块吗？ http://search.cpan.org/~gbarr/perl-ldap/lib/Net/LDAP/FAQ.pod#Ho_do_I_reset_a_user%27s_password_ ... 没有错误检查！ use Net::LDAP; my $ldap = Net::LDAP->new('ldaps://server.domain') or die "$@"; my $mesg = $ldap->bind('cn=Joe User,dc=pe ...

  • 如何找到OldPassword让用户更改它(How to find OldPassword to let the user change it)[2024-01-31]

    你不能！ 这实际上是一个安全功能。 您绝不应以纯文本格式存储密码。 好消息是，您不需要自己进行比较： 而是使用类似ValidateUser东西让成员资格提供程序验证提供的密码。 在幕后，此方法将散列密码并将其与数据库中包含的散列版本进行比较。 编辑： 另请注意，由于您使用的是WebSecurity类，因此有一个接受当前密码的方法ChangePassword 。 似乎该方法将检查当前密码是否与指定的currentPassword参数匹配。 也许你应该使用这个而不是ResetPassword You can't ...

  • IdentityUser如何验证旧密码？(IdentityUser how to verify old password?)[2021-12-25]

    ChangePasswordAsync()方法要求不对旧密码进行哈希处理。 试试这个。 _userManager.ChangePasswordAsync(appUser, model.OldPassword, model.NewPassword); The ChangePasswordAsync() method expects the old password not to be hashed. Try this instead. _userManager.ChangePasswordAsync(app ...

  • 需要一种方法来更改远程用户密码 - NetUserChangePassword失败，带有2245(Need a way to change a remote user password - NetUserChangePassword fails with 2245)[2022-02-21]

    错误2245也可能是密码历史记录问题。 新密码是最近使用的密码吗？ 编辑：看起来这个功能在Server 2003 SP 2之后破了。我在使用文档中的示例从C ++调用函数时遇到了同样的错误。 您可能需要使用NetUserSetInfo。 Error 2245 could also be a password history problem. Is the new password one that was used in the recent past? Edit: It looks like this f ...

  • 更改Umbraco会员密码并输出旧密码(Change Umbraco member password with out knowing old password)[2023-07-11]

    Umbraco使用Microsoft会员供应商。 您可能已将属性“ EnablePasswordRetrieval ”设置为false。 如果您不知道密码但需要更改密码，则可以使用ResetPassword方法重置密码bij。 Umbraco uses the Microsoft Membership Provider. You probably have set the property "EnablePasswordRetrieval" to false. If you don't know the p ...

  • 如何在Meteor中安全地更改密码？(How to change password securely in Meteor?)[2021-11-26]

    旧密码和新密码不是通过网络发送的纯文本，它在手册中说明了这一点。 密码在客户端编码并通过线路加密发送。 但是，changePassword（）附带了accounts-password包，并没有在服务器上提供回调，也没有允许配置设置来防止密码更改。 如果您不希望在客户端上使用它，请不要安装帐户密码并滚动您自己的包。 我不知道你为什么要那样做。 Old and new passwords are NOT sent plain text over the wire, it states so in the man ...

  • 是否可以更改npm用户？(Is it possible to change npm user?)[2023-07-09]

    我建议联系npm支持 。 AFAIK没有公开的，官方的方式来更改您的注册用户名。 根据我的经验，他们非常有帮助和好:-) I'd suggest to contact npm support for this. AFAIK there is no publicly available, official way to change your registered user name. From my experience, they are very helpful and nice :-)

  • SignOut后Asp.net身份更改密码(Asp.net Identity Change Password After SignOut)[2022-10-02]

    问题解决了！ var userName = HttpContext.Current.User.Identity.Name; var user = _userManager.Find(userName, oldPassword); user.PasswordHash = UserManager.PasswordHasher.HashPassword(password); IdentityResult result = await UserManager.UpdateAsync(user); Problem ...

  • 过期的密码更改失败(Expired password change fails)[2022-08-28]

    密码是否已过期或是否已禁用用户配置文件？ 如果配置文件被禁用，则根本无法登录...由于您无法登录，因此无法更改密码。 尝试在更改密码之前调用connect（AS400.SIGNON）方法。 如果所有其他方法都失败了，请尝试激活跟踪以查看系统在封面下尝试执行的操作。 Yes the user was not disabled (enabled then :P) only the password was expired. As it turned out the problem was that the fi ...

相关文章

更多
  • nutch安装,与solr整合
  • Apache Nutch（一）
  • Nutch&Solr小计
  • lucene2.0与2.3区别
  • 荐 安装nutch2+Hbase+Slor4
  • nutch
  • 使用nutch和solr建立搜索引擎
  • Solr3.6.2与nutch1.6的整合
  • Param's Blog: Nutch 1.3 and Solr Integration
  • Nutch安装指南

    • 最新问答

    更多
  • h2元素推动其他h2和div。(h2 element pushing other h2 and div down. two divs, two headers, and they're wrapped within a parent div)
  • 创建一个功能(Create a function)
  • 我投了份简历，是电脑编程方面的学徒，面试时说要培训三个月，前面
  • PDO语句不显示获取的结果(PDOstatement not displaying fetched results)
  • Qt冻结循环的原因？(Qt freezing cause of the loop?)
  • TableView重复youtube-api结果(TableView Repeating youtube-api result)
  • 如何使用自由职业者帐户登录我的php网站？(How can I login into my php website using freelancer account? [closed])
  • SQL Server 2014版本支持的最大数据库数(Maximum number of databases supported by SQL Server 2014 editions)
  • 我如何获得DynamicJasper 3.1.2（或更高版本）的Maven仓库？(How do I get the maven repository for DynamicJasper 3.1.2 (or higher)?)
  • 以编程方式创建UITableView(Creating a UITableView Programmatically)
  • 如何打破按钮上的生命周期循环(How to break do-while loop on button)
  • C＃使用EF访问MVC上的部分类的自定义属性(C# access custom attributes of a partial class on MVC with EF)
  • 如何获得facebook app的publish_stream权限？(How to get publish_stream permissions for facebook app?)
  • 如何防止调用冗余函数的postgres视图(how to prevent postgres views calling redundant functions)
  • Sql Server在欧洲获取当前日期时间(Sql Server get current date time in Europe)
  • 设置kotlin扩展名(Setting a kotlin extension)
  • 如何并排放置两个元件？(How to position two elements side by side?)
  • 如何在vim中启用python3？(How to enable python3 in vim?)
  • 在MySQL和/或多列中使用多个表用于Rails应用程序(Using multiple tables in MySQL and/or multiple columns for a Rails application)
  • 如何隐藏谷歌地图上的登录按钮？(How to hide the Sign in button from Google maps?)
  • Mysql左连接旋转90°表(Mysql Left join rotate 90° table)
  • dedecms如何安装？
  • 在哪儿学计算机最好?
  • 学php哪个的书 最好,本人菜鸟
  • 触摸时不要突出显示表格视图行(Do not highlight table view row when touched)
  • 如何覆盖错误堆栈getter(How to override Error stack getter)
  • 带有ImageMagick和许多图像的GIF动画(GIF animation with ImageMagick and many images)
  • USSD INTERFACE - > java web应用程序通信(USSD INTERFACE -> java web app communication)
  • 电脑高中毕业学习去哪里培训
  • 正则表达式验证SMTP响应(Regex to validate SMTP Responses)

    • Copyright ©2023 peixunduo.com All Rights Reserved.粤ICP备14003112号

    本站部分内容来源于互联网，仅供学习和参考使用，请莫用于商业用途。如有侵犯你的版权，请联系我们(neng862121861#163.com)，本站将尽快处理。谢谢合作！