巨大的插入HBase(Huge insert to HBase)

 当我尝试向HBase插入数据时遇到问题。 
 
 我有一个1200万行Spark DataFrame有2个字段： 
 
* KEY, a md5 hash
* MATCH, a boolean ("1" or "0")
 
 我需要将它存储在HBase表中，KEY是rowkey，MATCH是列。 
 
 我在rowkey上创建了一个分割表： 
 
create 'GTH_TEST', 'GTH_TEST', {SPLITS=> ['10000000000000000000000000000000',
'20000000000000000000000000000000','30000000000000000000000000000000',
'40000000000000000000000000000000','50000000000000000000000000000000',
'60000000000000000000000000000000','70000000000000000000000000000000',
'80000000000000000000000000000000','90000000000000000000000000000000',
'a0000000000000000000000000000000','b0000000000000000000000000000000',
'c0000000000000000000000000000000','d0000000000000000000000000000000',
'e0000000000000000000000000000000','f0000000000000000000000000000000']}
 
 我使用Hortonworks的HBase shc连接器，如下所示： 
 
df.write
  .options(Map(HBaseTableCatalog.tableCatalog -> cat_matrice))
  .format("org.apache.spark.sql.execution.datasources.hbase")
  .save()
 
 此代码永不结束。 它开始向HBase插入数据并永久运行（至少在我杀死它之前35小时）。 它执行11984/16000个任务，总是相同数量的任务。 
 
 我做了一个单独的更改： 
 
df.limit(Int.MaxValue)
  .write
  .options(Map(HBaseTableCatalog.tableCatalog -> cat_matrice))
  .format("org.apache.spark.sql.execution.datasources.hbase")
  .save()
 
 使用限制（Int.MaxValue） ，需要4/5分钟才能插入1200万行。 
 
 有人可以解释这种行为吗？ HBase端是否有max_connexions？ HBase或Spark方面有一些调整吗？ 
 
 谢谢 ！ 
 
 杰弗里 

I have an issue when I try to insert data to HBase.
 
I have a 12 million lines Spark DataFrame with 2 fields :
 
* KEY, a md5 hash
* MATCH, a boolean ("1" or "0")
 
I need to store it in an HBase table, KEY is the rowkey and MATCH is a column.
 
I created the table with a split on rowkey :
 
create 'GTH_TEST', 'GTH_TEST', {SPLITS=> ['10000000000000000000000000000000',
'20000000000000000000000000000000','30000000000000000000000000000000',
'40000000000000000000000000000000','50000000000000000000000000000000',
'60000000000000000000000000000000','70000000000000000000000000000000',
'80000000000000000000000000000000','90000000000000000000000000000000',
'a0000000000000000000000000000000','b0000000000000000000000000000000',
'c0000000000000000000000000000000','d0000000000000000000000000000000',
'e0000000000000000000000000000000','f0000000000000000000000000000000']}
 
I use the HBase shc connector from Hortonworks like this :
 
df.write
  .options(Map(HBaseTableCatalog.tableCatalog -> cat_matrice))
  .format("org.apache.spark.sql.execution.datasources.hbase")
  .save()
 
This code never ends. It starts inserting data to HBase and runs forever (at least 35 hours before I killed it). It performs 11984/16000 tasks, always the same number of tasks.
 
I made a single change :
 
df.limit(Int.MaxValue)
  .write
  .options(Map(HBaseTableCatalog.tableCatalog -> cat_matrice))
  .format("org.apache.spark.sql.execution.datasources.hbase")
  .save()
 
With the limit(Int.MaxValue), it takes 4/5 minutes to insert 12 million lines.
 
Can somebody explain this behaviour ? Is there a max_connexions on HBase side ? Is there some tuning to do on HBase or Spark side ?
 
Thanks !
 
Geoffrey

原文：https://stackoverflow.com/questions/37968215