标签正在列表标签中(The label is falling down in the list tag)

 我有几点建议： 
 
 
 
 如果你使用NGrams考虑更细粒度的标记化器。 这里的目标是纠正错误拼写： 
RegexTokenizer regexTokenizer = new RegexTokenizer()
   .setInputCol("text")
   .setOutputCol("words")
   .setPattern("");

NGram ngramTransformer = new NGram()
  .setN(3)
  .setInputCol("words")
  .setOutputCol("ngrams");
 使用您当前的代码（ NGram(3)和句子三个单词由\W分割）三，您将只获得一个标记，没有相似性。 
 
 
 增加LSH的哈希表数（ setNumHashTables ）。 除了简单的例子，默认值（1）对于任何东西都是小的。 
 
 
 规范化Unicode字符串。 有一个Scala Transformer 在PySpark中用apache spark数据帧删除重音的最佳方法是什么？ 
 
 
 删除大写。 您可以使用SQLTransformer ： 
import org.apache.spark.ml.feature.SQLTransformer

val sqlTrans = new SQLTransformer().setStatement(
   "SELECT *, lower(normalized_text) FROM __THIS__")
 

I have a few suggestions:
 
 
 
If you use NGrams consider more granular tokenizer. The goal here is to correct for misspellings: 
 
RegexTokenizer regexTokenizer = new RegexTokenizer()
   .setInputCol("text")
   .setOutputCol("words")
   .setPattern("");

NGram ngramTransformer = new NGram()
  .setN(3)
  .setInputCol("words")
  .setOutputCol("ngrams");
 
With your current code (NGram(3) and sentence three words split by \W) three you'll get only one token and no similarity.
 
 
Increase number of hash tables (setNumHashTables) for LSH. Default value (1) is to small for anything but simple examples.
 
 
Normalize Unicode strings. There is a Scala Transformer in What is the best way to remove accents with apache spark dataframes in PySpark?
 
 
Remove capitalization. You can use SQLTransformer:
 
import org.apache.spark.ml.feature.SQLTransformer

val sqlTrans = new SQLTransformer().setStatement(
   "SELECT *, lower(normalized_text) FROM __THIS__")