Spark - 为udf提供额外参数(Spark - provide extra parameter to udf)

 我正在尝试创建一个spark scala udf，以便转换以下形状的MongoDB对象： 
 
Object:
    "1": 50.3
    "8": 2.4
    "117": 1.0
 
 进入Spark ml SparseVector。 问题是，为了创建一个SparseVector，我还需要一个输入参数 - 它的大小。 在我的应用程序中，我将Vector大小保存在单独的MongoDB集合中。 所以，我定义了以下UDF函数： 
 
val mapToSparseVectorUdf = udf {
  (myMap: Map[String, Double], size: Int) => {
    val vb: VectorBuilder[Double] = new VectorBuilder(length = -1)
    vb.use(myMap.keys.map(key => key.toInt).toArray, myMap.values.toArray, size)
    vb.toSparseVector
  }
}
 
 而我试图这样称呼它： 
 
df.withColumn("VecColumn", mapToSparseVectorUdf(col("MapColumn"), vecSize)).drop("MapColumn")
 
 但是，我的IDE对该udf调用说“不适用”。 有没有办法使这种UDF可以采取额外的参数？ 

I'm trying to create a spark scala udf in order to transform MongoDB objects of the following shape:
 
Object:
    "1": 50.3
    "8": 2.4
    "117": 1.0
 
Into Spark ml SparseVector. The problem is that in order to create a SparseVector, I need one more input parameter - its size. And in my app I keep the Vector sizes in a separate MongoDB collection. So, I defined the following UDF function:
 
val mapToSparseVectorUdf = udf {
  (myMap: Map[String, Double], size: Int) => {
    val vb: VectorBuilder[Double] = new VectorBuilder(length = -1)
    vb.use(myMap.keys.map(key => key.toInt).toArray, myMap.values.toArray, size)
    vb.toSparseVector
  }
}
 
And I was trying to call it like this:
 
df.withColumn("VecColumn", mapToSparseVectorUdf(col("MapColumn"), vecSize)).drop("MapColumn")
 
However, my IDE says "Not applicable" to that udf call. Is there a way to make this kind of UDF that can take an extra parameter?

原文：https://stackoverflow.com/questions/49388781