首页 \ 问答 \ 当我编写PairDStreamFunctions.reduceByKey时,为什么scala编译器无法推断类型(why scala compiler failed to infer type when i write PairDStreamFunctions.reduceByKey)

当我编写PairDStreamFunctions.reduceByKey时,为什么scala编译器无法推断类型(why scala compiler failed to infer type when i write PairDStreamFunctions.reduceByKey)

我想知道为什么当我使用PairDStreamFunctions.reduceByKey时scala编译器无法推断出我的函数参数的类型,这里是代码:

val ssc = new StreamingContext(conf, Seconds(10))
ssc.checkpoint(".checkpoint")
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" "))
val wordCounts = words
  .map((_, 1))
  .reduceByKey((x: Int, y: Int) => x + y, 4)  //here i must specify the type Int,and this format can't work : reduceByKey((x, y) => x + y, 4)

这里我必须指定我的函数参数的类型Int ,如reduceByKey((x:Int,y:Int)=> x + y,4)当我使用PairDStreamFunctions .reduceByKey时,这种格式无法工作reduceByKey((x ,y)=> x + y,4)

另一方面,当我使用PairRDDFunctions.reduceByKey api时,它可以推断出类型,这里是代码:

val conf = new SparkConf()
val sc = new SparkContext(conf)
val rdd = sc.parallelize(List(
  "hi what"
  , "show you"
  , "matter how"
))
rdd.flatMap(_.split(" "))
  .map((_, 1))
  .reduceByKey((x, y) => x + y, 4)//in this code,scala compiler could infer the type of my function parameter (x,y) => x+y

当我使用PairRDDFunctions .reduceByKey时, reduceByKey((x,y)=> x + y,4)可以工作。 我真的不明白是什么让它与众不同?


I wonder why scala compiler couldn't infer a type of my function parameter when i using PairDStreamFunctions.reduceByKey,here is code:

val ssc = new StreamingContext(conf, Seconds(10))
ssc.checkpoint(".checkpoint")
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" "))
val wordCounts = words
  .map((_, 1))
  .reduceByKey((x: Int, y: Int) => x + y, 4)  //here i must specify the type Int,and this format can't work : reduceByKey((x, y) => x + y, 4)

here i must specify the type Int of my function parameter like reduceByKey((x: Int, y: Int) => x + y, 4) when i use PairDStreamFunctions.reduceByKey ,and this format couldn't work : reduceByKey((x, y) => x + y, 4).

On the other hand,when i use PairRDDFunctions.reduceByKey api,it can infer the type,here is the code:

val conf = new SparkConf()
val sc = new SparkContext(conf)
val rdd = sc.parallelize(List(
  "hi what"
  , "show you"
  , "matter how"
))
rdd.flatMap(_.split(" "))
  .map((_, 1))
  .reduceByKey((x, y) => x + y, 4)//in this code,scala compiler could infer the type of my function parameter (x,y) => x+y

When i use PairRDDFunctions.reduceByKey, reduceByKey((x, y) => x + y, 4) could work. I really don't understand what makes it different?


原文:https://stackoverflow.com/questions/49103223
更新时间:2022-04-25 08:04

最满意答案

我想到了。 在进入SSAS之前,必须指定日期列的数据类型并将其转换为SSIS中的日期类型。


i figured it out. Must specify and transform the datatype of the date column to Date type in SSIS before proceed to SSAS.

相关问答

更多
  • SSRS (SQL Server Reporting Services)是某些Microsoft SQL Server版本的一部分(具有高级服务的SQL Server Express以及非免费版本)。 它允许您处理服务器端报告( .rdl文件)。 Visual Studio中包含的ReportViewer控件允许您处理客户端报告( .rdlc文件)。 它不需要SQL Server。 由于这两个组件使用相同的报告引擎,因此RDL和RDLC使用相同的XML模式。 因此,您在网上找到的大多数SSRS资料也适用于R ...
  • 我想到了。 在进入SSAS之前,必须指定日期列的数据类型并将其转换为SSIS中的日期类型。 i figured it out. Must specify and transform the datatype of the date column to Date type in SSIS before proceed to SSAS.
  • 代替 $ server ='LOCALHOST:9000'; 给 $ server = '10 .0.0.155:9000'; instead of $server = 'LOCALHOST:9000'; give $server = '10.0.0.155:9000';
  • 一些包含概述的链接,但可能还有更多工具: http://www.ssas-info.com/analysis-services-client-tools-frontend http://ssas-info.com/forum/8-choosing-ssas-client-tools http://cwebbbi.wordpress.com/category/client-tools/ 当然 - 根据您所处的分析类型或市场,还有更具体的解决方案。 Some links that contain overvie ...
  • 你的表达应该使用SUM而不是count =SUM(IIF(Fields!Sales_Transaction.Value=0,0,1)) Your expression should use SUM instead of count =SUM(IIF(Fields!Sales_Transaction.Value=0,0,1))
  • 使用SQL Server Reporting Services 2008 R2,您可以使用查找功能从第二个数据集中查找数据: http : //blog.datainspirations.com/2010/03/19/sql-server-2008-r2-reporting-服务 - 看起来 - 看起来全方位 - 我/非常方便。 从较旧版本的SSRS中,您可以创建占据表中单元格的子报表:可以使用每行的不同参数调用子报表,从而有效地“连接”到第二个数据集。 http://technet.microsoft.c ...
  • 对于任何间隔,伪代码将是一种 int intervals[11]={0,10,20,30,40,50,60,70,80,90}; for(int i=1; i<11; i++) { if(x>=intervals[i-1] && x=(i-1)*10 && x
  • 我发现我遇到的问题是我试图返回的数据量太大而且还没有加载。 我正在更改我的xml以手动过滤数据,然后为报告提供多个数据集。 上面的表达式是正确的。 The problem I found that I was having was the amount of data that I was trying to return was to much and it all wasn't loaded yet. I am changing my xml to filter the data before hand ...
  • 如果您在SSRS中使用MDX查询构建器,则有一个“显示空单元格”的按钮... ...或者如果您手动构建MDX,只需删除NonEmpty函数... i actually end up using linked server with stored proc and it works like a charm and the performance is good because i basically filter the data from cube.
  • 没有理由不能在多个程序或不同的人中使用同一个表。 如果您只需要1000行进行查询,并且每秒运行多次,您可以考虑在查询中创建并填充它作为临时表,因为1000行只是增量数对引擎没有任何意义。 There is no reason you couldn't use the same table in multiple procedures or by different people. If you only need 1000 rows for you query and it doenst run multi ...

相关文章

更多

最新问答

更多
  • 您如何使用git diff文件,并将其应用于同一存储库的副本的本地分支?(How do you take a git diff file, and apply it to a local branch that is a copy of the same repository?)
  • 将长浮点值剪切为2个小数点并复制到字符数组(Cut Long Float Value to 2 decimal points and copy to Character Array)
  • OctoberCMS侧边栏不呈现(OctoberCMS Sidebar not rendering)
  • 页面加载后对象是否有资格进行垃圾回收?(Are objects eligible for garbage collection after the page loads?)
  • codeigniter中的语言不能按预期工作(language in codeigniter doesn' t work as expected)
  • 在计算机拍照在哪里进入
  • 使用cin.get()从c ++中的输入流中丢弃不需要的字符(Using cin.get() to discard unwanted characters from the input stream in c++)
  • No for循环将在for循环中运行。(No for loop will run inside for loop. Testing for primes)
  • 单页应用程序:页面重新加载(Single Page Application: page reload)
  • 在循环中选择具有相似模式的列名称(Selecting Column Name With Similar Pattern in a Loop)
  • System.StackOverflow错误(System.StackOverflow error)
  • KnockoutJS未在嵌套模板上应用beforeRemove和afterAdd(KnockoutJS not applying beforeRemove and afterAdd on nested templates)
  • 散列包括方法和/或嵌套属性(Hash include methods and/or nested attributes)
  • android - 如何避免使用Samsung RFS文件系统延迟/冻结?(android - how to avoid lag/freezes with Samsung RFS filesystem?)
  • TensorFlow:基于索引列表创建新张量(TensorFlow: Create a new tensor based on list of indices)
  • 企业安全培训的各项内容
  • 错误:RPC失败;(error: RPC failed; curl transfer closed with outstanding read data remaining)
  • C#类名中允许哪些字符?(What characters are allowed in C# class name?)
  • NumPy:将int64值存储在np.array中并使用dtype float64并将其转换回整数是否安全?(NumPy: Is it safe to store an int64 value in an np.array with dtype float64 and later convert it back to integer?)
  • 注销后如何隐藏导航portlet?(How to hide navigation portlet after logout?)
  • 将多个行和可变行移动到列(moving multiple and variable rows to columns)
  • 提交表单时忽略基础href,而不使用Javascript(ignore base href when submitting form, without using Javascript)
  • 对setOnInfoWindowClickListener的意图(Intent on setOnInfoWindowClickListener)
  • Angular $资源不会改变方法(Angular $resource doesn't change method)
  • 在Angular 5中不是一个函数(is not a function in Angular 5)
  • 如何配置Composite C1以将.m和桌面作为同一站点提供服务(How to configure Composite C1 to serve .m and desktop as the same site)
  • 不适用:悬停在悬停时:在元素之前[复制](Don't apply :hover when hovering on :before element [duplicate])
  • 常见的python rpc和cli接口(Common python rpc and cli interface)
  • Mysql DB单个字段匹配多个其他字段(Mysql DB single field matching to multiple other fields)
  • 产品页面上的Magento Up出售对齐问题(Magento Up sell alignment issue on the products page)