从hbase启动读取数据,工作人员是否需要从远程驱动程序获取分区数据?(spark read data from hbase, did workers need to get paritions data from remote driver program?)
spark从hbase读取数据,例如// create rdd
val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result])
例如,hBaseRDD有5个分区,现在worker上的执行者获取分区数据进行计算,他们必须从远程驱动程序获取数据? (不喜欢从hdfs读取,每个工作者作为hadoop奴隶有hdfs文件复制)
spark read data from hbase,such as //create rdd
val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result])
for example, hBaseRDD has 5 partitions, now executor on worker get partition data to compute, they must get data from remote driver program? (not like read from hdfs, each worker as hadoop slave has the hdfs file replication)
原文:https://stackoverflow.com/questions/45364077
最满意答案
首先,您必须检查您的网络是否可用
wifiManager = (WifiManager) context.getSystemService(Context.WIFI_SERVICE); if (!wifiManager.isWifiEnabled()) { wifiManager.setWifiEnabled(true); } wifiResultsReceiver = new BroadcastReceiver() { @Override public void onReceive(Context context, Intent intent) { context.unregisterReceiver(wifiResultsReceiver); List<ScanResult> wifiList; wifiList = wifiManager.getScanResults(); for (int i = 0; i < wifiList.size(); i++) { String[] networkInfo = wifiList.get(i).toString().split(","); if (networkInfo[0].trim().equals(YOUR_SSID)) { //Check if your ssid is available & connect(); } } }; context.registerReceiver(wifiResultsReceiver, new IntentFilter(WifiManager.SCAN_RESULTS_AVAILABLE_ACTION)); wifiManager.startScan();
然后,如果网络可用,您可以连接到您的SSID(这是我的配置之一):
public static WifiConfiguration getConnectionConfig(Context context, String ssidName) { Validator.validateArgument(context, "Context can't be null"); WifiManager wifiManager = (WifiManager) context.getSystemService(context.WIFI_SERVICE); if (!wifiManager.isWifiEnabled()) wifiManager.setWifiEnabled(true); WifiConfiguration wifiConfig = new WifiConfiguration(); wifiConfig.SSID = ssidName; List<WifiConfiguration> curentList = wifiManager.getConfiguredNetworks(); if (curentList != null && !curentList.isEmpty()) { curentList.removeAll(Collections.singleton(null)); for (int i = 0; i < curentList.size(); i++) { WifiConfiguration item = curentList.get(i); if (ssidName.equals(item.SSID)) { wifiManager.removeNetwork(item.networkId); } } } wifiConfig.status = WifiConfiguration.Status.DISABLED; wifiConfig.priority = 100; wifiConfig.allowedKeyManagement.set(WifiConfiguration.KeyMgmt.NONE); wifiConfig.allowedProtocols.set(WifiConfiguration.Protocol.RSN); wifiConfig.allowedProtocols.set(WifiConfiguration.Protocol.WPA); wifiConfig.allowedAuthAlgorithms.clear(); wifiConfig.allowedPairwiseCiphers.set(WifiConfiguration.PairwiseCipher.CCMP); wifiConfig.allowedPairwiseCiphers.set(WifiConfiguration.PairwiseCipher.TKIP); wifiConfig.allowedGroupCiphers.set(WifiConfiguration.GroupCipher.WEP40); wifiConfig.allowedGroupCiphers.set(WifiConfiguration.GroupCipher.WEP104); wifiConfig.allowedGroupCiphers.set(WifiConfiguration.GroupCipher.CCMP); wifiConfig.allowedGroupCiphers.set(WifiConfiguration.GroupCipher.TKIP); return wifiConfig; } private void connect() { wifiManager = (WifiManager) context.getSystemService(Context.WIFI_SERVICE); int networkID = wifiManager.addNetwork(getConnectionConfig(context, YOUR_SSID)); wifiManager.disconnect(); wifiManager.enableNetwork(networkID, true); wifiManager.reconnect(); }
不要忘记在AndroidManifest上添加权限
PS:三星设备在4.4及早期版本的SSID扫描中存在一些问题,请小心。 PPS:对于身份验证,在我打开的情况下,您必须使用AP连接的特定配置。 问候
First of all you must check if available your network
wifiManager = (WifiManager) context.getSystemService(Context.WIFI_SERVICE); if (!wifiManager.isWifiEnabled()) { wifiManager.setWifiEnabled(true); } wifiResultsReceiver = new BroadcastReceiver() { @Override public void onReceive(Context context, Intent intent) { context.unregisterReceiver(wifiResultsReceiver); List<ScanResult> wifiList; wifiList = wifiManager.getScanResults(); for (int i = 0; i < wifiList.size(); i++) { String[] networkInfo = wifiList.get(i).toString().split(","); if (networkInfo[0].trim().equals(YOUR_SSID)) { //Check if your ssid is available & connect(); } } }; context.registerReceiver(wifiResultsReceiver, new IntentFilter(WifiManager.SCAN_RESULTS_AVAILABLE_ACTION)); wifiManager.startScan();
Then if network is available, you can connect to your SSID(this is one of my config):
public static WifiConfiguration getConnectionConfig(Context context, String ssidName) { Validator.validateArgument(context, "Context can't be null"); WifiManager wifiManager = (WifiManager) context.getSystemService(context.WIFI_SERVICE); if (!wifiManager.isWifiEnabled()) wifiManager.setWifiEnabled(true); WifiConfiguration wifiConfig = new WifiConfiguration(); wifiConfig.SSID = ssidName; List<WifiConfiguration> curentList = wifiManager.getConfiguredNetworks(); if (curentList != null && !curentList.isEmpty()) { curentList.removeAll(Collections.singleton(null)); for (int i = 0; i < curentList.size(); i++) { WifiConfiguration item = curentList.get(i); if (ssidName.equals(item.SSID)) { wifiManager.removeNetwork(item.networkId); } } } wifiConfig.status = WifiConfiguration.Status.DISABLED; wifiConfig.priority = 100; wifiConfig.allowedKeyManagement.set(WifiConfiguration.KeyMgmt.NONE); wifiConfig.allowedProtocols.set(WifiConfiguration.Protocol.RSN); wifiConfig.allowedProtocols.set(WifiConfiguration.Protocol.WPA); wifiConfig.allowedAuthAlgorithms.clear(); wifiConfig.allowedPairwiseCiphers.set(WifiConfiguration.PairwiseCipher.CCMP); wifiConfig.allowedPairwiseCiphers.set(WifiConfiguration.PairwiseCipher.TKIP); wifiConfig.allowedGroupCiphers.set(WifiConfiguration.GroupCipher.WEP40); wifiConfig.allowedGroupCiphers.set(WifiConfiguration.GroupCipher.WEP104); wifiConfig.allowedGroupCiphers.set(WifiConfiguration.GroupCipher.CCMP); wifiConfig.allowedGroupCiphers.set(WifiConfiguration.GroupCipher.TKIP); return wifiConfig; } private void connect() { wifiManager = (WifiManager) context.getSystemService(Context.WIFI_SERVICE); int networkID = wifiManager.addNetwork(getConnectionConfig(context, YOUR_SSID)); wifiManager.disconnect(); wifiManager.enableNetwork(networkID, true); wifiManager.reconnect(); }
Don't forget add permissions on AndroidManifest
P.S: Samsung devices has some problem with SSID scanning in 4.4 and early version, be carefully. P.P.S: for auth you must use specific configuration for AP connection in my case it open. Regards
相关问答
更多-
Android中的iOS NSNotificationCenter相当于?(Equivalent of iOS NSNotificationCenter in Android?)[2022-11-12]
在Android中,没有像ios那样的中央通知中心。 但你基本上可以使用Observable和Observer对象来实现你的任务。 你可以像下面这样定义一个类,只需修改它以供单例使用,并为同时使用添加同步,但其想法是相同的: public class ObservingService { HashMapobservables; public ObservingService() { observables = new HashMap ... -
什么是Android等效的CaptiveNetwork API(iOS)?(What is the Android equivalent of the CaptiveNetwork API (iOS)?)[2020-12-06]
首先,您必须检查您的网络是否可用 wifiManager = (WifiManager) context.getSystemService(Context.WIFI_SERVICE); if (!wifiManager.isWifiEnabled()) { wifiManager.setWifiEnabled(true); } wifiResultsReceiver = new BroadcastReceiver() { ... -
你不应该这样做。 要创建费用,您必须使用您的Secret API密钥。 如果您的Secret API密钥到达您的应用程序,那么有人可以在他的终端检索它,然后使用它代表您进行任何API调用。 这意味着他们可以创建费用,退款,转账等。 您必须在服务器端执行此操作,并且您的iOS(或Android)应用程序永远不应该拥有您的Secret API密钥。 You should not do this. To create a charge you have to use your Secret API key. If ...
-
感谢Christopher(http://stackoverflow.com/questions/1993471/android-can-i-mute-currently-playing-audio-applications),我引用了: Android上的音频处理在一段时间内会非常糟糕。 这些API非常奇怪,文档记录不完善,并且在版本之间不断变化/不赞成/中断。 即使是AudioManager代码也有FIXMEs。 无论如何,Android中有多种类型的流(音乐,通知,电话等),应用程序旨在选择合适的播放 ...
-
尝试将视图高约束更改为0,这样依赖于此视图的其他视图应该像Android线性布局中一样 更新1 从iOS 9.0 UIStackView可用,其作用与android中的LinearLayout相同。 要启用此功能,您可以在Interface Builder中选择2个或更多视图,在底部可以单击堆栈视图按钮: Try to change view high constraint to 0, this way other views which depends on this one should act lik ...
-
我使用NSURLConnection解决了这个问题。 并获得该代表的响应数据。 - (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response I solved the problem using NSURLConnection. And get response data by this delegate. - (void)connection:(NSURLConnection * ...
-
您可以在没有Cocoapods的情况下使用适用于iOS的Google Play游戏服务SDK。 请参阅https://developers.google.com/games/services/ios/quickstart#option_2_manual_installation 出于好奇,为什么不使用Cocoapods来管理依赖? You can use the Google Play Game Services SDK for iOS without Cocoapods. See https://deve ...
-
是的,你需要覆盖onLowMemory()函数。 yes it's onLowMemory() function you need to overwrite.
-
它对Android和iOS都是免费的。 我使用Mobile Vision的条形码API,从未遇到任何问题。 适用于Android和iOS的实施示例 It's free for both, Android & iOS. I use the Barcode API from Mobile Vision and never had any issues. Implementation Samples for Android & iOS
-
我相信你需要viewWillAppear方法: override func viewWillAppear(_ animated: Bool) { super.viewWillAppear(animated) //your code here } 每次在视图显示之前调用它。 因此,当第一次显示视图时,它也会被调用。 如果要在第一次调用viewWillAppear避免运行代码,则必须添加一个标志属性并检查它是否先前已设置过。 I believe you need viewWillAppear ...