Spark的HBase客户端无法使用SASL在ZooKeeper中进行身份验证(HBase client for Spark cannot be authenticated in ZooKeeper using SASL)
我正在处理Spark中的HBase表(EMR,在Yarn模式下)。 其实,PySpark--我认为这不重要。 我通过HBase集群之外的单独Thrift服务调用HBase。
它看起来像我能够连接到Thrift服务器,但我与ZooKeeper有一些问题(因为错误点我到ZooKeeper端口2181)。
为什么会发生这种情况,我该如何解决这个问题?
17/08/02 20:21:31 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp 17/08/02 20:21:31 INFO ZooKeeper: Client environment:java.compiler=<NA> 17/08/02 20:21:31 INFO ZooKeeper: Client environment:os.name=Linux 17/08/02 20:21:31 INFO ZooKeeper: Client environment:os.arch=amd64 17/08/02 20:21:31 INFO ZooKeeper: Client environment:os.version=4.4.35-33.55.amzn1.x86_64 17/08/02 20:21:31 INFO ZooKeeper: Client environment:user.name=hadoop 17/08/02 20:21:31 INFO ZooKeeper: Client environment:user.home=/home/hadoop 17/08/02 20:21:31 INFO ZooKeeper: Client environment:user.dir=/home/hadoop/data 17/08/02 20:21:31 INFO ZooKeeper: Initiating client connection, connectString=thrift-internal.production.k8s.prod.node.io:2181 sessionTimeout=180000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@2818bc0e 17/08/02 20:21:31 INFO ClientCnxn: Opening socket connection to server ip-172-23-115-152.us-west-2.compute.internal/172.23.115.152:2181. Will not attempt to authenticate using SASL (unknown error)
I'm processing HBase tables from Spark (EMR, in Yarn mode). Actually, PySpark - I don't think it is important. I call HBase through separate Thrift service from outside of the HBase cluster.
It looks like I was able to connect to the Thrift servers but I have some issue with ZooKeeper (because of the error point me to ZooKeeper port 2181).
Why does that happen and how can I fix that?
17/08/02 20:21:31 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp 17/08/02 20:21:31 INFO ZooKeeper: Client environment:java.compiler=<NA> 17/08/02 20:21:31 INFO ZooKeeper: Client environment:os.name=Linux 17/08/02 20:21:31 INFO ZooKeeper: Client environment:os.arch=amd64 17/08/02 20:21:31 INFO ZooKeeper: Client environment:os.version=4.4.35-33.55.amzn1.x86_64 17/08/02 20:21:31 INFO ZooKeeper: Client environment:user.name=hadoop 17/08/02 20:21:31 INFO ZooKeeper: Client environment:user.home=/home/hadoop 17/08/02 20:21:31 INFO ZooKeeper: Client environment:user.dir=/home/hadoop/data 17/08/02 20:21:31 INFO ZooKeeper: Initiating client connection, connectString=thrift-internal.production.k8s.prod.node.io:2181 sessionTimeout=180000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@2818bc0e 17/08/02 20:21:31 INFO ClientCnxn: Opening socket connection to server ip-172-23-115-152.us-west-2.compute.internal/172.23.115.152:2181. Will not attempt to authenticate using SASL (unknown error)
原文:https://stackoverflow.com/questions/45471743
最满意答案
从pip 1.3起 ,有一个
pip show
命令。$ pip show Jinja2 --- Name: Jinja2 Version: 2.7.3 Location: /path/to/virtualenv/lib/python2.7/site-packages Requires: markupsafe
在旧版本中,点击
pip freeze
和grep
应该做得很好。$ pip freeze | grep Jinja2 Jinja2==2.7.3
As of pip 1.3, there is a
pip show
command.$ pip show Jinja2 --- Name: Jinja2 Version: 2.7.3 Location: /path/to/virtualenv/lib/python2.7/site-packages Requires: markupsafe
In older versions,
pip freeze
andgrep
should do the job nicely.$ pip freeze | grep Jinja2 Jinja2==2.7.3
相关问答
更多-
安装一个软件包并用pip写入requirements.txt(Install a package and write to requirements.txt with pip)[2023-11-21]
要获取版本信息,您可以在安装后选择使用pip freeze。 这是一个应该做你所要求的功能: pip_install_save() { package_name=$1 requirements_file=$2 if [[ -z $requirements_file ]] then requirements_file='./requirements.txt' fi pip install $package_name && pip freeze | ... -
你可以这样做: pip -V或pip --version You can do this: pip -V or: pip --version
-
你可以做: $ pip install "package>=0.2,<0.3" 并且pip将寻找最佳匹配,假设版本至少为0.2,小于0.3。 这也适用于pip 需求文件 。 请参阅PEP 440中的版本说明符的完整详细信息。 You can do: $ pip install "package>=0.2,<0.3" And pip will look for the best match, assuming the version is at least 0.2, and less than 0.3. ...
-
从pip 1.3起 ,有一个pip show命令。 $ pip show Jinja2 --- Name: Jinja2 Version: 2.7.3 Location: /path/to/virtualenv/lib/python2.7/site-packages Requires: markupsafe 在旧版本中,点击pip freeze和grep应该做得很好。 $ pip freeze | grep Jinja2 Jinja2==2.7.3 As of pip 1.3, there is a p ...
-
首先,我看到你想要做的两件事情。 由于您已经安装了版本,您应该卸载当前现有的驱动程序或使用pip install -I MySQL_python==1.2.2 但是,很快就会发现这不行。 如果你看一下pip的安装日志,或者如果你做一个pip install -Iv MySQL_python==1.2.2你会发现PyPI URL链接不适用于MySQL_python v1.2.2。 您可以在这里验证: http : //pypi.python.org/pypi/MySQL-python/1.2.2 由于sour ...
-
你的pip正在赶上3.5版本。 您必须专门安装2.7版本的pip才能使代码正常工作。 这就是它的完成方式: $ sudo apt-get install python2-pip $ sudo pip2 install crc16 Your pip is catching the 3.5 version. You must specifically install pip for 2.7 version for your code to work. This is how it's done: $ sudo ...
-
如果软件包未在pypi上更新,则无法使用pip安装软件包名称来获取最新版本。 如果你绝对想要最新版本,你可以直接从git安装它 git clone
cd进入目录并安装软件包: sudo python setup.py install If a package is not updated on pypi, you cannot pip install packagename to get the latest version. If you absolutely ... -
谁更新了pip包版本?(Who updates pip package versions?)[2022-07-15]
可以使用PIP下载的所有包实际上都托管在Python包索引上 。 Python组织与项目维护人员合作来托管项目。 All packages that can be downloaded with PIP are actually hosted on the Python Package Index. The Python organization collaborates with project maintainers to host the projects. -
Pillow库的可导入名称是PIL因为它是原始PIL的分支。 做就是了 from PIL import * 或者甚至更好,只导入你真正需要的名字 - 看看这个SO线程 。 The importable name of the Pillow library is PIL because it is a fork of the original PIL. Just do from PIL import * Or even better, only import those names you actuall ...
-
这是因为操作系统附带预先确定的存储库列表,如果软件更新,作者必须更新其存储库中的软件,以便您拥有最新版本。 pip能够自行更新,因此您会注意到当前版本与存储库中提供的版本之间的差异。 进一步阅读这里 。 示例:如果您正在运行Ubuntu xenial(16.04LTS),那么在测试Ubuntu版本时,pip的版本是8.1.1-2: all 。 如果您将运行Ubuntu 17.04(Zesty Zapus)并执行sudo apt install pip您将获得9.0.1-2: all 。 有关各种Ubuntu ...