使用SLURM的openMPI作业进行分段错误(segmention fault using openMPI jobs with SLURM)
尝试使用slurm SBATCH作业或使用MPI超过infiniband的SRUN作业时出现问题。
安装了OpenMPI，如果我使用
mpirun -n 30 ./hello启动以下测试程序（称为hello ），mpirun -n 30 ./hello可以工作。// compilation: mpicc -o helloMPI helloMPI.c #include <mpi.h> #include <stdio.h> int main ( int argc, char * argv [] ) { int myrank, nproc; MPI_Init ( &argc, &argv ); MPI_Comm_size ( MPI_COMM_WORLD, &nproc ); MPI_Comm_rank ( MPI_COMM_WORLD, &myrank ); printf ( "hello from rank %d of %d\n", myrank, nproc ); MPI_Barrier ( MPI_COMM_WORLD ); MPI_Finalize (); return 0; }所以：
user@master:~/hello$ mpicc -o hello hello.c user@master:~/hello$ mpirun -n 30 ./hello -------------------------------------------------------------------------- [[5627,1],2]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: usNIC Host: master Another transport will be used instead, although this may result in lower performance. -------------------------------------------------------------------------- hello from rank 25 of 30 hello from rank 1 of 30 hello from rank 6 of 30 [...] hello from rank 17 of 30当我尝试通过SLURM启动它时，我得到这样的分段错误：
user@master:~/hello$ srun -n 20 ./hello [node05:01937] *** Process received signal *** [node05:01937] Signal: Segmentation fault (11) [node05:01937] Signal code: Address not mapped (1) [node05:01937] Failing at address: 0x30 [node05:01937] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fcf6bf7ecb0] [node05:01937] [ 1] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x244c6)[0x7fcf679b64c6] [node05:01937] [ 2] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x254cb)[0x7fcf679b74cb] [node05:01937] [ 3] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0xb1)[0x7fcf679b2141] [node05:01937] [ 4] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x10ad0)[0x7fcf679a2ad0] [node05:01937] [ 5] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_btl_base_select+0x114)[0x7fcf6c209b34] [node05:01937] [ 6] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x12)[0x7fcf67bca652] [node05:01937] [ 7] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_bml_base_init+0x69)[0x7fcf6c209359] [node05:01937] [ 8] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_pml_ob1.so(+0x5975)[0x7fcf65d1b975] [node05:01937] [ 9] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_pml_base_select+0x35c)[0x7fcf6c21a0bc] [node05:01937] [10] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(ompi_mpi_init+0x4ed)[0x7fcf6c1cb89d] [node05:01937] [11] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(MPI_Init+0x16b)[0x7fcf6c1eb56b] [node05:01937] [12] /home/user/hello/./hello[0x400826] [node05:01937] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fcf6bbd076d] [node05:01937] [14] /home/user/hello/./hello[0x400749] [node05:01937] *** End of error message *** [node05:01938] *** Process received signal *** [node05:01938] Signal: Segmentation fault (11) [node05:01938] Signal code: Address not mapped (1) [node05:01938] Failing at address: 0x30 [node05:01940] *** Process received signal *** [node05:01940] Signal: Segmentation fault (11) [node05:01940] Signal code: Address not mapped (1) [node05:01940] Failing at address: 0x30 [node05:01938] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f68b2e10cb0] [node05:01938] [ 1] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x244c6)[0x7f68ae8484c6] [node05:01938] [ 2] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x254cb)[0x7f68ae8494cb] [node05:01940] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f8af1d82cb0] [node05:01940] [ 1] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x244c6)[0x7f8aed7ba4c6] [node05:01940] [ 2] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x254cb)[0x7f8aed7bb4cb] [node05:01940] [ 3] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0xb1)[0x7f8aed7b6141] [node05:01940] [ 4] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x10ad0)[0x7f8aed7a6ad0] [node05:01938] [ 3] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0xb1)[0x7f68ae844141] [node05:01938] [ 4] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x10ad0)[0x7f68ae834ad0] [node05:01938] [ 5] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_btl_base_select+0x114)[0x7f68b309bb34] [node05:01938] [ 6] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x12)[0x7f68aea5c652] [node05:01940] [ 5] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_btl_base_select+0x114)[0x7f8af200db34] [node05:01940] [ 6] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x12)[0x7f8aed9ce652] [node05:01938] [ 7] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_bml_base_init+0x69)[0x7f68b309b359] [node05:01938] [ 8] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_pml_ob1.so(+0x5975)[0x7f68acbad975] [node05:01940] [ 7] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_bml_base_init+0x69)[0x7f8af200d359] [node05:01940] [ 8] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_pml_ob1.so(+0x5975)[0x7f8aebb1f975] [node05:01940] [ 9] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_pml_base_select+0x35c)[0x7f8af201e0bc] [node05:01938] [ 9] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_pml_base_select+0x35c)[0x7f68b30ac0bc] [node05:01938] [10] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(ompi_mpi_init+0x4ed)[0x7f68b305d89d] [node05:01940] [10] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(ompi_mpi_init+0x4ed)[0x7f8af1fcf89d] [node05:01938] [11] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(MPI_Init+0x16b)[0x7f68b307d56b] [node05:01938] [12] /home/user/hello/./hello[0x400826] [node05:01940] [11] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(MPI_Init+0x16b)[0x7f8af1fef56b] [node05:01940] [12] /home/user/hello/./hello[0x400826] [node05:01938] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f68b2a6276d] [node05:01938] [14] /home/user/hello/./hello[0x400749] [node05:01938] *** End of error message *** [node05:01940] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f8af19d476d] [node05:01940] [14] /home/user/hello/./hello[0x400749] [node05:01940] *** End of error message *** [node05:01939] *** Process received signal *** [node05:01939] Signal: Segmentation fault (11) [node05:01939] Signal code: Address not mapped (1) [node05:01939] Failing at address: 0x30 [...]etc有谁知道这是什么问题？ 我已经构建了具有Slurm支持的openMPI，并安装了相同版本的编译器和库，实际上所有的库都在安装在每个节点上的NFS共享中。
备注：
它应该使用infiniband，因为它已安装。 但是当我和mpirun一起玩时，我注意到了
[[5627,1],2]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: usNIC Host: cluster我想这意味着“没有通过infiniband”。 我已经安装了infiniband驱动程序，并在Infiniband上设置了IP。 Slurm配置为使用infiniband IP运行：它是正确的配置吗？
在此先感谢最好的问候
编辑：
我刚刚尝试使用MPICH2而不是openMPI编译它，它可以与SLURM一起使用。 所以问题可能与openMPI有关，而不是Slurm配置？
编辑2：实际上，我已经看到使用openMPI 1.6.5（而不是1.8）与SBATCH命令而不是SRUN我的脚本被执行（即它返回线程号，等级和主机。但它显示与openfabric供应商相关的警告和注册记忆的分配：
The OpenFabrics (openib) BTL failed to initialize while trying to allocate some locked memory. This typically can indicate that the memlock limits are set too low. For most HPC installations, the memlock limits should be set to "unlimited". The failure occured here: Local host: node05 OMPI source: btl_openib_component.c:1216 Function: ompi_free_list_init_ex_new() Device: mlx4_0 Memlock limit: 65536 You may need to consult with your system administrator to get this problem fixed. This FAQ entry on the Open MPI web site may also be helpful: http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There was an error initializing an OpenFabrics device. Local host: node05 Local device: mlx4_0 -------------------------------------------------------------------------- Hello world from process 025 out of 048, processor name node06 Hello world from process 030 out of 048, processor name node06 Hello world from process 032 out of 048, processor name node06 Hello world from process 046 out of 048, processor name node07 Hello world from process 031 out of 048, processor name node06 Hello world from process 041 out of 048, processor name node07 Hello world from process 034 out of 048, processor name node06 Hello world from process 044 out of 048, processor name node07 Hello world from process 033 out of 048, processor name node06 Hello world from process 045 out of 048, processor name node07 Hello world from process 026 out of 048, processor name node06 Hello world from process 043 out of 048, processor name node07 Hello world from process 024 out of 048, processor name node06 Hello world from process 038 out of 048, processor name node07 Hello world from process 014 out of 048, processor name node05 Hello world from process 027 out of 048, processor name node06 Hello world from process 036 out of 048, processor name node07 Hello world from process 019 out of 048, processor name node05 Hello world from process 028 out of 048, processor name node06 Hello world from process 040 out of 048, processor name node07 Hello world from process 023 out of 048, processor name node05 Hello world from process 042 out of 048, processor name node07 Hello world from process 018 out of 048, processor name node05 Hello world from process 039 out of 048, processor name node07 Hello world from process 021 out of 048, processor name node05 Hello world from process 047 out of 048, processor name node07 Hello world from process 037 out of 048, processor name node07 Hello world from process 015 out of 048, processor name node05 Hello world from process 035 out of 048, processor name node06 Hello world from process 020 out of 048, processor name node05 Hello world from process 029 out of 048, processor name node06 Hello world from process 016 out of 048, processor name node05 Hello world from process 017 out of 048, processor name node05 Hello world from process 022 out of 048, processor name node05 Hello world from process 012 out of 048, processor name node05 Hello world from process 013 out of 048, processor name node05 Hello world from process 000 out of 048, processor name node04 Hello world from process 001 out of 048, processor name node04 Hello world from process 002 out of 048, processor name node04 Hello world from process 003 out of 048, processor name node04 Hello world from process 006 out of 048, processor name node04 Hello world from process 009 out of 048, processor name node04 Hello world from process 011 out of 048, processor name node04 Hello world from process 004 out of 048, processor name node04 Hello world from process 007 out of 048, processor name node04 Hello world from process 008 out of 048, processor name node04 Hello world from process 010 out of 048, processor name node04 Hello world from process 005 out of 048, processor name node04 [node04:04390] 47 more processes have sent help message help-mpi-btl-openib.txt / init-fail-no-mem [node04:04390] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [node04:04390] 47 more processes have sent help message help-mpi-btl-openib.txt / error in device init我从中理解的是，a）v.1.6.5有更好的错误处理和b）我必须配置更多注册内存大小的openMPI和/或infiniband驱动程序。 我看到这个页面 ，显然我只需要修改openMPI的东西？ 我得测试一下......
I have a problem when trying tu use slurm SBATCH jobs or SRUN jobs with MPI over infiniband.
OpenMPI is installed, and if I launch the following test program (called hello) with
mpirun -n 30 ./helloit works.// compilation: mpicc -o helloMPI helloMPI.c #include <mpi.h> #include <stdio.h> int main ( int argc, char * argv [] ) { int myrank, nproc; MPI_Init ( &argc, &argv ); MPI_Comm_size ( MPI_COMM_WORLD, &nproc ); MPI_Comm_rank ( MPI_COMM_WORLD, &myrank ); printf ( "hello from rank %d of %d\n", myrank, nproc ); MPI_Barrier ( MPI_COMM_WORLD ); MPI_Finalize (); return 0; }so :
user@master:~/hello$ mpicc -o hello hello.c user@master:~/hello$ mpirun -n 30 ./hello -------------------------------------------------------------------------- [[5627,1],2]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: usNIC Host: master Another transport will be used instead, although this may result in lower performance. -------------------------------------------------------------------------- hello from rank 25 of 30 hello from rank 1 of 30 hello from rank 6 of 30 [...] hello from rank 17 of 30When I try to launch it through SLURM I get segmentation faults like this:
user@master:~/hello$ srun -n 20 ./hello [node05:01937] *** Process received signal *** [node05:01937] Signal: Segmentation fault (11) [node05:01937] Signal code: Address not mapped (1) [node05:01937] Failing at address: 0x30 [node05:01937] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fcf6bf7ecb0] [node05:01937] [ 1] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x244c6)[0x7fcf679b64c6] [node05:01937] [ 2] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x254cb)[0x7fcf679b74cb] [node05:01937] [ 3] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0xb1)[0x7fcf679b2141] [node05:01937] [ 4] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x10ad0)[0x7fcf679a2ad0] [node05:01937] [ 5] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_btl_base_select+0x114)[0x7fcf6c209b34] [node05:01937] [ 6] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x12)[0x7fcf67bca652] [node05:01937] [ 7] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_bml_base_init+0x69)[0x7fcf6c209359] [node05:01937] [ 8] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_pml_ob1.so(+0x5975)[0x7fcf65d1b975] [node05:01937] [ 9] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_pml_base_select+0x35c)[0x7fcf6c21a0bc] [node05:01937] [10] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(ompi_mpi_init+0x4ed)[0x7fcf6c1cb89d] [node05:01937] [11] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(MPI_Init+0x16b)[0x7fcf6c1eb56b] [node05:01937] [12] /home/user/hello/./hello[0x400826] [node05:01937] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fcf6bbd076d] [node05:01937] [14] /home/user/hello/./hello[0x400749] [node05:01937] *** End of error message *** [node05:01938] *** Process received signal *** [node05:01938] Signal: Segmentation fault (11) [node05:01938] Signal code: Address not mapped (1) [node05:01938] Failing at address: 0x30 [node05:01940] *** Process received signal *** [node05:01940] Signal: Segmentation fault (11) [node05:01940] Signal code: Address not mapped (1) [node05:01940] Failing at address: 0x30 [node05:01938] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f68b2e10cb0] [node05:01938] [ 1] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x244c6)[0x7f68ae8484c6] [node05:01938] [ 2] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x254cb)[0x7f68ae8494cb] [node05:01940] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f8af1d82cb0] [node05:01940] [ 1] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x244c6)[0x7f8aed7ba4c6] [node05:01940] [ 2] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x254cb)[0x7f8aed7bb4cb] [node05:01940] [ 3] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0xb1)[0x7f8aed7b6141] [node05:01940] [ 4] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x10ad0)[0x7f8aed7a6ad0] [node05:01938] [ 3] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0xb1)[0x7f68ae844141] [node05:01938] [ 4] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_btl_openib.so(+0x10ad0)[0x7f68ae834ad0] [node05:01938] [ 5] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_btl_base_select+0x114)[0x7f68b309bb34] [node05:01938] [ 6] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x12)[0x7f68aea5c652] [node05:01940] [ 5] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_btl_base_select+0x114)[0x7f8af200db34] [node05:01940] [ 6] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x12)[0x7f8aed9ce652] [node05:01938] [ 7] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_bml_base_init+0x69)[0x7f68b309b359] [node05:01938] [ 8] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_pml_ob1.so(+0x5975)[0x7f68acbad975] [node05:01940] [ 7] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_bml_base_init+0x69)[0x7f8af200d359] [node05:01940] [ 8] /opt/cluster/spool/openMPI/1.8/gcc/lib/openmpi/mca_pml_ob1.so(+0x5975)[0x7f8aebb1f975] [node05:01940] [ 9] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_pml_base_select+0x35c)[0x7f8af201e0bc] [node05:01938] [ 9] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(mca_pml_base_select+0x35c)[0x7f68b30ac0bc] [node05:01938] [10] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(ompi_mpi_init+0x4ed)[0x7f68b305d89d] [node05:01940] [10] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(ompi_mpi_init+0x4ed)[0x7f8af1fcf89d] [node05:01938] [11] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(MPI_Init+0x16b)[0x7f68b307d56b] [node05:01938] [12] /home/user/hello/./hello[0x400826] [node05:01940] [11] /opt/cluster/spool/openMPI/1.8/gcc/lib/libmpi.so.1(MPI_Init+0x16b)[0x7f8af1fef56b] [node05:01940] [12] /home/user/hello/./hello[0x400826] [node05:01938] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f68b2a6276d] [node05:01938] [14] /home/user/hello/./hello[0x400749] [node05:01938] *** End of error message *** [node05:01940] [13] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f8af19d476d] [node05:01940] [14] /home/user/hello/./hello[0x400749] [node05:01940] *** End of error message *** [node05:01939] *** Process received signal *** [node05:01939] Signal: Segmentation fault (11) [node05:01939] Signal code: Address not mapped (1) [node05:01939] Failing at address: 0x30 [...]etcDoes anyone know what is the problem? I have built openMPI with Slurm support, and installed the same version of compilers and libs, in fact all the libs are in a NFS share which is mounted on each node.
remarks:
It should use infiniband, as it is installed. But when I lauch openmpi with mpirun I notice the
[[5627,1],2]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: usNIC Host: clusterwhich I guess means "not running through infiniband". I have installed infiniband drivers, and set up IP over Infiniband. Slurm is configured to run with infiniband IPs : is it a right configuration?
Thanks in advance Best regards
EDIT :
I have just tried to compile it with MPICH2 instead of openMPI and it work with SLURM. So the problem is probably related to openMPI and not Slurm configuration?
EDIT 2: Actually, I have seen that using openMPI 1.6.5 (instead of 1.8) with SBATCH command instead of SRUN my script is executed (i.e. it returns the thread number, rank and host. But it shows warnings related to the openfabric vendor and allocation of registered memory:
The OpenFabrics (openib) BTL failed to initialize while trying to allocate some locked memory. This typically can indicate that the memlock limits are set too low. For most HPC installations, the memlock limits should be set to "unlimited". The failure occured here: Local host: node05 OMPI source: btl_openib_component.c:1216 Function: ompi_free_list_init_ex_new() Device: mlx4_0 Memlock limit: 65536 You may need to consult with your system administrator to get this problem fixed. This FAQ entry on the Open MPI web site may also be helpful: http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There was an error initializing an OpenFabrics device. Local host: node05 Local device: mlx4_0 -------------------------------------------------------------------------- Hello world from process 025 out of 048, processor name node06 Hello world from process 030 out of 048, processor name node06 Hello world from process 032 out of 048, processor name node06 Hello world from process 046 out of 048, processor name node07 Hello world from process 031 out of 048, processor name node06 Hello world from process 041 out of 048, processor name node07 Hello world from process 034 out of 048, processor name node06 Hello world from process 044 out of 048, processor name node07 Hello world from process 033 out of 048, processor name node06 Hello world from process 045 out of 048, processor name node07 Hello world from process 026 out of 048, processor name node06 Hello world from process 043 out of 048, processor name node07 Hello world from process 024 out of 048, processor name node06 Hello world from process 038 out of 048, processor name node07 Hello world from process 014 out of 048, processor name node05 Hello world from process 027 out of 048, processor name node06 Hello world from process 036 out of 048, processor name node07 Hello world from process 019 out of 048, processor name node05 Hello world from process 028 out of 048, processor name node06 Hello world from process 040 out of 048, processor name node07 Hello world from process 023 out of 048, processor name node05 Hello world from process 042 out of 048, processor name node07 Hello world from process 018 out of 048, processor name node05 Hello world from process 039 out of 048, processor name node07 Hello world from process 021 out of 048, processor name node05 Hello world from process 047 out of 048, processor name node07 Hello world from process 037 out of 048, processor name node07 Hello world from process 015 out of 048, processor name node05 Hello world from process 035 out of 048, processor name node06 Hello world from process 020 out of 048, processor name node05 Hello world from process 029 out of 048, processor name node06 Hello world from process 016 out of 048, processor name node05 Hello world from process 017 out of 048, processor name node05 Hello world from process 022 out of 048, processor name node05 Hello world from process 012 out of 048, processor name node05 Hello world from process 013 out of 048, processor name node05 Hello world from process 000 out of 048, processor name node04 Hello world from process 001 out of 048, processor name node04 Hello world from process 002 out of 048, processor name node04 Hello world from process 003 out of 048, processor name node04 Hello world from process 006 out of 048, processor name node04 Hello world from process 009 out of 048, processor name node04 Hello world from process 011 out of 048, processor name node04 Hello world from process 004 out of 048, processor name node04 Hello world from process 007 out of 048, processor name node04 Hello world from process 008 out of 048, processor name node04 Hello world from process 010 out of 048, processor name node04 Hello world from process 005 out of 048, processor name node04 [node04:04390] 47 more processes have sent help message help-mpi-btl-openib.txt / init-fail-no-mem [node04:04390] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [node04:04390] 47 more processes have sent help message help-mpi-btl-openib.txt / error in device initWhat I understand from that, is that a) v.1.6.5 has a better error handling and b) I have to configure openMPI and/or infiniband drivers with more registered memory size. I see this page and apparently I only need to modify openMPI stuff? I have to test it...
原文：https://stackoverflow.com/questions/23156560
更新时间：2022-03-25 08:03
最满意答案
你可能需要一些
libuv魔法才能让主 node.js / v8线程从另一个线程执行你的回调。 这将涉及：
一个uv_async_t句柄 ，用作主v8线程的唤醒调用：
extern uv_async_t async;一个
uv_async_init调用，uv_async_init绑定到V8默认循环：uv_async_init(uv_default_loop(), &async, async_cb_handler);以及一个事件处理程序，它将作用于v8主线程上的uvasync_t事件：
void async_cb_handler(uv_async_t *handle) { NotifInfo *notif; mutex::scoped_lock sl(zqueue_mutex); while (!zqueue.empty()) { notif = zqueue.front(); handleNotification(notif); delete notif; zqueue.pop(); } }最后，您还可能需要一个受互斥锁保护的队列，以便能够将一些数据从c ++插件线程传递到Node / V8：
extern mutex zqueue_mutex; extern std::queue<NotifInfo *> zqueue;当您的C ++线程中发生某些事情时，只需将一个新项目推送到受互斥锁保护的队列，然后调用
uv_async_send唤醒V8的默认事件循环（所谓的“主线程”）来处理该项目（然后可以调用您的Javascript）回调）void ozw_watcher_callback(OpenZWave::Notification const *cb, void *ctx) { NotifInfo *notif = new NotifInfo(); notif->type = cb->GetType(); notif->homeid = cb->GetHomeId(); ... mutex::scoped_lock sl(zqueue_mutex); zqueue.push(notif); uv_async_send(&async); }（代码片段取自OpenZWave的官方Node.JS插件 ）
You probably need a bit of
libuvmagic to get the main node.js/v8 thread to execute your callback from another thread. This will involve:
A uv_async_t handle which acts as the wake-up call for the main v8 thread:
extern uv_async_t async;A
uv_async_initcall which binds the uv_async_t to the V8 default loop:uv_async_init(uv_default_loop(), &async, async_cb_handler);And a event handler which will act on the uvasync_t event on the v8 main thread:
void async_cb_handler(uv_async_t *handle) { NotifInfo *notif; mutex::scoped_lock sl(zqueue_mutex); while (!zqueue.empty()) { notif = zqueue.front(); handleNotification(notif); delete notif; zqueue.pop(); } }Finally, you'll also probably need a mutex-protected queue to be able to pass some data from the c++ addon thread to Node/V8:
extern mutex zqueue_mutex; extern std::queue<NotifInfo *> zqueue;When something happens in your C++ thread, just push a new item to the mutex-protected queue, and call
uv_async_sendto wake up V8's default event loop (the so called 'main thread') to process the item (which can then call your Javascript callbacks)void ozw_watcher_callback(OpenZWave::Notification const *cb, void *ctx) { NotifInfo *notif = new NotifInfo(); notif->type = cb->GetType(); notif->homeid = cb->GetHomeId(); ... mutex::scoped_lock sl(zqueue_mutex); zqueue.push(notif); uv_async_send(&async); }(Code snippets taken from the official Node.JS addon for OpenZWave)
相关问答
更多-
如何在v8 Javascript中跨多个函数使用相同的上下文？(How to use the same context across multiple functions in v8 Javascript?)[2023-07-13]
我想通了自己。 错误是我每次都在所有功能中运行脚本。 相反，我应该在构造函数中运行一次，就像编译一样。 I figured out myself. The mistake was that I was running the script each time in all functions. Rather, I should run just once in the constructor like compiling. -
V8多线程功能(V8 Multithreaded function)[2023-03-20]
你可能需要一些libuv魔法才能让主 node.js / v8线程从另一个线程执行你的回调。 这将涉及： 一个uv_async_t句柄 ，用作主v8线程的唤醒调用： extern uv_async_t async; 一个uv_async_init调用， uv_async_init绑定到V8默认循环： uv_async_init(uv_default_loop(), &async, async_cb_handler); 以及一个事件处理程序，它将作用于v8主线程上的uvasync_t事件： v ... -
我看到2个问题： t.eval返回线程本身（从控制台输出中可以看到）。 如果提供了代码，则将代码执行的结果传递给回调 你需要在传递给eval的代码中使用addon，闭包语义在这里不起作用。 但看起来这是不可能的：在线程上下文中没有定义require。 我认为这是为了防止竞争条件问题。 另请参阅https://github.com/audreyt/node-webworker-threads/issues/15 I see 2 issues: t.eval returns thread itself (as ...
-
至于node.js，它不是并发的，而是异步的。 有一个线程和一个事件循环，它处理node.js中的所有IO。 node.js中有一些并发工具，主要围绕多个进程。 但是像所有node.js一样，它们还处于开发的早期阶段。 例如，您可以使用赋格来管理多个工作进程，或者您可以使用可能熟悉的Web Workers API使用node-worker 。 As for node.js, it's not concurrent but asynchronous. There's a single thread, and ...
-
v8::Primitive是v8::Value一个子类，所以对它进行反转是绝对好的。 代码不会编译，因为您无法将v8::Handle转换为v8::Local 。 只需更改callback_arguments的类型即可： v8::Handle
-
V8是否支持Unicode？(Does V8 have Unicode support?)[2023-06-24]
我不确定在问这个问题时是否就是这种情况，但目前V8 API有许多支持UTF-8，UTF-16和Latin-1编码文本的函数： https://github.com/v8/v8/blob/master/include/v8.h 创建新字符串对象的相关函数是： String :: NewFromUtf8（显然是UTF-8编码） String :: NewFromOneByte（Latin-1编码） String :: NewFromTwoByte（UTF-16编码） 或者，您可以避免复制字符串数据并构造引用现有 ... -
你试过v8 :: Value :: IsTypedArray（）吗？ Did you try v8::Value::IsTypedArray() ?
-
如果您按照README并使用带-v标志的node-gyp ，您可以看到编译如下所示： g++ '-DNODE_GYP_MODULE_NAME=addon' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-DBUILDING_NODE_EXTENSION' -I/home/yannick/.node-gyp/5.9.0/include/node -I/home/yannick/.node-gyp/5.9.0/src -I/home/yannick/.node- ...
-
问题是缺少生成隔离的代码 currentIsolate->Enter(); 所以正确的JNI功能是 void JSfunc() { v8::Isolate* currentIsolate = v8::Isolate::GetCurrent(); if(!currentIsolate) { currentIsolate = v8::Isolate::New(); currentIsolate->Enter(); } v8::HandleScop ...
-
如何更新我的函数以使用新的v8 FunctionTemplates？(How do I update my function to use the new v8 FunctionTemplates?)[2022-02-25]
SetTrainingAlgorithm签名是： void SetTrainingAlgorithm(v8::Local