node017.23692env IPATH_DISABLE_MMAP_MALLOC Disable mmap for malloc() => NO node017.23692env IPATH_NO_CPUAFFINITY Prevent PSM from setting affinity => NO node017.23692env IPATH_UNIT Device Unit number (-1 autodetects) => -1 node017.23692env PSM_DEVICES Ordered list of PSM-level devices => shm,ipath (default was self,shm,ipath) node017.23692psmi_parse_devices: PSM Device allocation order: amsh,ips node017.23692env PSM_MEMORY Memory usage mode (normal or large) => normal node017.23692env PSM_SHAREDCONTEXTS Enable shared contexts => YES (default was YES) node017.23692ipath_setaffinity: PORT_INFO returned unit_id=0/1,port=1/4,hwports=4,subport=0/0,nproc=8 node017.23692ipath_setaffinity: Set CPU affinity to 0, port 0:1:0 (1 active chips) node017.23692ipath_userinit: Driver is not QLogic-built node017.23692ipath_userinit: Runtime flags are 0x46, explicit mallopt mmap disable in malloc is off node017.23692psmi_port_open: Opened port 1.0 on device /dev/ipath (LID=14,epid=e0001,flags=46) node017.23692env PSM_RCVTHREAD Recv thread flags (0 disables thread) => 0x1 node017:1.0.env PSM_MQ_SENDREQS_MAX Max num of isend requests in flight => 1048576 node017:1.0.env PSM_MQ_RECVREQS_MAX Max num of irecv requests in flight => 1048576 node017:1.0.env PSM_MQ_RNDV_IPATH_THRESH ipath eager-to-rendezvous switchover => 64000 node017:1.0.env PSM_MQ_RNDV_SHM_THRESH shm eager-to-rendezvous switchover => 16000 node017:1.0.ips_spio_init: PIO copy uses forced ordering node017:1.0.env PSM_TID Tid proto flags (0 disables protocol) => 0x1 node017:1.0.ips_protoexp_init: Tid control message settings: timeout min=200us/max=1000us, interrupt when trying attempt #2 node017:1.0.ips_proto_init: Tid error control: warning every 30 secs, fatal error after 250 tid errors node017:1.0.ips_proto_init: Ethernet Host IP=10.141.0.17 and PID=23692 node017:1.0.psmi_shm_attach: Registered as master to key /psm_shm.d999e196-868e-c6e6-0d4a-bc2c78de85f1 node017:1.0.psmi_shm_attach: Mapped shm control object at 0x2aaaab25a000 node017:1.0.psmi_shm_attach: Mapped and initalized shm object control page at 0x2aaaab25a000,size=4096 node017:1.0.psmi_shm_attach: Grabbed shmidx 0 node017:1.0.amsh_init_segment: Grew shared segment for 1 procs, size=5.93 MB node017:1.0.am_remap_segment: shm segment remap from 0x2aaaab25a000..4096 to 0x2aaab26b3000..6217728 (relocated=YES) node017:1.0.ips_ptl_pollintr: Enabled communication thread on URG packets node017.23691env IPATH_DISABLE_MMAP_MALLOC Disable mmap for malloc() => NO node017.23691env IPATH_NO_CPUAFFINITY Prevent PSM from setting affinity => NO node017.23691env IPATH_UNIT Device Unit number (-1 autodetects) => -1 node017.23691env PSM_DEVICES Ordered list of PSM-level devices => shm,ipath (default was self,shm,ipath) node017.23691psmi_parse_devices: PSM Device allocation order: amsh,ips node017.23691env PSM_MEMORY Memory usage mode (normal or large) => normal node017.23691env PSM_SHAREDCONTEXTS Enable shared contexts => YES (default was YES) node017.23691ipath_setaffinity: PORT_INFO returned unit_id=0/1,port=2/4,hwports=4,subport=0/0,nproc=8 node017.23691ipath_setaffinity: Set CPU affinity to 1, port 0:2:0 (1 active chips) node017.23691ipath_userinit: Driver is not QLogic-built node017.23691ipath_userinit: Runtime flags are 0x46, explicit mallopt mmap disable in malloc is off node017.23691psmi_port_open: Opened port 2.0 on device /dev/ipath (LID=14,epid=e0002,flags=46) node017.23691env PSM_RCVTHREAD Recv thread flags (0 disables thread) => 0x1 node017:2.0.env PSM_MQ_SENDREQS_MAX Max num of isend requests in flight => 1048576 node017:2.0.env PSM_MQ_RECVREQS_MAX Max num of irecv requests in flight => 1048576 node017:2.0.env PSM_MQ_RNDV_IPATH_THRESH ipath eager-to-rendezvous switchover => 64000 node017:2.0.env PSM_MQ_RNDV_SHM_THRESH shm eager-to-rendezvous switchover => 16000 node017:2.0.ips_spio_init: PIO copy uses forced ordering node017:2.0.env PSM_TID Tid proto flags (0 disables protocol) => 0x1 node017:2.0.ips_protoexp_init: Tid control message settings: timeout min=200us/max=1000us, interrupt when trying attempt #2 node017:2.0.ips_proto_init: Tid error control: warning every 30 secs, fatal error after 250 tid errors node017:2.0.ips_proto_init: Ethernet Host IP=10.141.0.17 and PID=23691 node017:2.0.psmi_shm_attach: Registered as slave to key /psm_shm.d999e196-868e-c6e6-0d4a-bc2c78de85f1 node017:2.0.psmi_shm_attach: Mapped shm control object at 0x2aaaab25a000 node017:2.0.psmi_shm_attach: Slave synchronized object control page at 0x2aaaab25a000, size=4096 node017:2.0.psmi_shm_attach: Grabbed shmidx 1 node017:2.0.amsh_init_segment: Grew shared segment for 2 procs, size=11.86 MB node017:2.0.am_remap_segment: shm segment remap from 0x2aaaab25a000..4096 to 0x2aaab26b3000..12431360 (relocated=YES) node017.23694env IPATH_DISABLE_MMAP_MALLOC Disable mmap for malloc() => NO node017.23694env IPATH_NO_CPUAFFINITY Prevent PSM from setting affinity => NO node017:2.0.ips_ptl_pollintr: Enabled communication thread on URG packets node017.23694env IPATH_UNIT Device Unit number (-1 autodetects) => -1 node017.23694env PSM_DEVICES Ordered list of PSM-level devices => shm,ipath (default was self,shm,ipath) node017.23694psmi_parse_devices: PSM Device allocation order: amsh,ips node017.23694env PSM_MEMORY Memory usage mode (normal or large) => normal node017.23694env PSM_SHAREDCONTEXTS Enable shared contexts => YES (default was YES) node017.23694ipath_setaffinity: PORT_INFO returned unit_id=0/1,port=3/4,hwports=4,subport=0/0,nproc=8 node017.23694ipath_setaffinity: Set CPU affinity to 2, port 0:3:0 (1 active chips) node017.23698env IPATH_DISABLE_MMAP_MALLOC Disable mmap for malloc() => NO node017.23698env IPATH_NO_CPUAFFINITY Prevent PSM from setting affinity => NO node017.23698env IPATH_UNIT Device Unit number (-1 autodetects) => -1 node017.23698env PSM_DEVICES Ordered list of PSM-level devices => shm,ipath (default was self,shm,ipath) node017.23698psmi_parse_devices: PSM Device allocation order: amsh,ips node017.23698env PSM_MEMORY Memory usage mode (normal or large) => normal node017.23698env PSM_SHAREDCONTEXTS Enable shared contexts => YES (default was YES) node017.23698ipath_setaffinity: PORT_INFO returned unit_id=0/1,port=4/4,hwports=4,subport=0/0,nproc=8 node017.23694ipath_userinit: Driver is not QLogic-built node017.23694ipath_userinit: Runtime flags are 0x46, explicit mallopt mmap disable in malloc is off node017.23698ipath_setaffinity: Set CPU affinity to 3, port 0:4:0 (1 active chips) node017.23694psmi_port_open: Opened port 3.0 on device /dev/ipath (LID=14,epid=e0003,flags=46) node017.23694env PSM_RCVTHREAD Recv thread flags (0 disables thread) => 0x1 node017:3.0.env PSM_MQ_SENDREQS_MAX Max num of isend requests in flight => 1048576 node017:3.0.env PSM_MQ_RECVREQS_MAX Max num of irecv requests in flight => 1048576 node017:3.0.env PSM_MQ_RNDV_IPATH_THRESH ipath eager-to-rendezvous switchover => 64000 node017:3.0.env PSM_MQ_RNDV_SHM_THRESH shm eager-to-rendezvous switchover => 16000 node017:3.0.ips_spio_init: PIO copy uses forced ordering node017.23698ipath_userinit: Driver is not QLogic-built node017.23698ipath_userinit: Runtime flags are 0x46, explicit mallopt mmap disable in malloc is off node017.23698psmi_port_open: Opened port 4.0 on device /dev/ipath (LID=14,epid=e0004,flags=46) node017.23698env PSM_RCVTHREAD Recv thread flags (0 disables thread) => 0x1 node017.23696env IPATH_DISABLE_MMAP_MALLOC Disable mmap for malloc() => NO node017:4.0.env PSM_MQ_SENDREQS_MAX Max num of isend requests in flight => 1048576 node017.23696env IPATH_NO_CPUAFFINITY Prevent PSM from setting affinity => NO node017.23696env IPATH_UNIT Device Unit number (-1 autodetects) => -1 node017.23696env PSM_DEVICES Ordered list of PSM-level devices => shm,ipath (default was self,shm,ipath) node017.23696psmi_parse_devices: PSM Device allocation order: amsh,ips node017.23696env PSM_MEMORY Memory usage mode (normal or large) => normal node017:3.0.env PSM_TID Tid proto flags (0 disables protocol) => 0x1 node017.23696env PSM_SHAREDCONTEXTS Enable shared contexts => YES (default was YES) node017.23696ipath_userinit: assign_port command failed: Device or resource busy node017:4.0.env PSM_MQ_RECVREQS_MAX Max num of irecv requests in flight => 1048576 node017.23696psmi_port_open: /dev/ipath open failed: 25 (Device or resource busy) [node017:23696] Open MPI failed to open a PSM endpoint: No free InfiniPath contexts available on /dev/ipath [node017:23696] Error in psm_ep_open (error No free ports could be obtained) node017:4.0.env PSM_MQ_RNDV_IPATH_THRESH ipath eager-to-rendezvous switchover => 64000 node017:4.0.env PSM_MQ_RNDV_SHM_THRESH shm eager-to-rendezvous switchover => 16000 node017:4.0.ips_spio_init: PIO copy uses forced ordering node017:3.0.ips_protoexp_init: Tid control message settings: timeout min=200us/max=1000us, interrupt when trying attempt #2 node017:3.0.ips_proto_init: Tid error control: warning every 30 secs, fatal error after 250 tid errors node017:4.0.env PSM_TID Tid proto flags (0 disables protocol) => 0x1 node017:3.0.ips_proto_init: Ethernet Host IP=10.141.0.17 and PID=23694 node017:4.0.ips_protoexp_init: Tid control message settings: timeout min=200us/max=1000us, interrupt when trying attempt #2 node017:4.0.ips_proto_init: Tid error control: warning every 30 secs, fatal error after 250 tid errors node017:3.0.psmi_shm_attach: Registered as slave to key /psm_shm.d999e196-868e-c6e6-0d4a-bc2c78de85f1 node017:3.0.psmi_shm_attach: Mapped shm control object at 0x2aaaab25a000 node017:3.0.psmi_shm_attach: Slave synchronized object control page at 0x2aaaab25a000, size=4096 node017:3.0.psmi_shm_attach: Grabbed shmidx 2 node017:3.0.amsh_init_segment: Grew shared segment for 3 procs, size=17.78 MB node017:3.0.am_remap_segment: shm segment remap from 0x2aaaab25a000..4096 to 0x2aaab26b3000..18644992 (relocated=YES) node017:4.0.ips_proto_init: Ethernet Host IP=10.141.0.17 and PID=23698 node017:4.0.psmi_shm_attach: Registered as slave to key /psm_shm.d999e196-868e-c6e6-0d4a-bc2c78de85f1 node017:4.0.psmi_shm_attach: Mapped shm control object at 0x2aaaab25a000 node017:4.0.psmi_shm_attach: Slave synchronized object control page at 0x2aaaab25a000, size=4096 node017:4.0.ips_ptl_pollintr: Enabled communication thread on URG packets node017.23695env IPATH_DISABLE_MMAP_MALLOC Disable mmap for malloc() => NO node017.23695env IPATH_NO_CPUAFFINITY Prevent PSM from setting affinity => NO node017.23695env IPATH_UNIT Device Unit number (-1 autodetects) => -1 node017.23695env PSM_DEVICES Ordered list of PSM-level devices => shm,ipath (default was self,shm,ipath) node017.23695psmi_parse_devices: PSM Device allocation order: amsh,ips node017.23695env PSM_MEMORY Memory usage mode (normal or large) => normal node017.23695env PSM_SHAREDCONTEXTS Enable shared contexts => YES (default was YES) node017.23695ipath_userinit: assign_port command failed: Device or resource busy node017.23695psmi_port_open: /dev/ipath open failed: 25 (Device or resource busy) [node017:23695] Open MPI failed to open a PSM endpoint: No free InfiniPath contexts available on /dev/ipath [node017:23695] Error in psm_ep_open (error No free ports could be obtained) node017.23697env IPATH_DISABLE_MMAP_MALLOC Disable mmap for malloc() => NO node017.23697env IPATH_NO_CPUAFFINITY Prevent PSM from setting affinity => NO node017.23697env IPATH_UNIT Device Unit number (-1 autodetects) => -1 node017.23697env PSM_DEVICES Ordered list of PSM-level devices => shm,ipath (default was self,shm,ipath) node017.23697psmi_parse_devices: PSM Device allocation order: amsh,ips node017.23697env PSM_MEMORY Memory usage mode (normal or large) => normal node017.23693env IPATH_DISABLE_MMAP_MALLOC Disable mmap for malloc() => NO node017.23697env PSM_SHAREDCONTEXTS Enable shared contexts => YES (default was YES) node017.23693env IPATH_NO_CPUAFFINITY Prevent PSM from setting affinity => NO node017.23693env IPATH_UNIT Device Unit number (-1 autodetects) => -1 node017.23697ipath_userinit: assign_port command failed: Device or resource busy node017.23693env PSM_DEVICES Ordered list of PSM-level devices => shm,ipath (default was self,shm,ipath) node017.23693psmi_parse_devices: PSM Device allocation order: amsh,ips node017.23693env PSM_MEMORY Memory usage mode (normal or large) => normal node017.23697psmi_port_open: /dev/ipath open failed: 25 (Device or resource busy) [node017:23697] Open MPI failed to open a PSM endpoint: No free InfiniPath contexts available on /dev/ipath [node017:23697] Error in psm_ep_open (error No free ports could be obtained) node017.23693env PSM_SHAREDCONTEXTS Enable shared contexts => YES (default was YES) node017.23693ipath_userinit: assign_port command failed: Device or resource busy node017.23693psmi_port_open: /dev/ipath open failed: 25 (Device or resource busy) [node017:23693] Open MPI failed to open a PSM endpoint: No free InfiniPath contexts available on /dev/ipath [node017:23693] Error in psm_ep_open (error No free ports could be obtained) node017:4.0.psmi_shm_attach: Grabbed shmidx 3 node017:4.0.amsh_init_segment: Grew shared segment for 4 procs, size=23.71 MB node017:4.0.am_remap_segment: shm segment remap from 0x2aaaab25a000..4096 to 0x2aaab26b3000..24858624 (relocated=YES) node017:3.0.ips_ptl_pollintr: Enabled communication thread on URG packets