Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Call stack upon MPI routine error
From: Ralph Castain (rhc_at_[hidden])
Date: 2014-04-04 15:53:41


Running out of file descriptors sounds likely here - if you have 20 procs/node, and fully connect, each node will see 20*220 connections (you don't use tcp between procs on the same node), with each connection requiring a file descriptor.

On Apr 4, 2014, at 11:26 AM, Vince Grimes <tom.grimes_at_[hidden]> wrote:

> Dear all:
>
> The subject heading is a little misleading because this is in response to part of that original contact. I tried the first two suggestions below (disabling eager DMA and using tcp btl), but to no avail. In all cases I am running over 20 12-core nodes through SGE. In the first case, I get the errors:
>
> ***
> [[30430,1],234][btl_openib_component.c:3492:handle_wc] from compute-1-18.local to: compute-6-10 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 2c41e80 opcode 128 vendor error 244 qp_idx 0
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: compute-4-13.local
> PID: 22356
>
> This process may still be running and/or consuming resources.
>
> --------------------------------------------------------------------------
> [compute-6-1.local:22658] 2 more processes have sent help message help-odls-default.txt / odls-default:could-not-kill
> [compute-6-1.local:22658] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
> --------------------------------------------------------------------------
> ***
>
> The first error is at the same place as before ([btl_openib_component.c:3492:handle_wc]) and the message is only slightly different (LP -> HP).
>
> For the second suggestion, using tcp btl, I got a whole load of these:
>
> ***
> [compute-3-1.local][[20917,1],74][btl_tcp_endpoint.c:653:mca_btl_tcp_endpoint_complete_connect] connect() to 10.7.36.244 failed: Connection timed out (110)
> ***
>
> there are 1826 "Connection timed out" errors at an earlier spot in the code than in the case above. I checked iptables and there is no reason the connection would have been refused. Is it possible I'm out of file descriptors (because sockets count as files)? `ulimit -n` yields 1024.
>
> T. Vince Grimes, Ph.D.
> CCC System Administrator
>
> Texas Tech University
> Dept. of Chemistry and Biochemistry (10A)
> Box 41061
> Lubbock, TX 79409-1061
>
> (806) 834-0813 (voice); (806) 742-1289 (fax)
>
> On 03/22/2014 11:00 AM, users-request_at_[hidden] wrote:
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Fri, 21 Mar 2014 20:16:31 +0000
>> From: Joshua Ladd <joshual_at_[hidden]>
>> To: Open MPI Users <users_at_[hidden]>
>> Subject: Re: [OMPI users] Call stack upon MPI routine error
>> Message-ID:
>> <8EDEBDDE2C39D447A738659597BBB63A3ED121AB_at_[hidden]>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Hi, Vince
>>
>> Couple of ideas off the top of my head:
>>
>> 1. Try disabling eager RDMA. Eager RDMA can consume significant resources: "-mca btl_openib_use_eager_rdma 0"
>>
>> 2. Try using the TCP BTL - is the error still present?
>>
>> 3. Try the poor man's debugger - print the pid and hostname of the process when and then put a while(1) at btl_openib_component.c:3492 so that the process will hang when it hits this error. Hop over to the node and attach to the hung process. You can move up the call stack from here.
>>
>> Best,
>>
>> Josh
>>
>> -----Original Message-----
>> From: users [mailto:users-bounces_at_[hidden]] On Behalf Of Vince Grimes
>> Sent: Friday, March 21, 2014 3:52 PM
>> To: users_at_[hidden]
>> Subject: [OMPI users] Call stack upon MPI routine error
>>
>> OpenMPI folks:
>>
>> I have mentioned before a problem with an in-house code (ScalIT) that generates the error message
>>
>> [[31552,1],84][btl_openib_component.c:3492:handle_wc] from compute-4-5.local to: compute-4-13 error polling LP CQ with status LOCAL QP OPERATION ERROR status number 2 for wr_id 246f300 opcode 128 vendor error 107 qp_idx 0
>>
>> at a specific, reproducible point. It was suggested that the error could be due to memory problems, such as the amount of registered memory. I have already corrected the amount of registered memory per the URLs that were given to me. My question today is two-fold:
>>
>> First, is it possible that ScalIT uses so much memory that there is no memory to register for IB communications? ScalIT is very memory-intensive and has to run distributed just to get a large matrix in memory (split between nodes).
>>
>> Second, is there a way to trap that error so I can see the call stack, showing the MPI function called and exactly where in the code the error was generated?
>>
>> --
>> T. Vince Grimes, Ph.D.
>> CCC System Administrator
>>
>> Texas Tech University
>> Dept. of Chemistry and Biochemistry (10A) Box 41061 Lubbock, TX 79409-1061
>>
>> (806) 834-0813 (voice); (806) 742-1289 (fax)
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users