Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] BTL add procs errors
From: Sylvain Jeaugey (sylvain.jeaugey_at_[hidden])
Date: 2010-06-02 04:43:42


Couldn't explain it better. Thanks Jeff for the summary !

On Tue, 1 Jun 2010, Jeff Squyres wrote:

> On May 31, 2010, at 10:27 AM, Ralph Castain wrote:
>
>> Just curious - your proposed fix sounds exactly like what was done in
>> the OPAL SOS work. Are you therefore proposing to use SOS to provide a
>> more informational status return?
>
> No, I think Sylvain's talking about slightly modifying the existing
> mechanism:
>
> 1. Return OMPI_SUCCESS: bml then obeys whatever is in the connectivity
> bitmask -- even if the bitmask indicates that this BTL can't talk to
> anyone.
>
> 2. Return != OMPI_SUCCESS: treat the problem as a fatal error.
>
> I think Sylvain's point is that OMPI_SUCCESS can be returned for
> non-fatal errors if a BTL just wants to be ignored. In such cases, the
> BTL can just set its connectivity mask to 0. This will allow OMPI to
> continue the job.
>
> For example, if verbs is borked (e.g., can't create CQ's), it can return
> a connectivity mask of 0 and OMPI_SUCCESS. The BML is then free to fail
> over to some other BTL.
>
> But if a malloc() fails down in some BTL, then the job is hosed anyway
> -- so why not return != OMPI_SUCCESS and try to abort cleanly?
>
> For sites that want to treat verbs failures as fatal, we can add a new
> MCA param either in the openib BTL that says "treat all init failures as
> fatal to the job" or perhaps a new MCA param in R2 that says "if the
> connectivity map for BTL <list> is empty, abort the job". Or something
> like that.
>
>> If so, then it would seem the only real dispute here is: is there -any-
>> condition whereby a given BTL should have the authority to tell OMPI to
>> terminate an application, even if other BTLs could still function?
>
> I think his cited example was if malloc() fails.
>
> I could see some sites wanting to abort if their high-speed network was
> down (e.g., MX or openib BTLs failed to init) -- they wouldn't want OMPI
> to fail over to TCP. The flip side of this argument is that the
> sysadmin could set "btl = ^tcp" in the system file, and then if
> openib/mx fails, the BML will abort because some peers won't be
> reachable.
>
>> I understand that the current code may not yet support that operation,
>> but I do believe that was the intent of the design. So only the case
>> where -all- BTLs say "I can't do it" would result in termination.
>> Rather than change that design, I believe the intent is to work towards
>> completing that implementation. In the interim, it would seem most
>> sensible to me that we add an MCA param that specifies the termination
>> behavior (i.e., attempt to continue or terminate on first fatal BTL
>> error).
>
> Agreed.
>
> I think that there are multiple different exit conditions from a BTL
> init:
>
> 1. BTL succeeded in initializing, and some peers are reachable 2. BTL
> succeeded in initializing, and no peers are reachable 3. BTL failed to
> initialize, but that failure is localized to the BTL (e.g., openib
> failed to create a CQ) 4. BTL failed to initialize, and the error is
> global in nature (e.g., malloc() fail)
>
> I think it might be a site-specific decision as to whether to abort the
> job for condition 3 or not. Today we default to not failing and pair
> that with an indirect method of failing (i.e., setting btl=^tcp).
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>