Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] BTL add procs errors
From: Sylvain Jeaugey (sylvain.jeaugey_at_[hidden])
Date: 2010-06-02 04:43:42


Couldn't explain it better. Thanks Jeff for the summary !

On Tue, 1 Jun 2010, Jeff Squyres wrote:

> On May 31, 2010, at 10:27 AM, Ralph Castain wrote:
>
>> Just curious - your proposed fix sounds exactly like what was done in
>> the OPAL SOS work. Are you therefore proposing to use SOS to provide a
>> more informational status return?
>
> No, I think Sylvain's talking about slightly modifying the existing
> mechanism:
>
> 1. Return OMPI_SUCCESS: bml then obeys whatever is in the connectivity
> bitmask -- even if the bitmask indicates that this BTL can't talk to
> anyone.
>
> 2. Return != OMPI_SUCCESS: treat the problem as a fatal error.
>
> I think Sylvain's point is that OMPI_SUCCESS can be returned for
> non-fatal errors if a BTL just wants to be ignored. In such cases, the
> BTL can just set its connectivity mask to 0. This will allow OMPI to
> continue the job.
>
> For example, if verbs is borked (e.g., can't create CQ's), it can return
> a connectivity mask of 0 and OMPI_SUCCESS. The BML is then free to fail
> over to some other BTL.
>
> But if a malloc() fails down in some BTL, then the job is hosed anyway
> -- so why not return != OMPI_SUCCESS and try to abort cleanly?
>
> For sites that want to treat verbs failures as fatal, we can add a new
> MCA param either in the openib BTL that says "treat all init failures as
> fatal to the job" or perhaps a new MCA param in R2 that says "if the
> connectivity map for BTL <list> is empty, abort the job". Or something
> like that.
>
>> If so, then it would seem the only real dispute here is: is there -any-
>> condition whereby a given BTL should have the authority to tell OMPI to
>> terminate an application, even if other BTLs could still function?
>
> I think his cited example was if malloc() fails.
>
> I could see some sites wanting to abort if their high-speed network was
> down (e.g., MX or openib BTLs failed to init) -- they wouldn't want OMPI
> to fail over to TCP. The flip side of this argument is that the
> sysadmin could set "btl = ^tcp" in the system file, and then if
> openib/mx fails, the BML will abort because some peers won't be
> reachable.
>
>> I understand that the current code may not yet support that operation,
>> but I do believe that was the intent of the design. So only the case
>> where -all- BTLs say "I can't do it" would result in termination.
>> Rather than change that design, I believe the intent is to work towards
>> completing that implementation. In the interim, it would seem most
>> sensible to me that we add an MCA param that specifies the termination
>> behavior (i.e., attempt to continue or terminate on first fatal BTL
>> error).
>
> Agreed.
>
> I think that there are multiple different exit conditions from a BTL
> init:
>
> 1. BTL succeeded in initializing, and some peers are reachable 2. BTL
> succeeded in initializing, and no peers are reachable 3. BTL failed to
> initialize, but that failure is localized to the BTL (e.g., openib
> failed to create a CQ) 4. BTL failed to initialize, and the error is
> global in nature (e.g., malloc() fail)
>
> I think it might be a site-specific decision as to whether to abort the
> job for condition 3 or not. Today we default to not failing and pair
> that with an indirect method of failing (i.e., setting btl=^tcp).
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>