Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] About MPI_TAG_UB
From: Sébastien Boisvert (sebastien.boisvert.3_at_[hidden])
Date: 2012-09-28 09:08:54


Hello,

My application has 191 MPI tags allocated with allocateMessageTagHandle, so
7 bits is not enough.

Indeed, tags can be valued from 0 to 2147483647 inclusively with this MPI_TAG_UB
value in Open-MPI. I misused the returned pointer.

In Open-MPI, MPI_ANY_TAG is -1. I removed the boundary check in MPI_Isend and
MPI_Recv to allow values from -2147483648 to 2147483647 inclusively.

  https://raw.github.com/sebhtml/patches/master/ompi-1.6.2-ray-4096-routing.patch

As long as my tag is not MPI_ANY_TAG, I guess it should work fine although
it is not MPI-compliant. I will test that.

On 28/09/12 03:50 AM, Iliev, Hristo wrote:
> Hello,
>
> MPI_TAG_UB in Open MPI is INT_MAX == 2^31-1 == 2147483647. The value of
> 17438272 (0x10A1640) is a bit strange for MPI_TAG_UB. I would rather say
> that it is the value of a pointer to someplace in the heap, i.e. you have
> missed the fact that the attribute value as returned by MPI_Comm_get_attr /
> MPI_Attr_get is a pointer to the actual value (for MPI_TAG_UB it is a
> pointer to int).
>
> MPI_TAG_UB is predefined attribute and according to §8.1.2 of the MPI
> standard its value cannot be changed by the user.
>
> You have to find another solution, e.g. reduce the tag space to 7 bits or
> put the routing info inside the message payload.
>
> Best regards,
> Hristo Iliev
> --
> Hristo Iliev, Ph.D. -- High Performance Computing
> RWTH Aachen University, Center for Computing and Communication
> Rechen- und Kommunikationszentrum der RWTH Aachen
> Seffenter Weg 23, D 52074 Aachen (Germany)
>
>> -----Original Message-----
>> From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]]
>> On Behalf Of Sébastien Boisvert
>> Sent: Friday, September 28, 2012 1:22 AM
>> To: users_at_[hidden]
>> Subject: [OMPI users] About MPI_TAG_UB
>>
>> Hello,
>>
>> I am running Ray (a distributed software in genomics) with Open-MPI on
>> 2048 processes and everything runs fine. Ray has a any-to-any
>> communication pattern.
>> To avoid using too much memory, I implemented a virtual message router.
>>
>> Without the virtual message router, I get messages like these:
>>
>> [cp2558][[30209,1],0][connect/btl_openib_connect_oob.c:490:qp_create_o
>> ne] error creating qp errno says Cannot allocate memory
>>
>> We did some tests on the Cray XE6 on 4096 processing elements (4096 MPI
>> ranks) without the virtual message router and everything runs fine as is.
> So
>> using the virtual message router is not required.
>>
>> The real message tag, the real source and the real destination are stored
> in
>> the MPI tag. I know, this is ugly, but it works. I can not store this
> information
>> in the message buffer because the buffer can be NULL.
>>
>> bits 0 to 7: tag (8 bits, values from 0 to 255, 256 possible values) bits
> 8 to 19:
>> true source (12 bits, values from 0 to 4095, 4096 possible values) bits 20
> to 31:
>> true destination (12 bits, values from 0 to 4095, 4096 possible values)
>>
>> Without the virtual router, my code is compliant with the fact that
>> MPI_Comm_get_attr(MPI_COMM_WORLD, MPI_TAG_UB,...) is at least
>> 32767 (my tags are <= 255).
>>
>>
>> When I try jobs with 4096 processes with the virtual message router, I get
> the
>> error:
>>
>> MPI_ERR_TAG: invalid tag.
>>
>> Without the virtual message router I get:
>>
>> [cp2558][[30209,1],0][connect/btl_openib_connect_oob.c:490:qp_create_o
>> ne] error creating qp errno says Cannot allocate memory
>>
>> With Open-MPI 1.5.4, the upper bound is 17438272 (at least in our build).
>> That explains MPI_ERR_TAG.
>>
>>
>> My 2 questions:
>>
>> 1. Is there a better way to store routing information ?
>>
>> 2. Can I create my own communicator and set its MPI_TAG_UB to whatever I
>> want ?
>>
>>
>> Thanks !
>>
>> ***
>> Sébastien Boisvert
>> Ph.D. student
>> http://boisvert.info/
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users