Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2005-08-10 05:11:36


Thanks for finding this, George!

On Aug 10, 2005, at 12:37 AM, George Bosilca wrote:

> I run all the ex-Pallas test and the same error happens. We try to
> malloc
> 0 bytes and we hang somewhere. Let me explain what I found. First of
> all,
> most of the tests seems to work perfectly (at least with the PTL/BTL I
> was
> able to run: sm, tcp, mx). The deadlock as well as the memory
> allocation
> problem happens in the reduce_scatter operation.
>
> Problem 1: allocating 0 bytes
> - it's not a datatype problem. The datatype return the correct extent,
> true_extent, lb. The problem is that we miss one case in the
> collective
> communications. How about the case when the user do a reduce_scatter
> with all the counts set to zero ? We check if they are greater than
> zero
> and it's the case. Then we add them together and as expected a sum of
> zero is zero. So in the coll_basic_reduce_scatter line 79 we will
> allocate zero bytes because the extent and the true_extent of the
> MPI_FLOAT datatype are equal and (count - 1) is -1 !!! There is a
> simple
> fix for this problem, if count == 0 then free_buffer should be set to
> NULL (as we don't send or receive anything in this buffer it will
> just
> work fine) at the PTL/PML level.
> - the same error can happens on the reduce function if the count is
> zero.
> I will protect this function too.
>
> Problem 2: hanging
> - somehow a strange optimization get inside the scatterv function. In
> the
> case where the sender has to send zero bytes it completly skip the
> send
> operation. But the receiver still expect to get a message. Anyway,
> this
> optimization is not correct, all messages have to be send. I know
> that
> it can (slightly) increase the time for the collective but it give
> us a
> simple way of checking the correctness of the global communication
> (as
> the PML handle the truncate case). Patch is on the way.
>
> Once these 2 problems corrected we pass all the Pallas MPI1 tests. I
> run
> the tests with the PML ob1, teg and uniq and the PTL/BTL sm, tcp, gm
> (PTL)
> and mx(PTL) with 2 and 8 processes.
>
> george.
>
> PS: the patches will be commited soon.
>
>> On Aug 9, 2005, at 1:53 PM, Galen Shipman wrote:
>>
>> Hi Sridhar,
>>
>> I have committed changes that allow you to set the debg verbosity,
>>
>> OMPI_MCA_btl_base_debug
>> 0 - no debug output
>> 1 - standard debug output
>> 2 - very verbose debug output
>>
>> Also we have run the Pallas tests and are not able to reproduce your
>> failures. We do see a warning in
>> the Reduce test but it does not hang and runs to completion. Attached
>> is a simple ping pong program,
>> try running this and let us know the results.
>>
>> Thanks,
>>
>> Galen
>>
>>
>>
>> <mpi-ping.c>
>>
>> On Aug 9, 2005, at 8:15 AM, Sridhar Chirravuri wrote:
>>
>>
>> The same kind of output while running Pallas "pingpong" test.
>>
>> -Sridhar
>>
>> -----Original Message-----
>> From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]]
>> On
>> Behalf Of Sridhar Chirravuri
>> Sent: Tuesday, August 09, 2005 7:44 PM
>> To: Open MPI Developers
>> Subject: Re: [O-MPI devel] Fwd: Regarding MVAPI Component in Open MPI
>>
>>
>> I have run sendrecv function in Pallas but it failed to run it. Here
>> is
>> the output
>>
>> [root_at_micrompi-2 SRC_PMB]# mpirun -np 2 PMB-MPI1 sendrecv
>> Could not join a running, existing universe
>> Establishing a new one named: default-universe-5097
>> [0,1,1][btl_mvapi.c:130:mca_btl_mvapi_del_procs] Stub
>> [0,1,1][btl_mvapi.c:130:mca_btl_mvapi_del_procs] Stub
>>
>>
>> [0,1,0][btl_mvapi.c:130:mca_btl_mvapi_del_procs] Stub
>>
>> [0,1,0][btl_mvapi.c:130:mca_btl_mvapi_del_procs] Stub
>>
>> [0,1,0][btl_mvapi_endpoint.c:542:mca_btl_mvapi_endpoint_send]
>> Connection
>> to endpoint closed ... connecting ...
>> [0,1,0][btl_mvapi_endpoint.c:318:mca_btl_mvapi_endpoint_start_connect]
>> Initialized High Priority QP num = 263177, Low Priority QP num =
>> 263178,
>> LID = 785
>>
>> [0,1,0][btl_mvapi_endpoint.c:190:
>> mca_btl_mvapi_endpoint_send_connect_req
>> ] Sending High Priority QP num = 263177, Low Priority QP num = 263178,
>> LID = 785[0,1,0][btl_mvapi_endpoint.c:542:mca_btl_mvapi_endpoint_send]
>> Connection to endpoint closed ... connecting ...
>> [0,1,0][btl_mvapi_endpoint.c:318:mca_btl_mvapi_endpoint_start_connect]
>> Initialized High Priority QP num = 263179, Low Priority QP num =
>> 263180,
>> LID = 786
>>
>> [0,1,0][btl_mvapi_endpoint.c:190:
>> mca_btl_mvapi_endpoint_send_connect_req
>> ] Sending High Priority QP num = 263179, Low Priority QP num = 263180,
>> LID = 786#---------------------------------------------------
>> #    PALLAS MPI Benchmark Suite V2.2, MPI-1 part
>> #---------------------------------------------------
>> # Date       : Tue Aug  9 07:11:25 2005
>> # Machine    : x86_64# System     : Linux
>> # Release    : 2.6.9-5.ELsmp
>> # Version    : #1 SMP Wed Jan 5 19:29:47 EST 2005
>>
>> #
>> # Minimum message length in bytes:   0
>> # Maximum message length in bytes:   4194304
>> #
>> # MPI_Datatype                   :   MPI_BYTE
>> # MPI_Datatype for reductions    :   MPI_FLOAT
>> # MPI_Op                         :   MPI_SUM
>> #
>> #
>>
>> # List of Benchmarks to run:
>>
>> # Sendrecv
>> [0,1,1][btl_mvapi_endpoint.c:368:
>> mca_btl_mvapi_endpoint_reply_start_conn
>> ect] Initialized High Priority QP num = 263177, Low Priority QP num =
>> 263178,  LID = 777
>>
>> [0,1,1][btl_mvapi_endpoint.c:266:
>> mca_btl_mvapi_endpoint_set_remote_info]
>> Received High Priority QP num = 263177, Low Priority QP num 263178, 
>> LID
>> = 785
>>
>> [0,1,1][btl_mvapi_endpoint.c:756:mca_btl_mvapi_endpoint_qp_init_query]
>> Modified to init..Qp
>> 7080096[0,1,1][btl_mvapi_endpoint.c:791:
>> mca_btl_mvapi_endpoint_qp_init_q
>> uery] Modified to RTR..Qp
>> 7080096[0,1,1][btl_mvapi_endpoint.c:814:
>> mca_btl_mvapi_endpoint_qp_init_q
>> uery] Modified to RTS..Qp 7080096
>>
>> [0,1,1][btl_mvapi_endpoint.c:756:mca_btl_mvapi_endpoint_qp_init_query]
>> Modified to init..Qp 7240736
>> [0,1,1][btl_mvapi_endpoint.c:791:mca_btl_mvapi_endpoint_qp_init_query]
>> Modified to RTR..Qp
>> 7240736[0,1,1][btl_mvapi_endpoint.c:814:
>> mca_btl_mvapi_endpoint_qp_init_q
>> uery] Modified to RTS..Qp 7240736
>> [0,1,1][btl_mvapi_endpoint.c:190:
>> mca_btl_mvapi_endpoint_send_connect_req
>> ] Sending High Priority QP num = 263177, Low Priority QP num = 263178,
>> LID = 777
>> [0,1,0][btl_mvapi_endpoint.c:266:
>> mca_btl_mvapi_endpoint_set_remote_info]
>> Received High Priority QP num = 263177, Low Priority QP num 263178, 
>> LID
>> = 777
>> [0,1,0][btl_mvapi_endpoint.c:756:mca_btl_mvapi_endpoint_qp_init_query]
>> Modified to init..Qp 7081440
>> [0,1,0][btl_mvapi_endpoint.c:791:mca_btl_mvapi_endpoint_qp_init_query]
>> Modified to RTR..Qp 7081440
>> [0,1,0][btl_mvapi_endpoint.c:814:mca_btl_mvapi_endpoint_qp_init_query]
>> Modified to RTS..Qp 7081440
>> [0,1,0][btl_mvapi_endpoint.c:756:mca_btl_mvapi_endpoint_qp_init_query]
>> Modified to init..Qp 7241888
>> [0,1,0][btl_mvapi_endpoint.c:791:mca_btl_mvapi_endpoint_qp_init_query]
>> Modified to RTR..Qp
>> 7241888[0,1,0][btl_mvapi_endpoint.c:814:
>> mca_btl_mvapi_endpoint_qp_init_q
>> uery] Modified to RTS..Qp 7241888
>> [0,1,1][btl_mvapi_component.c:523:mca_btl_mvapi_component_progress]
>> Got
>> a recv completion
>>
>>
>> Thanks
>> -Sridhar
>>
>>
>>
>>
>> -----Original Message-----
>> From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]]
>> On
>> Behalf Of Brian Barrett
>> Sent: Tuesday, August 09, 2005 7:35 PM
>> To: Open MPI Developers
>> Subject: Re: [O-MPI devel] Fwd: Regarding MVAPI Component in Open MPI
>>
>> On Aug 9, 2005, at 8:48 AM, Sridhar Chirravuri wrote:
>>
>>
>> Does r6774 has lot of changes that are related to 3rd generation
>> point-to-point? I am trying to run some benchmark tests (ex:
>> pallas) with Open MPI stack and just want to compare the
>> performance figures with MVAPICH 095 and MVAPICH 092.
>>
>> In order to use 3rd generation p2p communication, I have added the
>> following line in the /openmpi/etc/openmpi-mca-params.conf
>>
>> pml=ob1
>>
>> I also exported (as double check) OMPI_MCA_pml=ob1.
>>
>> Then, I have tried running on the same machine. My machine has got
>> 2 processors.
>>
>> Mpirun -np 2 ./PMB-MPI1
>>
>> I still see the following lines
>>
>> Request for 0 bytes (coll_basic_reduce_scatter.c, 79)
>> Request for 0 bytes (coll_basic_reduce.c, 193)
>> Request for 0 bytes (coll_basic_reduce_scatter.c, 79)
>> Request for 0 bytes (coll_basic_reduce.c, 193)
>>
>>
>> These errors are coming from the collective routines, not the PML/BTL
>> layers.  It looks like the reduction codes are trying to call malloc
>> (0), which doesn't work so well.  We'll take a look as soon as we
>> can.  In the mean time, can you just not run the tests that call the
>> reduction collectives?
>>
>> Brian
>>
>>
>> -- 
>>    Brian Barrett
>>    Open MPI developer
>>    http://www.open-mpi.org/
>>
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>>
>>
>
> "We must accept finite disappointment, but we must never lose infinite
> hope."
> Martin Luther King
>
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>

-- 
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/