Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Performance difference on OpenMPI, IntelMPI and ScaliMPI
From: Torgny Faxen (faxen_at_[hidden])
Date: 2009-08-06 08:48:49


Pasha,
se attached file.

I have traced how MPI_IPROBE is called and also managed to significantly
reduce the number of calls to MPI_IPROBE. Unfortunately this only
resulted in the program spending time in other routines. Basically the
code runs through a number of timesteps and after each timestep all
slave nodes wait for the master to give go ahead for the next step and
this is were a lot of time is being spent. Either a load inbalance or
just waiting for all MPI_BSEND:s to complete or something else.

I am kind of stuck right now and will have to do some more
investigations . Strange that this works so much better using Scali MPI.

Regards / Torgny

Pavel Shamis (Pasha) wrote:
>
>> However, setting:
>> -mca btl_openib_eager_limit 65536
>> gave a 15% improvement so OpenMPI is now down to 326 (from previous
>> 376 seconds). Still a lot more than ScaliMPI with 214 seconds.
> Can you please run ibv_devinfo on one of compute nodes ? It is
> interesting to know what kind of IB HW you have on our cluster.
>
> Pasha
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

-- 
---------------------------------------------------------
   Torgny Faxén		
   National Supercomputer Center
   Linköping University	
   S-581 83 Linköping
   Sweden	
   Email:faxen_at_[hidden]
   Telephone: +46 13 285798 (office) +46 13 282535  (fax)
   http://www.nsc.liu.se
---------------------------------------------------------

hca_id: mlx4_0
        fw_ver: 2.5.000
        node_guid: 001e:0bff:ff4c:1bf4
        sys_image_guid: 001e:0bff:ff4c:1bf7
        vendor_id: 0x02c9
        vendor_part_id: 25418
        hw_ver: 0xA0
        board_id: HP_09D0000001
        phys_port_cnt: 2
                port: 1
                        state: active (4)
                        max_mtu: 2048 (4)
                        active_mtu: 2048 (4)
                        sm_lid: 1
                        port_lid: 132
                        port_lmc: 0x00

                port: 2
                        state: down (1)
                        max_mtu: 2048 (4)
                        active_mtu: 2048 (4)
                        sm_lid: 0
                        port_lid: 0
                        port_lmc: 0x00