Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] How do I run OpenMPI safely on a Nehalem standalone machine?
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2010-05-06 17:34:26


On May 6, 2010, at 2:01 PM, Gus Correa wrote:

> 1) Now I can see and use the btl_sm_num_fifos component:
>
> I had committed already "btl = ^sm" to the openmpi-mca-params.conf
> file. This apparently hides the btl_sm_num_fifos from ompi_info.
>
> After I switched to no options in openmpi-mca-params.conf,
> then ompi_info showed the btl_sm_num_fifos component.
>
> ompi_info --all | grep btl_sm_num_fifos
> MCA btl: parameter "btl_sm_num_fifos" (current value: "1", data source: default value)
>
> A side comment:
> This means that the system administrator can
> hide some Open MPI options from the users, depending on what
> he puts in the openmpi-mca-params.conf file, right?

Correct.

BUT: a user can always override the "btl" MCA param and see them again. For example, you could also have done this:

   echo "btl =" > ~/.openmpi/mca-params.conf
   ompi_info --all | grep btl_sm_num_fifos
   # ...will show the sm params...

> 2) However, running with "sm" still breaks, unfortunately:
>
> Boomer!

Doh!

> I get the same errors that I reported in my very
> first email, if I increase the number of processes to 16,
> to explore the hyperthreading range.
>
> This is using "sm" (i.e. not excluded in the mca config file),
> and btl_sm_num_fifos (mpiexec command line)
>
> The machine hangs, requires a hard reboot, etc, etc,
> as reported earlier. See the below, please.

I saw that only some probably-unrelated dmesg messages were emitted. Was there anything else revealing on the console and/or /var/log/* files? Hard reboots absolutely should not be caused by Open MPI.

> So, I guess the conclusion is that I can use sm,
> but I have to remain within the range of physical cores (8),
> not oversubscribe, not try to explore the HT range.
> Should I expect it to work also for np>number of physical cores?

Your prior explanations of when HT is useful seemed pretty reasonable to me. Meaning: Nehalem HT will help only in some kinds of codes. Dense computation codes with few conditional branches may not benefit much from HT.

But OMPI applications should always run *correctly*, regardless of HT or not-HT -- even if you're oversubscribing. The performance may suffer (sometimes dramatically) if you oversubscribe physical cores with dense computational code, but it should always run *correctly*.

> I wonder if this would still work with np<=8, but with heavier code.
> (I only used hello_c.c so far.)

If hello_c is crashing your computer - even if you're running np>8 or np>16 -- something is wrong outside of Open MPI. I routinely run np=100 hello_c on machines.

> $ mpiexec -mca btl_sm_num_fifos 16 -np 16 a.out
> --------------------------------------------------------------------------
> mpiexec noticed that process rank 8 with PID 3659 on node spinoza.ldeo.columbia.edu exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
> $
>
> Message from syslogd_at_spinoza at May 6 13:38:13 ...
> kernel:------------[ cut here ]------------
>
> Message from syslogd_at_spinoza at May 6 13:38:13 ...
> kernel:invalid opcode: 0000 [#1] SMP
>
> Message from syslogd_at_spinoza at May 6 13:38:13 ...
> kernel:last sysfs file: /sys/devices/system/cpu/cpu15/topology/physical_package_id
>
> Message from syslogd_at_spinoza at May 6 13:38:13 ...
> kernel:Stack:
>
> Message from syslogd_at_spinoza at May 6 13:38:13 ...
> kernel:Call Trace:
>
> Message from syslogd_at_spinoza at May 6 13:38:13 ...
> kernel:Code: 48 89 45 a0 4c 89 ff e8 e0 dd 2b 00 41 8b b6 58 03 00 00 4c 89 e7 ff c6 e8 b5 bc ff ff 41 8b 96 5c 03 00 00 48 98 48 39 d0 73 04 <0f> 0b eb fe 48 29 d0 48 89 45 a8 66 41 ff 07 49 8b 94 24 00 01

I unfortunately don't know what these messages mean...

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/