Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] openMPI shared with NFS, but says different version
From: Cristobal Navarro (axischire_at_[hidden])
Date: 2010-07-28 18:04:04


Gus
my kernel for all nodes is this one:
Linux 2.6.32-22-server #36-Ubuntu SMP Thu Jun 3 20:38:33 UTC 2010 x86_64
GNU/Linux

at least for the moment i will use this configuration, at least for
deveplopment/testing of the parallel programs.
lag is minimum :)

whenever i get another kernel update, i will test again to check if sm
works, would be good to know that suddenly another distribution supports
nehalem sm.

best regards and thanks again
Cristobal
ps: guess what are the names of the other 2 nodes lol

On Wed, Jul 28, 2010 at 5:50 PM, Gus Correa <gus_at_[hidden]> wrote:

> Hi Cristobal
>
> Please, read my answer (way down the message) below.
>
> Cristobal Navarro wrote:
>
>>
>>
>> On Wed, Jul 28, 2010 at 3:28 PM, Gus Correa <gus_at_[hidden]<mailto:
>> gus_at_[hidden]>> wrote:
>>
>> Hi Cristobal
>>
>> Cristobal Navarro wrote:
>>
>>
>>
>> On Wed, Jul 28, 2010 at 11:09 AM, Gus Correa
>> <gus_at_[hidden] <mailto:gus_at_[hidden]>
>> <mailto:gus_at_[hidden] <mailto:gus_at_[hidden]>>>
>> wrote:
>>
>> Hi Cristobal
>>
>> In case you are not using full path name for mpiexec/mpirun,
>> what does "which mpirun" say?
>>
>>
>> --> $which mpirun
>> /opt/openmpi-1.4.2
>>
>>
>> Often times this is a source of confusion, old versions may
>> be first on the PATH.
>>
>> Gus
>>
>>
>> openMPI version problem is now gone, i can confirm that the
>> version is consistent now :), thanks.
>>
>>
>> This is good news.
>>
>>
>> however, i keep getting this kernel crash randomnly when i
>> execute with -np higher than 5
>> these are Xeons, with Hyperthreading On, is that a problem??
>>
>>
>> The problem may be with Hyperthreading, maybe not.
>> Which Xeons?
>>
>>
>> --> they are not so old, not so new either
>> fcluster_at_agua:~$ cat /proc/cpuinfo | more
>> processor : 0
>> vendor_id : GenuineIntel
>> cpu family : 6
>> model : 26
>> model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
>> stepping : 5
>> cpu MHz : 1596.000
>> cache size : 8192 KB
>> physical id : 0
>> siblings : 8
>> core id : 0
>> cpu cores : 4
>> apicid : 0
>> initial apicid : 0
>> fpu : yes
>> fpu_exception : yes
>> cpuid level : 11
>> wp : yes
>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
>> pse36 clflush dts acpi mmx fxsr sse sse2 ss h
>> t tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
>> xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_
>> cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida
>> tpr_shadow vnmi flexpriority ept vpid
>> bogomips : 4522.21
>> clflush size : 64
>> cache_alignment : 64
>> address sizes : 40 bits physical, 48 bits virtual
>> power management:
>> ...same for cpu1, 2, 3, ..., 15.
>>
>>
> AHA! Nehalems!
>
> Here they are E5540, just a different clock speed, I suppose.
>
>
> information on how the cpu is distributed
>> fcluster_at_agua:~$ lstopo
>> System(7992MB)
>> Socket#0 + L3(8192KB)
>> L2(256KB) + L1(32KB) + Core#0
>> P#0
>> P#8
>> L2(256KB) + L1(32KB) + Core#1
>> P#2
>> P#10
>> L2(256KB) + L1(32KB) + Core#2
>> P#4
>> P#12
>> L2(256KB) + L1(32KB) + Core#3
>> P#6
>> P#14
>> Socket#1 + L3(8192KB)
>> L2(256KB) + L1(32KB) + Core#0
>> P#1
>> P#9
>> L2(256KB) + L1(32KB) + Core#1
>> P#3
>> P#11
>> L2(256KB) + L1(32KB) + Core#2
>> P#5
>> P#13
>> L2(256KB) + L1(32KB) + Core#3
>> P#7
>> P#15
>>
>>
>>
>>
>> If I remember right, the old hyperthreading on old Xeons was
>> problematic.
>>
>> OTOH, about 1-2 months ago I had trouble with OpenMPI on a
>> relatively new Xeon Nehalem machine with (the new) Hyperthreading
>> turned on,
>> and Fedora Core 13.
>> The machine would hang with the OpenMPI connectivity example.
>> I reported this to the list, you may find in the archives.
>>
>>
>> --i foudn the archives recently about an hour ago, was not sure if it was
>> the same problem but i removed HT for testing with setting the online flag
>> to 0 on the extra cpus showed with lstopo, unfortenately i also crashes, so
>> HT may not be the problem.
>>
>
> It didn't fix the problem in our Nehalem machine here either,
> although it was FC13, and I don't know what OS and kernel you're using.
>
>
> Apparently other people got everything (OpenMPI with HT on Nehalem)
>> working in more stable distributions (CentOS, RHEL, etc).
>>
>> That problem was likely to be in the FC13 kernel,
>> because even turning off HT I still had the machine hanging.
>> Nothing worked with shared memory turned on,
>> so I had to switch OpenMPI to use tcp instead,
>> which is kind of ridiculous in a standalone machine.
>>
>>
>> --> very interesting, sm can be the problem
>>
>>
>>
>>
>> im trying to locate the kernel error on logs, but after
>> rebooting a crash, the error is not in the kern.log (neither
>> kern.log.1).
>> all i remember is that it starts with "Kernel BUG..."
>> and somepart it mentions a certain CPU X, where that cpu can be
>> any from 0 to 15 (im testing only in main node). Someone knows
>> where the log of kernel error could be?
>>
>>
>> Have you tried to turn off hyperthreading?
>>
>>
>> --> yes, tried, same crashes.
>>
>> In any case, depending on the application, it may not help much
>> performance to have HT on.
>>
>> A more radical alternative is to try
>> -mca btl tcp,self
>> in the mpirun command line.
>> That is what worked in the case I mentioned above.
>>
>>
>> wow!, this worked really :), you pointed out the problem, it was shared
>> memory.
>>
>
> Great news!
> That's exactly the problem we had here.
> Glad that the same solution worked for you.
>
> Over a year ago another fellow reported the same problem on Nehalem,
> on the very early days of Nehalem.
> The thread should be in the archives.
> Somebody back then (Ralph, or Jeff, or other?)
> suggested that turning off "sm" might work.
> So, I take no credit for this.
>
>
> i have 4 nodes, so anyways there will be node comunication, do you think i
>> can rely on working with -mca btl tcp,self?? i dont mind small lag.
>>
>>
> Well, this may be it, short from reinstalling the OS.
>
> Some people reported everything works with OpenMPI+HT+sm in CentOS and
> RHEL, see the thread I mentioned in the archives from 1-2 months ago.
> I don't administer that machine, and didn't have the time to do OS
> reinstall either.
> So I left it with -mca btl tcp,self, and the user/machine owner
> is happy that he can run his programs right,
> and with a performance that he considers good.
>
>
> i just have one more question, is this a problem of the ubuntu server
>> kernel?? from the Nehalem Cpus?? from openMPI (i dont think) ??
>>
>
> I don't have any idea.
> It may be a problem with some kernels, not sure.
> Which kernel do you have?
>
> Ours was FC-13, maybe FC-12, I don't remember exactly.
> Currently that machine has kernel 2.6.33.6-147.fc13.x86_64 #1 SMP.
> However, it may have been a slightly older kernel when I installed
> OpenMPI there.
> It may have been 2.6.33.5-124.fc13.x86_64 or 2.6.32.14-127.fc12.x86_64.
> My colleague here updates the machines with yum,
> so it may have gotten a new kernel since then.
>
> Our workhorse machines in the clusters that I take care
> of are AMD Opteron, never had this problem there.
> Maybe the kernels have yet to catch up with Nehalem,
> now Westmere, soon another one.
>
>
> and on what depends that in the future, sm could be possible on the same
>> configuration i have?? kernel update?.
>>
>>
> You may want to try CentOS or RHEL, but I can't guarantee the results.
> Somebody else in the list may have had the direct experience,
> and may speak out.
>
> It may be worth the effort anyway.
> After all, intra-node communication should be
> running on shared memory.
> Having to turn it off is outrageous.
>
> If you try another OS distribution,
> and if it works, please report the results back to the list:
> OS/distro, kernel, OpenMPI version, HT on or off,
> mca btl sm/tcp/self/etc choices, compilers, etc.
> This type of information is a real time saving for everybody.
>
>
>
> Thanks very much Gus, really!
>> Cristobal
>>
>>
>>
> My pleasure.
> Glad that there was a solution, even though not the best.
> Enjoy your cluster with vocano-named nodes!
> Have fun with OpenMPI and PETSc!
>
> Gus Correa
> ---------------------------------------------------------------------
> Gustavo Correa
> Lamont-Doherty Earth Observatory - Columbia University
> Palisades, NY, 10964-8000 - USA
> ---------------------------------------------------------------------
>
>
>>
>> My $0.02
>> Gus Correa
>>
>>
>> Cristobal Navarro wrote:
>>
>>
>> On Tue, Jul 27, 2010 at 7:29 PM, Gus Correa
>> <gus_at_[hidden] <mailto:gus_at_[hidden]>
>> <mailto:gus_at_[hidden] <mailto:gus_at_[hidden]>>
>> <mailto:gus_at_[hidden]
>> <mailto:gus_at_[hidden]> <mailto:gus_at_[hidden]
>> <mailto:gus_at_[hidden]>>>>
>>
>> wrote:
>>
>> Hi Cristobal
>>
>> Does it run only on the head node alone?
>> (Fuego? Agua? Acatenango?)
>> Try to put only the head node on the hostfile and execute
>> with mpiexec.
>>
>> --> i will try only with the head node, and post results
>> back
>> This may help sort out what is going on.
>> Hopefully it will run on the head node.
>>
>> Also, do you have Infinband connecting the nodes?
>> The error messages refer to the openib btl (i.e.
>> Infiniband),
>> and complains of
>>
>>
>> no we are just using normal network 100MBit/s , since i
>> am just
>> testing yet.
>>
>>
>> "perhaps a missing symbol, or compiled for a different
>> version of Open MPI?".
>> It sounds as a mixup of versions/builds.
>>
>>
>> --> i agree, somewhere there must be the remains of the
>> older
>> version
>>
>> Did you configure/build OpenMPI from source, or did
>> you install
>> it with apt-get?
>> It may be easier/less confusing to install from source.
>> If you did, what configure options did you use?
>>
>>
>> -->i installed from source, ./configure
>> --prefix=/opt/openmpi-1.4.2 --with-sge --without-xgid
>> --disable--static
>>
>> Also, as for the OpenMPI runtime environment,
>> it is not enough to set it on
>> the command line, because it will be effective only on
>> the
>> head node.
>> You need to either add them to the PATH and
>> LD_LIBRARY_PATH
>> on your .bashrc/.cshrc files (assuming these files and
>> your home
>> directory are *also* shared with the nodes via NFS),
>> or use the --prefix option of mpiexec to point to the
>> OpenMPI
>> main
>> directory.
>>
>>
>> yes, all nodes have their PATH and LD_LIBRARY_PATH set up
>> properly inside the login scripts ( .bashrc in my case )
>>
>> Needless to say, you need to check and ensure that the
>> OpenMPI
>> directory (and maybe your home directory, and your work
>> directory)
>> is (are)
>> really mounted on the nodes.
>>
>>
>> --> yes, doublechecked that they are
>>
>> I hope this helps,
>>
>>
>> --> thanks really!
>>
>> Gus Correa
>>
>> Update: i just reinstalled openMPI, with the same
>> parameters,
>> and it
>> seems that the problem has gone, i couldnt test
>> entirely but
>> when i
>> get back to lab ill confirm.
>>
>> best regards! Cristobal
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden] <mailto:users_at_[hidden]>
>> <mailto:users_at_[hidden] <mailto:users_at_[hidden]>>
>>
>>
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden] <mailto:users_at_[hidden]>
>> <mailto:users_at_[hidden] <mailto:users_at_[hidden]>>
>>
>>
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden] <mailto:users_at_[hidden]>
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden] <mailto:users_at_[hidden]>
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>