Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Segfault in mca_odls_default.so with > ~100 process.
From: Ralph Castain (rhc_at_[hidden])
Date: 2010-02-27 02:46:18


Yeah, the system won't like this. Your approach makes it look like you are launching 136 app_contexts. We currently only support up to 128 app_contexts. I don't think anyone anticipated somebody trying to use the system this way.

I can expand the number to something larger. Will have to see how big a change it requires (mostly a question of how many places are touched) before we know what release this might show up in.

On Feb 26, 2010, at 2:34 AM, Oliver Ford wrote:

>
> I am trying to run an MPI code across 136 processing using an appfile
> (attached), since every process needs to be run with a host/process
> dependent parameter.
>
> This whole system works wonderfully for up to around 100 processes but
> usually fails with a segfault, apparently in in mca_odls_default.so,
> during initialization.
> The attached appfile is an attempt at 136 processes. If I split the
> appfile into two, both halves will initialize OK and successfully pass
> an MPI_Barrier() (the program won't actually work without all 136 nodes,
> but I'm happy MPI is doing its job). Because both halves work, I think
> it has to be related to the number of processes - not a problem with a
> specific appfile entry or machine.
>
> The cluster I am running it on has openmpi-1.3.3 but I have also
> installed 1.4.1 from the website in my home dir and that does the same
> (and is from which the attached data comes).
>
> The actual segfault is:
> [jac-11:12300] *** Process received signal ***
> [jac-11:12300] Signal: Segmentation fault (11)
> [jac-11:12300] Signal code: Address not mapped (1)
> [jac-11:12300] Failing at address: 0x40
> [jac-11:12300] [ 0] [0x74640c]
> [jac-11:12300] [ 1] /home/oford/openmpi/lib/openmpi/mca_odls_default.so
> [0x8863d4]
> [jac-11:12300] [ 2] /home/oford/openmpi/lib/libopen-rte.so.0 [0x76ffe9]
> [jac-11:12300] [ 3]
> /home/oford/openmpi/lib/libopen-rte.so.0(orte_daemon_cmd_processor+0x2f6)
> [0x771b86]
> [jac-11:12300] [ 4] /home/oford/openmpi/lib/libopen-pal.so.0 [0x5d6ba8]
> [jac-11:12300] [ 5]
> /home/oford/openmpi/lib/libopen-pal.so.0(opal_event_loop+0x27) [0x5d6e47]
> [jac-11:12300] [ 6]
> /home/oford/openmpi/lib/libopen-pal.so.0(opal_progress+0xce) [0x5ca00e]
> [jac-11:12300] [ 7]
> /home/oford/openmpi/lib/libopen-rte.so.0(orte_plm_base_launch_apps+0x355)
> [0x7815f5]
> [jac-11:12300] [ 8] /home/oford/openmpi/lib/openmpi/mca_plm_rsh.so
> [0xc73d1b]
> [jac-11:12300] [ 9] mpirun [0x804a8f0]
> [jac-11:12300] [10] mpirun [0x8049ef6]
> [jac-11:12300] [11] /lib/libc.so.6(__libc_start_main+0xe5) [0x1406e5]
> [jac-11:12300] [12] mpirun [0x8049e41]
> [jac-11:12300] *** End of error message ***
> Segmentation fault
>
>
> The full output with '-d' and the config.log from the build of 1.4.1 are
> also attached.
>
> I don't know the exact setup of the network, but I can ask our sysadmin
> anything else that might help.
>
> Thanks in advance,
>
>
> Oliver Ford
>
> Culham Centre for Fusion Energy
> Oxford, UK
>
>
>
>
> -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-39-80115 Y 11 11 133 debug
> -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-26-81244 N 11 11 133 debug
> -np 1 --host batch-020 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-122-75993 N 11 11 133 debug
> -np 1 --host batch-037 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-157-15286 N 11 11 133 debug
> -np 1 --host batch-042 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-114-89529 N 11 11 133 debug
> -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-35-90257 N 11 11 133 debug
> -np 1 --host batch-020 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-151-56062 N 11 11 133 debug
> -np 1 --host batch-004 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-16-2723 N 11 11 133 debug
> -np 1 --host batch-003 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-156-65790 N 11 11 133 debug
> -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-198-63239 N 11 11 133 debug
> -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-105-12753 N 11 11 133 debug
> -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-12-25631 N 11 11 133 debug
> -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-196-35421 N 11 11 133 debug
> -np 1 --host batch-045 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-103-98246 N 11 11 133 debug
> -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-142-44009 N 11 11 133 debug
> -np 1 --host batch-044 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-117-30325 N 11 11 133 debug
> -np 1 --host batch-003 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-143-21739 N 11 11 133 debug
> -np 1 --host batch-042 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-112-64293 N 11 11 133 debug
> -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-57-11238 N 11 11 133 debug
> -np 1 --host batch-025 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-170-80831 N 11 11 133 debug
> -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-32-47578 N 11 11 133 debug
> -np 1 --host batch-027 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-146-27711 N 11 11 133 debug
> -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-25-36312 N 11 11 133 debug
> -np 1 --host batch-023 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-154-17110 N 11 11 133 debug
> -np 1 --host batch-023 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-136-3872 N 11 11 133 debug
> -np 1 --host batch-036 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-132-1105 N 11 11 133 debug
> -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-192-11991 N 11 11 133 debug
> -np 1 --host batch-045 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-115-41110 N 11 11 133 debug
> -np 1 --host batch-036 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-155-67802 N 11 11 133 debug
> -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-40-46953 N 11 11 133 debug
> -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-125-79465 N 11 11 133 debug
> -np 1 --host batch-022 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-128-79309 N 11 11 133 debug
> -np 1 --host batch-033 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-152-82320 N 11 11 133 debug
> -np 1 --host batch-026 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-158-68939 N 11 11 133 debug
> -np 1 --host batch-020 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-137-41369 N 11 11 133 debug
> -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-2-66663 N 11 11 133 debug
> -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-52-62955 N 11 11 133 debug
> -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-195-10991 N 11 11 133 debug
> -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-193-50774 N 11 11 133 debug
> -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-177-45516 N 11 11 133 debug
> -np 1 --host batch-032 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-171-96942 N 11 11 133 debug
> -np 1 --host batch-038 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-175-99761 N 11 11 133 debug
> -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-118-35065 N 11 11 133 debug
> -np 1 --host batch-014 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-17-49750 N 11 11 133 debug
> -np 1 --host batch-030 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-169-10604 N 11 11 133 debug
> -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-46-74041 N 11 11 133 debug
> -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-187-7833 N 11 11 133 debug
> -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-200-36312 N 11 11 133 debug
> -np 1 --host batch-027 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-159-37278 N 11 11 133 debug
> -np 1 --host batch-045 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-11-37181 N 11 11 133 debug
> -np 1 --host batch-013 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-149-11377 N 11 11 133 debug
> -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-179-64715 N 11 11 133 debug
> -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-44-46953 N 11 11 133 debug
> -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-130-64980 N 11 11 133 debug
> -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-13-39413 N 11 11 133 debug
> -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-34-46573 N 11 11 133 debug
> -np 1 --host batch-027 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-134-85354 N 11 11 133 debug
> -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-194-95760 N 11 11 133 debug
> -np 1 --host batch-013 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-133-18010 N 11 11 133 debug
> -np 1 --host batch-024 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-173-20377 N 11 11 133 debug
> -np 1 --host batch-013 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-160-43487 N 11 11 133 debug
> -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-106-65988 N 11 11 133 debug
> -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-18-18834 N 11 11 133 debug
> -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-180-19539 N 11 11 133 debug
> -np 1 --host batch-022 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-140-53706 N 11 11 133 debug
> -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-184-80559 N 11 11 133 debug
> -np 1 --host batch-023 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-126-6794 N 11 11 133 debug
> -np 1 --host batch-044 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-111-83886 N 11 11 133 debug
> -np 1 --host batch-005 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-150-26152 N 11 11 133 debug
> -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-29-81244 N 11 11 133 debug
> -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-30-5208 N 11 11 133 debug
> -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-185-82710 N 11 11 133 debug
> -np 1 --host batch-022 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-153-49288 N 11 11 133 debug
> -np 1 --host batch-026 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-131-48260 N 11 11 133 debug
> -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-21-85978 N 11 11 133 debug
> -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-186-67937 N 11 11 133 debug
> -np 1 --host batch-033 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-135-94983 N 11 11 133 debug
> -np 1 --host batch-044 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-107-10462 N 11 11 133 debug
> -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-113-29689 N 11 11 133 debug
> -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-27-60637 N 11 11 133 debug
> -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-36-26250 N 11 11 133 debug
> -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-119-1554 N 11 11 133 debug
> -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-24-17859 N 11 11 133 debug
> -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-31-5480 N 11 11 133 debug
> -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-23-24746 N 11 11 133 debug
> -np 1 --host batch-005 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-138-53784 N 11 11 133 debug
> -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-33-43810 N 11 11 133 debug
> -np 1 --host batch-005 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-121-38059 N 11 11 133 debug
> -np 1 --host batch-036 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-145-79515 N 11 11 133 debug
> -np 1 --host batch-042 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-109-66314 N 11 11 133 debug
> -np 1 --host batch-028 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-168-62760 N 11 11 133 debug
> -np 1 --host batch-001 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-166-35091 N 11 11 133 debug
> -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-3-30335 N 11 11 133 debug
> -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-108-43856 N 11 11 133 debug
> -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-116-98652 N 11 11 133 debug
> -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-189-70592 N 11 11 133 debug
> -np 1 --host batch-004 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-148-12571 N 11 11 133 debug
> -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-38-79978 N 11 11 133 debug
> -np 1 --host batch-003 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-129-21483 N 11 11 133 debug
> -np 1 --host batch-033 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-124-3865 N 11 11 133 debug
> -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-191-79048 N 11 11 133 debug
> -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-188-93446 N 11 11 133 debug
> -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-37-71686 N 11 11 133 debug
> -np 1 --host batch-037 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-162-83683 N 11 11 133 debug
> -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-1-48706 N 11 11 133 debug
> -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-199-95838 N 11 11 133 debug
> -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-178-23994 N 11 11 133 debug
> -np 1 --host batch-026 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-144-79515 N 11 11 133 debug
> -np 1 --host batch-008 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-15-46689 N 11 11 133 debug
> -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-197-46966 N 11 11 133 debug
> -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-41-82354 N 11 11 133 debug
> -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-20-1112 N 11 11 133 debug
> -np 1 --host batch-039 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-176-80381 N 11 11 133 debug
> -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-101-91749 N 11 11 133 debug
> -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-10-96608 N 11 11 133 debug
> -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-139-62526 N 11 11 133 debug
> -np 1 --host batch-007 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-164-93486 N 11 11 133 debug
> -np 1 --host batch-018 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-174-37716 N 11 11 133 debug
> -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-104-25353 N 11 11 133 debug
> -np 1 --host batch-031 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-172-7131 N 11 11 133 debug
> -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-127-18397 N 11 11 133 debug
> -np 1 --host batch-034 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-147-65530 N 11 11 133 debug
> -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-14-40975 N 11 11 133 debug
> -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-123-45216 N 11 11 133 debug
> -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-190-27454 N 11 11 133 debug
> -np 1 --host batch-007 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-141-10541 N 11 11 133 debug
> -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-102-77190 N 11 11 133 debug
> -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-120-13012 N 11 11 133 debug
> -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-19-29582 N 11 11 133 debug
> -np 1 --host batch-008 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-161-20135 N 11 11 133 debug
> -np 1 --host batch-040 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-163-95832 N 11 11 133 debug
> -np 1 --host batch-019 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-167-60329 N 11 11 133 debug
> -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-28-77824 N 11 11 133 debug
> -np 1 --host batch-034 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-165-57829 N 11 11 133 debug
> -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-4-1153 N 11 11 133 debug
> -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-22-4961 N 11 11 133 debug
>
>
> [jac-11:15014] procdir: /tmp/openmpi-sessions-oford_at_jac-11_0/48099/0/0
> [jac-11:15014] jobdir: /tmp/openmpi-sessions-oford_at_jac-11_0/48099/0
> [jac-11:15014] top: openmpi-sessions-oford_at_jac-11_0
> [jac-11:15014] tmp: /tmp
> [jac-11:15014] mpirun: reset PATH: /home/oford/openmpi/bin:/home/oford/openmpi/bin:/home/oford/bin:/usr/lib/qt-3.3/bin:/usr/bin:/bin:/usr/local/depot/PGI/pgi-7.0/linux86/7.0/bin:/usr/local/bin:/jet/share/root/bin:/jet/share/bin:.:/u/sim/sh:/u/sim/cmg/sh:/home/oford/sh:/u/sim/sh/linux:/home/oford/sh/linux:/u/sim/edge2d/sh:/u/sim/edge2d/sh/linux:/home/oford/edge2d/sh:/home/oford/edge2d/sh/linux:/u/sim/jetto/sh
> [jac-11:15014] mpirun: reset LD_LIBRARY_PATH: /home/oford/openmpi/lib:/home/oford/openmpi/lib:/u/sim/mdsplus/default/lib/linux:/u/sim/jproc/default/lib/linux:/u/sim/eproc/default/lib/linux/fixed:/home/oford/oldXlib:/home/oford/hdf5:/jet/share/lib:/usr/local/lib
> [batch-020:17451] procdir: /tmp/openmpi-sessions-oford_at_batch-020_0/48099/0/2
> [batch-020:17451] jobdir: /tmp/openmpi-sessions-oford_at_batch-020_0/48099/0
> [batch-020:17451] top: openmpi-sessions-oford_at_batch-020_0
> [batch-020:17451] tmp: /tmp
> [batch-042:15862] procdir: /tmp/openmpi-sessions-oford_at_batch-042_0/48099/0/4
> [batch-042:15862] jobdir: /tmp/openmpi-sessions-oford_at_batch-042_0/48099/0
> [batch-042:15862] top: openmpi-sessions-oford_at_batch-042_0
> [batch-042:15862] tmp: /tmp
> [batch-023:09760] procdir: /tmp/openmpi-sessions-oford_at_batch-023_0/48099/0/19
> [batch-023:09760] jobdir: /tmp/openmpi-sessions-oford_at_batch-023_0/48099/0
> [batch-023:09760] top: openmpi-sessions-oford_at_batch-023_0
> [batch-023:09760] tmp: /tmp
> [batch-025:05770] procdir: /tmp/openmpi-sessions-oford_at_batch-025_0/48099/0/15
> [batch-025:05770] jobdir: /tmp/openmpi-sessions-oford_at_batch-025_0/48099/0
> [batch-025:05770] top: openmpi-sessions-oford_at_batch-025_0
> [batch-025:05770] tmp: /tmp
> [batch-037:17611] procdir: /tmp/openmpi-sessions-oford_at_batch-037_0/48099/0/3
> [batch-037:17611] jobdir: /tmp/openmpi-sessions-oford_at_batch-037_0/48099/0
> [batch-037:17611] top: openmpi-sessions-oford_at_batch-037_0
> [batch-037:17611] tmp: /tmp
> [jac-5:15183] procdir: /tmp/openmpi-sessions-oford_at_jac-5_0/48099/0/1
> [jac-5:15183] jobdir: /tmp/openmpi-sessions-oford_at_jac-5_0/48099/0
> [jac-5:15183] top: openmpi-sessions-oford_at_jac-5_0
> [jac-5:15183] tmp: /tmp
> [jac-9:22204] procdir: /tmp/openmpi-sessions-oford_at_jac-9_0/48099/0/5
> [jac-9:22204] jobdir: /tmp/openmpi-sessions-oford_at_jac-9_0/48099/0
> [jac-9:22204] top: openmpi-sessions-oford_at_jac-9_0
> [jac-9:22204] tmp: /tmp
> [batch-013:23865] procdir: /tmp/openmpi-sessions-oford_at_batch-013_0/48099/0/33
> [batch-013:23865] jobdir: /tmp/openmpi-sessions-oford_at_batch-013_0/48099/0
> [batch-013:23865] top: openmpi-sessions-oford_at_batch-013_0
> [batch-013:23865] tmp: /tmp
> [batch-022:15225] procdir: /tmp/openmpi-sessions-oford_at_batch-022_0/48099/0/23
> [batch-022:15225] jobdir: /tmp/openmpi-sessions-oford_at_batch-022_0/48099/0
> [batch-022:15225] top: openmpi-sessions-oford_at_batch-022_0
> [batch-022:15225] tmp: /tmp
> [batch-018:19925] procdir: /tmp/openmpi-sessions-oford_at_batch-018_0/48099/0/44
> [batch-018:19925] jobdir: /tmp/openmpi-sessions-oford_at_batch-018_0/48099/0
> [batch-018:19925] top: openmpi-sessions-oford_at_batch-018_0
> [batch-018:19925] tmp: /tmp
> [jac-4:06627] procdir: /tmp/openmpi-sessions-oford_at_jac-4_0/48099/0/37
> [jac-4:06627] jobdir: /tmp/openmpi-sessions-oford_at_jac-4_0/48099/0
> [jac-4:06627] top: openmpi-sessions-oford_at_jac-4_0
> [jac-4:06627] tmp: /tmp
> [batch-004:30088] procdir: /tmp/openmpi-sessions-oford_at_batch-004_0/48099/0/6
> [batch-004:30088] jobdir: /tmp/openmpi-sessions-oford_at_batch-004_0/48099/0
> [batch-004:30088] top: openmpi-sessions-oford_at_batch-004_0
> [batch-004:30088] tmp: /tmp
> [batch-046:00820] procdir: /tmp/openmpi-sessions-oford_at_batch-046_0/48099/0/8
> [batch-046:00820] jobdir: /tmp/openmpi-sessions-oford_at_batch-046_0/48099/0
> [batch-046:00820] top: openmpi-sessions-oford_at_batch-046_0
> [batch-046:00820] tmp: /tmp
> [batch-036:16988] procdir: /tmp/openmpi-sessions-oford_at_batch-036_0/48099/0/20
> [batch-036:16988] jobdir: /tmp/openmpi-sessions-oford_at_batch-036_0/48099/0
> [batch-036:16988] top: openmpi-sessions-oford_at_batch-036_0
> [batch-036:16988] tmp: /tmp
> [batch-030:25412] procdir: /tmp/openmpi-sessions-oford_at_batch-030_0/48099/0/31
> [batch-030:25412] jobdir: /tmp/openmpi-sessions-oford_at_batch-030_0/48099/0
> [batch-030:25412] top: openmpi-sessions-oford_at_batch-030_0
> [batch-030:25412] tmp: /tmp
> [jac-6:04101] procdir: /tmp/openmpi-sessions-oford_at_jac-6_0/48099/0/16
> [jac-6:04101] jobdir: /tmp/openmpi-sessions-oford_at_jac-6_0/48099/0
> [jac-6:04101] top: openmpi-sessions-oford_at_jac-6_0
> [jac-6:04101] tmp: /tmp
> [batch-006:16136] procdir: /tmp/openmpi-sessions-oford_at_batch-006_0/48099/0/12
> [batch-006:16136] jobdir: /tmp/openmpi-sessions-oford_at_batch-006_0/48099/0
> [batch-006:16136] top: openmpi-sessions-oford_at_batch-006_0
> [batch-006:16136] tmp: /tmp
> [jac-7:17379] procdir: /tmp/openmpi-sessions-oford_at_jac-7_0/48099/0/32
> [jac-7:17379] jobdir: /tmp/openmpi-sessions-oford_at_jac-7_0/48099/0
> [jac-7:17379] top: openmpi-sessions-oford_at_jac-7_0
> [jac-7:17379] tmp: /tmp
> [batch-003:23921] procdir: /tmp/openmpi-sessions-oford_at_batch-003_0/48099/0/7
> [batch-003:23921] jobdir: /tmp/openmpi-sessions-oford_at_batch-003_0/48099/0
> [batch-003:23921] top: openmpi-sessions-oford_at_batch-003_0
> [batch-003:23921] tmp: /tmp
> [batch-044:01101] procdir: /tmp/openmpi-sessions-oford_at_batch-044_0/48099/0/13
> [batch-044:01101] jobdir: /tmp/openmpi-sessions-oford_at_batch-044_0/48099/0
> [batch-044:01101] top: openmpi-sessions-oford_at_batch-044_0
> [batch-044:01101] tmp: /tmp
> [batch-039:02035] procdir: /tmp/openmpi-sessions-oford_at_batch-039_0/48099/0/42
> [batch-039:02035] jobdir: /tmp/openmpi-sessions-oford_at_batch-039_0/48099/0
> [batch-039:02035] top: openmpi-sessions-oford_at_batch-039_0
> [batch-039:02035] tmp: /tmp
> [batch-033:02863] procdir: /tmp/openmpi-sessions-oford_at_batch-033_0/48099/0/24
> [batch-033:02863] jobdir: /tmp/openmpi-sessions-oford_at_batch-033_0/48099/0
> [batch-033:02863] top: openmpi-sessions-oford_at_batch-033_0
> [batch-033:02863] tmp: /tmp
> [jac-8:26567] procdir: /tmp/openmpi-sessions-oford_at_jac-8_0/48099/0/18
> [jac-8:26567] jobdir: /tmp/openmpi-sessions-oford_at_jac-8_0/48099/0
> [jac-8:26567] top: openmpi-sessions-oford_at_jac-8_0
> [jac-8:26567] tmp: /tmp
> [batch-028:14068] procdir: /tmp/openmpi-sessions-oford_at_batch-028_0/48099/0/38
> [batch-028:14068] jobdir: /tmp/openmpi-sessions-oford_at_batch-028_0/48099/0
> [batch-028:14068] top: openmpi-sessions-oford_at_batch-028_0
> [batch-028:14068] tmp: /tmp
> [batch-019:32641] procdir: /tmp/openmpi-sessions-oford_at_batch-019_0/48099/0/48
> [batch-019:32641] jobdir: /tmp/openmpi-sessions-oford_at_batch-019_0/48099/0
> [batch-019:32641] top: openmpi-sessions-oford_at_batch-019_0
> [batch-019:32641] tmp: /tmp
> [batch-038:07716] procdir: /tmp/openmpi-sessions-oford_at_batch-038_0/48099/0/29
> [batch-038:07716] jobdir: /tmp/openmpi-sessions-oford_at_batch-038_0/48099/0
> [batch-038:07716] top: openmpi-sessions-oford_at_batch-038_0
> [batch-038:07716] tmp: /tmp
> [jac-2:05183] procdir: /tmp/openmpi-sessions-oford_at_jac-2_0/48099/0/22
> [jac-2:05183] jobdir: /tmp/openmpi-sessions-oford_at_jac-2_0/48099/0
> [jac-2:05183] top: openmpi-sessions-oford_at_jac-2_0
> [jac-2:05183] tmp: /tmp
> [batch-017:25718] procdir: /tmp/openmpi-sessions-oford_at_batch-017_0/48099/0/34
> [batch-017:25718] jobdir: /tmp/openmpi-sessions-oford_at_batch-017_0/48099/0
> [batch-017:25718] top: openmpi-sessions-oford_at_batch-017_0
> [batch-017:25718] tmp: /tmp
> [batch-026:09510] procdir: /tmp/openmpi-sessions-oford_at_batch-026_0/48099/0/25
> [batch-026:09510] jobdir: /tmp/openmpi-sessions-oford_at_batch-026_0/48099/0
> [batch-026:09510] top: openmpi-sessions-oford_at_batch-026_0
> [batch-026:09510] tmp: /tmp
> [batch-043:01668] procdir: /tmp/openmpi-sessions-oford_at_batch-043_0/48099/0/40
> [batch-043:01668] jobdir: /tmp/openmpi-sessions-oford_at_batch-043_0/48099/0
> [batch-043:01668] top: openmpi-sessions-oford_at_batch-043_0
> [batch-043:01668] tmp: /tmp
> [batch-008:17836] procdir: /tmp/openmpi-sessions-oford_at_batch-008_0/48099/0/41
> [batch-008:17836] jobdir: /tmp/openmpi-sessions-oford_at_batch-008_0/48099/0
> [batch-008:17836] top: openmpi-sessions-oford_at_batch-008_0
> [batch-008:17836] tmp: /tmp
> [jac-12:17562] procdir: /tmp/openmpi-sessions-oford_at_jac-12_0/48099/0/10
> [jac-12:17562] jobdir: /tmp/openmpi-sessions-oford_at_jac-12_0/48099/0
> [jac-12:17562] top: openmpi-sessions-oford_at_jac-12_0
> [jac-12:17562] tmp: /tmp
> [batch-005:03532] procdir: /tmp/openmpi-sessions-oford_at_batch-005_0/48099/0/36
> [batch-005:03532] jobdir: /tmp/openmpi-sessions-oford_at_batch-005_0/48099/0
> [batch-005:03532] top: openmpi-sessions-oford_at_batch-005_0
> [batch-005:03532] tmp: /tmp
> [batch-045:04261] procdir: /tmp/openmpi-sessions-oford_at_batch-045_0/48099/0/11
> [batch-045:04261] jobdir: /tmp/openmpi-sessions-oford_at_batch-045_0/48099/0
> [batch-045:04261] top: openmpi-sessions-oford_at_batch-045_0
> [batch-045:04261] tmp: /tmp
> [batch-001:22016] procdir: /tmp/openmpi-sessions-oford_at_batch-001_0/48099/0/39
> [batch-001:22016] jobdir: /tmp/openmpi-sessions-oford_at_batch-001_0/48099/0
> [batch-001:22016] top: openmpi-sessions-oford_at_batch-001_0
> [batch-001:22016] tmp: /tmp
> [batch-040:08361] procdir: /tmp/openmpi-sessions-oford_at_batch-040_0/48099/0/47
> [batch-040:08361] jobdir: /tmp/openmpi-sessions-oford_at_batch-040_0/48099/0
> [batch-040:08361] top: openmpi-sessions-oford_at_batch-040_0
> [batch-040:08361] tmp: /tmp
> [batch-007:05689] procdir: /tmp/openmpi-sessions-oford_at_batch-007_0/48099/0/43
> [batch-007:05689] jobdir: /tmp/openmpi-sessions-oford_at_batch-007_0/48099/0
> [batch-007:05689] top: openmpi-sessions-oford_at_batch-007_0
> [batch-007:05689] tmp: /tmp
> [batch-032:02059] procdir: /tmp/openmpi-sessions-oford_at_batch-032_0/48099/0/28
> [batch-032:02059] jobdir: /tmp/openmpi-sessions-oford_at_batch-032_0/48099/0
> [batch-032:02059] top: openmpi-sessions-oford_at_batch-032_0
> [batch-032:02059] tmp: /tmp
> [batch-031:01049] procdir: /tmp/openmpi-sessions-oford_at_batch-031_0/48099/0/45
> [batch-031:01049] jobdir: /tmp/openmpi-sessions-oford_at_batch-031_0/48099/0
> [batch-031:01049] top: openmpi-sessions-oford_at_batch-031_0
> [batch-031:01049] tmp: /tmp
> [batch-024:05301] procdir: /tmp/openmpi-sessions-oford_at_batch-024_0/48099/0/35
> [batch-024:05301] jobdir: /tmp/openmpi-sessions-oford_at_batch-024_0/48099/0
> [batch-024:05301] top: openmpi-sessions-oford_at_batch-024_0
> [batch-024:05301] tmp: /tmp
> [batch-034:24766] procdir: /tmp/openmpi-sessions-oford_at_batch-034_0/48099/0/46
> [batch-034:24766] jobdir: /tmp/openmpi-sessions-oford_at_batch-034_0/48099/0
> [batch-034:24766] top: openmpi-sessions-oford_at_batch-034_0
> [batch-034:24766] tmp: /tmp
> [jac-10:27707] procdir: /tmp/openmpi-sessions-oford_at_jac-10_0/48099/0/21
> [jac-10:27707] jobdir: /tmp/openmpi-sessions-oford_at_jac-10_0/48099/0
> [jac-10:27707] top: openmpi-sessions-oford_at_jac-10_0
> [jac-10:27707] tmp: /tmp
> [batch-014:30991] procdir: /tmp/openmpi-sessions-oford_at_batch-014_0/48099/0/30
> [batch-014:30991] jobdir: /tmp/openmpi-sessions-oford_at_batch-014_0/48099/0
> [batch-014:30991] top: openmpi-sessions-oford_at_batch-014_0
> [batch-014:30991] tmp: /tmp
> [jac-3:20114] procdir: /tmp/openmpi-sessions-oford_at_jac-3_0/48099/0/27
> [jac-3:20114] jobdir: /tmp/openmpi-sessions-oford_at_jac-3_0/48099/0
> [jac-3:20114] top: openmpi-sessions-oford_at_jac-3_0
> [jac-3:20114] tmp: /tmp
> [batch-015:01829] procdir: /tmp/openmpi-sessions-oford_at_batch-015_0/48099/0/9
> [batch-015:01829] jobdir: /tmp/openmpi-sessions-oford_at_batch-015_0/48099/0
> [batch-015:01829] top: openmpi-sessions-oford_at_batch-015_0
> [batch-015:01829] tmp: /tmp
> [batch-041:17727] procdir: /tmp/openmpi-sessions-oford_at_batch-041_0/48099/0/14
> [batch-041:17727] jobdir: /tmp/openmpi-sessions-oford_at_batch-041_0/48099/0
> [batch-041:17727] top: openmpi-sessions-oford_at_batch-041_0
> [batch-041:17727] tmp: /tmp
> [jac-13:26153] procdir: /tmp/openmpi-sessions-oford_at_jac-13_0/48099/0/26
> [jac-13:26153] jobdir: /tmp/openmpi-sessions-oford_at_jac-13_0/48099/0
> [jac-13:26153] top: openmpi-sessions-oford_at_jac-13_0
> [jac-13:26153] tmp: /tmp
> [batch-027:23746] procdir: /tmp/openmpi-sessions-oford_at_batch-027_0/48099/0/17
> [batch-027:23746] jobdir: /tmp/openmpi-sessions-oford_at_batch-027_0/48099/0
> [batch-027:23746] top: openmpi-sessions-oford_at_batch-027_0
> [batch-027:23746] tmp: /tmp
> [jac-11:15014] [[48099,0],0] node[0].name jac-11 daemon 0 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[1].name jac-5 daemon 1 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[2].name batch-020 daemon 2 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[3].name batch-037 daemon 3 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[4].name batch-042 daemon 4 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[5].name jac-9 daemon 5 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[6].name batch-004 daemon 6 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[7].name batch-003 daemon 7 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[8].name batch-046 daemon 8 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[9].name batch-015 daemon 9 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[10].name jac-12 daemon 10 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[11].name batch-045 daemon 11 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[12].name batch-006 daemon 12 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[13].name batch-044 daemon 13 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[14].name batch-041 daemon 14 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[15].name batch-025 daemon 15 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[16].name jac-6 daemon 16 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[17].name batch-027 daemon 17 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[18].name jac-8 daemon 18 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[19].name batch-023 daemon 19 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[20].name batch-036 daemon 20 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[21].name jac-10 daemon 21 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[22].name jac-2 daemon 22 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[23].name batch-022 daemon 23 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[24].name batch-033 daemon 24 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[25].name batch-026 daemon 25 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[26].name jac-13 daemon 26 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[27].name jac-3 daemon 27 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[28].name batch-032 daemon 28 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[29].name batch-038 daemon 29 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[30].name batch-014 daemon 30 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[31].name batch-030 daemon 31 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[32].name jac-7 daemon 32 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[33].name batch-013 daemon 33 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[34].name batch-017 daemon 34 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[35].name batch-024 daemon 35 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[36].name batch-005 daemon 36 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[37].name jac-4 daemon 37 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[38].name batch-028 daemon 38 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[39].name batch-001 daemon 39 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[40].name batch-043 daemon 40 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[41].name batch-008 daemon 41 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[42].name batch-039 daemon 42 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[43].name batch-007 daemon 43 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[44].name batch-018 daemon 44 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[45].name batch-031 daemon 45 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[46].name batch-034 daemon 46 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[47].name batch-040 daemon 47 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[48].name batch-019 daemon 48 arch ffca0200
> [jac-11:15014] *** Process received signal ***
> [jac-11:15014] Signal: Segmentation fault (11)
> [jac-11:15014] Signal code: Address not mapped (1)
> [jac-11:15014] Failing at address: 0x302c5d70
> [jac-11:15014] [ 0] [0x9f540c]
> [jac-11:15014] [ 1] /home/oford/openmpi/lib/openmpi/mca_odls_default.so [0x52c3d4]
> [jac-11:15014] [ 2] /home/oford/openmpi/lib/libopen-rte.so.0 [0x22afe9]
> [jac-11:15014] [ 3] /home/oford/openmpi/lib/libopen-rte.so.0(orte_daemon_cmd_processor+0x2f6) [0x22cb86]
> [jac-11:15014] [ 4] /home/oford/openmpi/lib/libopen-pal.so.0 [0x129ba8]
> [jac-11:15014] [ 5] /home/oford/openmpi/lib/libopen-pal.so.0(opal_event_loop+0x27) [0x129e47]
> [jac-11:15014] [ 6] /home/oford/openmpi/lib/libopen-pal.so.0(opal_progress+0xce) [0x11d00e]
> [jac-11:15014] [ 7] /home/oford/openmpi/lib/libopen-rte.so.0(orte_plm_base_launch_apps+0x355) [0x23c5f5]
> [jac-11:15014] [ 8] /home/oford/openmpi/lib/openmpi/mca_plm_rsh.so [0x391d1b]
> [jac-11:15014] [ 9] mpirun [0x804a8f0]
> [jac-11:15014] [10] mpirun [0x8049ef6]
> [jac-11:15014] [11] /lib/libc.so.6(__libc_start_main+0xe5) [0x5b36e5]
> [jac-11:15014] [12] mpirun [0x8049e41]
> [jac-11:15014] *** End of error message ***
> [batch-041:17727] [[48099,0],14] routed:binomial: Connection to lifeline [[48099,0],0] lo[jac-6:04101] sess_dir_finalize: proc session dir not empty - leaving
> eaving
> batch-034:24766] sess_dir_finalize: proc session dir not empty - leaving
>
> [jac-12:17562] sess_dir_finalize: proc session dir not empty - leaving
> ng
>
>
> Package: Open MPI oford_at_jac-12 Distribution
> Open MPI: 1.4.1
> Open MPI SVN revision: r22421
> Open MPI release date: Jan 14, 2010
> Open RTE: 1.4.1
> Open RTE SVN revision: r22421
> Open RTE release date: Jan 14, 2010
> OPAL: 1.4.1
> OPAL SVN revision: r22421
> OPAL release date: Jan 14, 2010
> Ident string: 1.4.1
> Prefix: /home/oford/openmpi
> Configured architecture: i686-pc-linux-gnu
> Configure host: jac-12
> Configured by: oford
> Configured on: Thu Feb 25 22:46:23 GMT 2010
> Configure host: jac-12
> Built by: oford
> Built on: Thu Feb 25 23:03:10 GMT 2010
> Built host: jac-2
> C bindings: yes
> C++ bindings: yes
> Fortran77 bindings: yes (all)
> Fortran90 bindings: yes
> Fortran90 bindings size: small
> C compiler: gcc
> C compiler absolute: /usr/bin/gcc
> C++ compiler: g++
> C++ compiler absolute: /usr/bin/g++
> Fortran77 compiler: gfortran
> Fortran77 compiler abs: /usr/bin/gfortran
> Fortran90 compiler: gfortran
> Fortran90 compiler abs: /usr/bin/gfortran
> C profiling: yes
> C++ profiling: yes
> Fortran77 profiling: yes
> Fortran90 profiling: yes
> C++ exceptions: no
> Thread support: posix (mpi: no, progress: no)
> Sparse Groups: no
> Internal debug support: no
> MPI parameter check: runtime
> Memory profiling support: no
> Memory debugging support: no
> libltdl support: yes
> Heterogeneous support: no
> mpirun default --prefix: yes
> MPI I/O support: yes
> MPI_WTIME support: gettimeofday
> Symbol visibility support: yes
> FT Checkpoint support: no (checkpoint thread: no)
> MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.4.1)
> MCA memory: ptmalloc2 (MCA v2.0, API v2.0, Component v1.4.1)
> MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.4.1)
> MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.4.1)
> MCA carto: file (MCA v2.0, API v2.0, Component v1.4.1)
> MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.4.1)
> MCA timer: linux (MCA v2.0, API v2.0, Component v1.4.1)
> MCA installdirs: env (MCA v2.0, API v2.0, Component v1.4.1)
> MCA installdirs: config (MCA v2.0, API v2.0, Component v1.4.1)
> MCA dpm: orte (MCA v2.0, API v2.0, Component v1.4.1)
> MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.4.1)
> MCA allocator: basic (MCA v2.0, API v2.0, Component v1.4.1)
> MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: basic (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: inter (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: self (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: sm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: sync (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: tuned (MCA v2.0, API v2.0, Component v1.4.1)
> MCA io: romio (MCA v2.0, API v2.0, Component v1.4.1)
> MCA mpool: fake (MCA v2.0, API v2.0, Component v1.4.1)
> MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.4.1)
> MCA mpool: sm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA pml: cm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA pml: csum (MCA v2.0, API v2.0, Component v1.4.1)
> MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.4.1)
> MCA pml: v (MCA v2.0, API v2.0, Component v1.4.1)
> MCA bml: r2 (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rcache: vma (MCA v2.0, API v2.0, Component v1.4.1)
> MCA btl: self (MCA v2.0, API v2.0, Component v1.4.1)
> MCA btl: sm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA btl: tcp (MCA v2.0, API v2.0, Component v1.4.1)
> MCA topo: unity (MCA v2.0, API v2.0, Component v1.4.1)
> MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.4.1)
> MCA osc: rdma (MCA v2.0, API v2.0, Component v1.4.1)
> MCA iof: hnp (MCA v2.0, API v2.0, Component v1.4.1)
> MCA iof: orted (MCA v2.0, API v2.0, Component v1.4.1)
> MCA iof: tool (MCA v2.0, API v2.0, Component v1.4.1)
> MCA oob: tcp (MCA v2.0, API v2.0, Component v1.4.1)
> MCA odls: default (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ras: slurm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rml: oob (MCA v2.0, API v2.0, Component v1.4.1)
> MCA routed: binomial (MCA v2.0, API v2.0, Component v1.4.1)
> MCA routed: direct (MCA v2.0, API v2.0, Component v1.4.1)
> MCA routed: linear (MCA v2.0, API v2.0, Component v1.4.1)
> MCA plm: rsh (MCA v2.0, API v2.0, Component v1.4.1)
> MCA plm: slurm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA filem: rsh (MCA v2.0, API v2.0, Component v1.4.1)
> MCA errmgr: default (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ess: env (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ess: hnp (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ess: singleton (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ess: slurm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ess: tool (MCA v2.0, API v2.0, Component v1.4.1)
> MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.4.1)
> MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.4.1)
>
>
> <config.log.gz>_______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users