Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI users] Segfault in mca_odls_default.so with > ~100 process.
From: Ralph Castain (rhc_at_[hidden])
Date: 2010-02-27 02:46:18


Yeah, the system won't like this. Your approach makes it look like you are launching 136 app_contexts. We currently only support up to 128 app_contexts. I don't think anyone anticipated somebody trying to use the system this way.

I can expand the number to something larger. Will have to see how big a change it requires (mostly a question of how many places are touched) before we know what release this might show up in.

On Feb 26, 2010, at 2:34 AM, Oliver Ford wrote:

>
> I am trying to run an MPI code across 136 processing using an appfile
> (attached), since every process needs to be run with a host/process
> dependent parameter.
>
> This whole system works wonderfully for up to around 100 processes but
> usually fails with a segfault, apparently in in mca_odls_default.so,
> during initialization.
> The attached appfile is an attempt at 136 processes. If I split the
> appfile into two, both halves will initialize OK and successfully pass
> an MPI_Barrier() (the program won't actually work without all 136 nodes,
> but I'm happy MPI is doing its job). Because both halves work, I think
> it has to be related to the number of processes - not a problem with a
> specific appfile entry or machine.
>
> The cluster I am running it on has openmpi-1.3.3 but I have also
> installed 1.4.1 from the website in my home dir and that does the same
> (and is from which the attached data comes).
>
> The actual segfault is:
> [jac-11:12300] *** Process received signal ***
> [jac-11:12300] Signal: Segmentation fault (11)
> [jac-11:12300] Signal code: Address not mapped (1)
> [jac-11:12300] Failing at address: 0x40
> [jac-11:12300] [ 0] [0x74640c]
> [jac-11:12300] [ 1] /home/oford/openmpi/lib/openmpi/mca_odls_default.so
> [0x8863d4]
> [jac-11:12300] [ 2] /home/oford/openmpi/lib/libopen-rte.so.0 [0x76ffe9]
> [jac-11:12300] [ 3]
> /home/oford/openmpi/lib/libopen-rte.so.0(orte_daemon_cmd_processor+0x2f6)
> [0x771b86]
> [jac-11:12300] [ 4] /home/oford/openmpi/lib/libopen-pal.so.0 [0x5d6ba8]
> [jac-11:12300] [ 5]
> /home/oford/openmpi/lib/libopen-pal.so.0(opal_event_loop+0x27) [0x5d6e47]
> [jac-11:12300] [ 6]
> /home/oford/openmpi/lib/libopen-pal.so.0(opal_progress+0xce) [0x5ca00e]
> [jac-11:12300] [ 7]
> /home/oford/openmpi/lib/libopen-rte.so.0(orte_plm_base_launch_apps+0x355)
> [0x7815f5]
> [jac-11:12300] [ 8] /home/oford/openmpi/lib/openmpi/mca_plm_rsh.so
> [0xc73d1b]
> [jac-11:12300] [ 9] mpirun [0x804a8f0]
> [jac-11:12300] [10] mpirun [0x8049ef6]
> [jac-11:12300] [11] /lib/libc.so.6(__libc_start_main+0xe5) [0x1406e5]
> [jac-11:12300] [12] mpirun [0x8049e41]
> [jac-11:12300] *** End of error message ***
> Segmentation fault
>
>
> The full output with '-d' and the config.log from the build of 1.4.1 are
> also attached.
>
> I don't know the exact setup of the network, but I can ask our sysadmin
> anything else that might help.
>
> Thanks in advance,
>
>
> Oliver Ford
>
> Culham Centre for Fusion Energy
> Oxford, UK
>
>
>
>
> -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-39-80115 Y 11 11 133 debug
> -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-26-81244 N 11 11 133 debug
> -np 1 --host batch-020 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-122-75993 N 11 11 133 debug
> -np 1 --host batch-037 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-157-15286 N 11 11 133 debug
> -np 1 --host batch-042 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-114-89529 N 11 11 133 debug
> -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-35-90257 N 11 11 133 debug
> -np 1 --host batch-020 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-151-56062 N 11 11 133 debug
> -np 1 --host batch-004 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-16-2723 N 11 11 133 debug
> -np 1 --host batch-003 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-156-65790 N 11 11 133 debug
> -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-198-63239 N 11 11 133 debug
> -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-105-12753 N 11 11 133 debug
> -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-12-25631 N 11 11 133 debug
> -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-196-35421 N 11 11 133 debug
> -np 1 --host batch-045 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-103-98246 N 11 11 133 debug
> -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-142-44009 N 11 11 133 debug
> -np 1 --host batch-044 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-117-30325 N 11 11 133 debug
> -np 1 --host batch-003 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-143-21739 N 11 11 133 debug
> -np 1 --host batch-042 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-112-64293 N 11 11 133 debug
> -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-57-11238 N 11 11 133 debug
> -np 1 --host batch-025 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-170-80831 N 11 11 133 debug
> -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-32-47578 N 11 11 133 debug
> -np 1 --host batch-027 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-146-27711 N 11 11 133 debug
> -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-25-36312 N 11 11 133 debug
> -np 1 --host batch-023 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-154-17110 N 11 11 133 debug
> -np 1 --host batch-023 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-136-3872 N 11 11 133 debug
> -np 1 --host batch-036 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-132-1105 N 11 11 133 debug
> -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-192-11991 N 11 11 133 debug
> -np 1 --host batch-045 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-115-41110 N 11 11 133 debug
> -np 1 --host batch-036 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-155-67802 N 11 11 133 debug
> -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-40-46953 N 11 11 133 debug
> -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-125-79465 N 11 11 133 debug
> -np 1 --host batch-022 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-128-79309 N 11 11 133 debug
> -np 1 --host batch-033 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-152-82320 N 11 11 133 debug
> -np 1 --host batch-026 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-158-68939 N 11 11 133 debug
> -np 1 --host batch-020 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-137-41369 N 11 11 133 debug
> -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-2-66663 N 11 11 133 debug
> -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-52-62955 N 11 11 133 debug
> -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-195-10991 N 11 11 133 debug
> -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-193-50774 N 11 11 133 debug
> -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-177-45516 N 11 11 133 debug
> -np 1 --host batch-032 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-171-96942 N 11 11 133 debug
> -np 1 --host batch-038 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-175-99761 N 11 11 133 debug
> -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-118-35065 N 11 11 133 debug
> -np 1 --host batch-014 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-17-49750 N 11 11 133 debug
> -np 1 --host batch-030 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-169-10604 N 11 11 133 debug
> -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-46-74041 N 11 11 133 debug
> -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-187-7833 N 11 11 133 debug
> -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-200-36312 N 11 11 133 debug
> -np 1 --host batch-027 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-159-37278 N 11 11 133 debug
> -np 1 --host batch-045 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-11-37181 N 11 11 133 debug
> -np 1 --host batch-013 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-149-11377 N 11 11 133 debug
> -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-179-64715 N 11 11 133 debug
> -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-44-46953 N 11 11 133 debug
> -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-130-64980 N 11 11 133 debug
> -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-13-39413 N 11 11 133 debug
> -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-34-46573 N 11 11 133 debug
> -np 1 --host batch-027 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-134-85354 N 11 11 133 debug
> -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-194-95760 N 11 11 133 debug
> -np 1 --host batch-013 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-133-18010 N 11 11 133 debug
> -np 1 --host batch-024 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-173-20377 N 11 11 133 debug
> -np 1 --host batch-013 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-160-43487 N 11 11 133 debug
> -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-106-65988 N 11 11 133 debug
> -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-18-18834 N 11 11 133 debug
> -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-180-19539 N 11 11 133 debug
> -np 1 --host batch-022 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-140-53706 N 11 11 133 debug
> -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-184-80559 N 11 11 133 debug
> -np 1 --host batch-023 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-126-6794 N 11 11 133 debug
> -np 1 --host batch-044 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-111-83886 N 11 11 133 debug
> -np 1 --host batch-005 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-150-26152 N 11 11 133 debug
> -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-29-81244 N 11 11 133 debug
> -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-30-5208 N 11 11 133 debug
> -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-185-82710 N 11 11 133 debug
> -np 1 --host batch-022 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-153-49288 N 11 11 133 debug
> -np 1 --host batch-026 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-131-48260 N 11 11 133 debug
> -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-21-85978 N 11 11 133 debug
> -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-186-67937 N 11 11 133 debug
> -np 1 --host batch-033 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-135-94983 N 11 11 133 debug
> -np 1 --host batch-044 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-107-10462 N 11 11 133 debug
> -np 1 --host batch-046 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-113-29689 N 11 11 133 debug
> -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-27-60637 N 11 11 133 debug
> -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-36-26250 N 11 11 133 debug
> -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-119-1554 N 11 11 133 debug
> -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-24-17859 N 11 11 133 debug
> -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-31-5480 N 11 11 133 debug
> -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-23-24746 N 11 11 133 debug
> -np 1 --host batch-005 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-138-53784 N 11 11 133 debug
> -np 1 --host jac-7 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-33-43810 N 11 11 133 debug
> -np 1 --host batch-005 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-121-38059 N 11 11 133 debug
> -np 1 --host batch-036 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-145-79515 N 11 11 133 debug
> -np 1 --host batch-042 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-109-66314 N 11 11 133 debug
> -np 1 --host batch-028 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-168-62760 N 11 11 133 debug
> -np 1 --host batch-001 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-166-35091 N 11 11 133 debug
> -np 1 --host jac-13 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-3-30335 N 11 11 133 debug
> -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-108-43856 N 11 11 133 debug
> -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-116-98652 N 11 11 133 debug
> -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-189-70592 N 11 11 133 debug
> -np 1 --host batch-004 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-148-12571 N 11 11 133 debug
> -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-38-79978 N 11 11 133 debug
> -np 1 --host batch-003 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-129-21483 N 11 11 133 debug
> -np 1 --host batch-033 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-124-3865 N 11 11 133 debug
> -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-191-79048 N 11 11 133 debug
> -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-188-93446 N 11 11 133 debug
> -np 1 --host jac-10 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-37-71686 N 11 11 133 debug
> -np 1 --host batch-037 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-162-83683 N 11 11 133 debug
> -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-1-48706 N 11 11 133 debug
> -np 1 --host jac-8 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-199-95838 N 11 11 133 debug
> -np 1 --host jac-3 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-178-23994 N 11 11 133 debug
> -np 1 --host batch-026 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-144-79515 N 11 11 133 debug
> -np 1 --host batch-008 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-15-46689 N 11 11 133 debug
> -np 1 --host jac-4 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-197-46966 N 11 11 133 debug
> -np 1 --host jac-2 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-41-82354 N 11 11 133 debug
> -np 1 --host jac-9 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-20-1112 N 11 11 133 debug
> -np 1 --host batch-039 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-176-80381 N 11 11 133 debug
> -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-101-91749 N 11 11 133 debug
> -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-10-96608 N 11 11 133 debug
> -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-139-62526 N 11 11 133 debug
> -np 1 --host batch-007 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-164-93486 N 11 11 133 debug
> -np 1 --host batch-018 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-174-37716 N 11 11 133 debug
> -np 1 --host batch-041 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-104-25353 N 11 11 133 debug
> -np 1 --host batch-031 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-172-7131 N 11 11 133 debug
> -np 1 --host batch-006 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-127-18397 N 11 11 133 debug
> -np 1 --host batch-034 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-147-65530 N 11 11 133 debug
> -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-14-40975 N 11 11 133 debug
> -np 1 --host batch-017 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-123-45216 N 11 11 133 debug
> -np 1 --host jac-6 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-190-27454 N 11 11 133 debug
> -np 1 --host batch-007 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-141-10541 N 11 11 133 debug
> -np 1 --host batch-043 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-102-77190 N 11 11 133 debug
> -np 1 --host batch-015 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-120-13012 N 11 11 133 debug
> -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-19-29582 N 11 11 133 debug
> -np 1 --host batch-008 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-161-20135 N 11 11 133 debug
> -np 1 --host batch-040 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-163-95832 N 11 11 133 debug
> -np 1 --host batch-019 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-167-60329 N 11 11 133 debug
> -np 1 --host jac-11 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-28-77824 N 11 11 133 debug
> -np 1 --host batch-034 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-165-57829 N 11 11 133 debug
> -np 1 --host jac-5 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-4-1153 N 11 11 133 debug
> -np 1 --host jac-12 /home/oford/java/mcServer/lgidmath/lgidmath /tmp/lgiStaging/mats-94280x5887-22-4961 N 11 11 133 debug
>
>
> [jac-11:15014] procdir: /tmp/openmpi-sessions-oford_at_jac-11_0/48099/0/0
> [jac-11:15014] jobdir: /tmp/openmpi-sessions-oford_at_jac-11_0/48099/0
> [jac-11:15014] top: openmpi-sessions-oford_at_jac-11_0
> [jac-11:15014] tmp: /tmp
> [jac-11:15014] mpirun: reset PATH: /home/oford/openmpi/bin:/home/oford/openmpi/bin:/home/oford/bin:/usr/lib/qt-3.3/bin:/usr/bin:/bin:/usr/local/depot/PGI/pgi-7.0/linux86/7.0/bin:/usr/local/bin:/jet/share/root/bin:/jet/share/bin:.:/u/sim/sh:/u/sim/cmg/sh:/home/oford/sh:/u/sim/sh/linux:/home/oford/sh/linux:/u/sim/edge2d/sh:/u/sim/edge2d/sh/linux:/home/oford/edge2d/sh:/home/oford/edge2d/sh/linux:/u/sim/jetto/sh
> [jac-11:15014] mpirun: reset LD_LIBRARY_PATH: /home/oford/openmpi/lib:/home/oford/openmpi/lib:/u/sim/mdsplus/default/lib/linux:/u/sim/jproc/default/lib/linux:/u/sim/eproc/default/lib/linux/fixed:/home/oford/oldXlib:/home/oford/hdf5:/jet/share/lib:/usr/local/lib
> [batch-020:17451] procdir: /tmp/openmpi-sessions-oford_at_batch-020_0/48099/0/2
> [batch-020:17451] jobdir: /tmp/openmpi-sessions-oford_at_batch-020_0/48099/0
> [batch-020:17451] top: openmpi-sessions-oford_at_batch-020_0
> [batch-020:17451] tmp: /tmp
> [batch-042:15862] procdir: /tmp/openmpi-sessions-oford_at_batch-042_0/48099/0/4
> [batch-042:15862] jobdir: /tmp/openmpi-sessions-oford_at_batch-042_0/48099/0
> [batch-042:15862] top: openmpi-sessions-oford_at_batch-042_0
> [batch-042:15862] tmp: /tmp
> [batch-023:09760] procdir: /tmp/openmpi-sessions-oford_at_batch-023_0/48099/0/19
> [batch-023:09760] jobdir: /tmp/openmpi-sessions-oford_at_batch-023_0/48099/0
> [batch-023:09760] top: openmpi-sessions-oford_at_batch-023_0
> [batch-023:09760] tmp: /tmp
> [batch-025:05770] procdir: /tmp/openmpi-sessions-oford_at_batch-025_0/48099/0/15
> [batch-025:05770] jobdir: /tmp/openmpi-sessions-oford_at_batch-025_0/48099/0
> [batch-025:05770] top: openmpi-sessions-oford_at_batch-025_0
> [batch-025:05770] tmp: /tmp
> [batch-037:17611] procdir: /tmp/openmpi-sessions-oford_at_batch-037_0/48099/0/3
> [batch-037:17611] jobdir: /tmp/openmpi-sessions-oford_at_batch-037_0/48099/0
> [batch-037:17611] top: openmpi-sessions-oford_at_batch-037_0
> [batch-037:17611] tmp: /tmp
> [jac-5:15183] procdir: /tmp/openmpi-sessions-oford_at_jac-5_0/48099/0/1
> [jac-5:15183] jobdir: /tmp/openmpi-sessions-oford_at_jac-5_0/48099/0
> [jac-5:15183] top: openmpi-sessions-oford_at_jac-5_0
> [jac-5:15183] tmp: /tmp
> [jac-9:22204] procdir: /tmp/openmpi-sessions-oford_at_jac-9_0/48099/0/5
> [jac-9:22204] jobdir: /tmp/openmpi-sessions-oford_at_jac-9_0/48099/0
> [jac-9:22204] top: openmpi-sessions-oford_at_jac-9_0
> [jac-9:22204] tmp: /tmp
> [batch-013:23865] procdir: /tmp/openmpi-sessions-oford_at_batch-013_0/48099/0/33
> [batch-013:23865] jobdir: /tmp/openmpi-sessions-oford_at_batch-013_0/48099/0
> [batch-013:23865] top: openmpi-sessions-oford_at_batch-013_0
> [batch-013:23865] tmp: /tmp
> [batch-022:15225] procdir: /tmp/openmpi-sessions-oford_at_batch-022_0/48099/0/23
> [batch-022:15225] jobdir: /tmp/openmpi-sessions-oford_at_batch-022_0/48099/0
> [batch-022:15225] top: openmpi-sessions-oford_at_batch-022_0
> [batch-022:15225] tmp: /tmp
> [batch-018:19925] procdir: /tmp/openmpi-sessions-oford_at_batch-018_0/48099/0/44
> [batch-018:19925] jobdir: /tmp/openmpi-sessions-oford_at_batch-018_0/48099/0
> [batch-018:19925] top: openmpi-sessions-oford_at_batch-018_0
> [batch-018:19925] tmp: /tmp
> [jac-4:06627] procdir: /tmp/openmpi-sessions-oford_at_jac-4_0/48099/0/37
> [jac-4:06627] jobdir: /tmp/openmpi-sessions-oford_at_jac-4_0/48099/0
> [jac-4:06627] top: openmpi-sessions-oford_at_jac-4_0
> [jac-4:06627] tmp: /tmp
> [batch-004:30088] procdir: /tmp/openmpi-sessions-oford_at_batch-004_0/48099/0/6
> [batch-004:30088] jobdir: /tmp/openmpi-sessions-oford_at_batch-004_0/48099/0
> [batch-004:30088] top: openmpi-sessions-oford_at_batch-004_0
> [batch-004:30088] tmp: /tmp
> [batch-046:00820] procdir: /tmp/openmpi-sessions-oford_at_batch-046_0/48099/0/8
> [batch-046:00820] jobdir: /tmp/openmpi-sessions-oford_at_batch-046_0/48099/0
> [batch-046:00820] top: openmpi-sessions-oford_at_batch-046_0
> [batch-046:00820] tmp: /tmp
> [batch-036:16988] procdir: /tmp/openmpi-sessions-oford_at_batch-036_0/48099/0/20
> [batch-036:16988] jobdir: /tmp/openmpi-sessions-oford_at_batch-036_0/48099/0
> [batch-036:16988] top: openmpi-sessions-oford_at_batch-036_0
> [batch-036:16988] tmp: /tmp
> [batch-030:25412] procdir: /tmp/openmpi-sessions-oford_at_batch-030_0/48099/0/31
> [batch-030:25412] jobdir: /tmp/openmpi-sessions-oford_at_batch-030_0/48099/0
> [batch-030:25412] top: openmpi-sessions-oford_at_batch-030_0
> [batch-030:25412] tmp: /tmp
> [jac-6:04101] procdir: /tmp/openmpi-sessions-oford_at_jac-6_0/48099/0/16
> [jac-6:04101] jobdir: /tmp/openmpi-sessions-oford_at_jac-6_0/48099/0
> [jac-6:04101] top: openmpi-sessions-oford_at_jac-6_0
> [jac-6:04101] tmp: /tmp
> [batch-006:16136] procdir: /tmp/openmpi-sessions-oford_at_batch-006_0/48099/0/12
> [batch-006:16136] jobdir: /tmp/openmpi-sessions-oford_at_batch-006_0/48099/0
> [batch-006:16136] top: openmpi-sessions-oford_at_batch-006_0
> [batch-006:16136] tmp: /tmp
> [jac-7:17379] procdir: /tmp/openmpi-sessions-oford_at_jac-7_0/48099/0/32
> [jac-7:17379] jobdir: /tmp/openmpi-sessions-oford_at_jac-7_0/48099/0
> [jac-7:17379] top: openmpi-sessions-oford_at_jac-7_0
> [jac-7:17379] tmp: /tmp
> [batch-003:23921] procdir: /tmp/openmpi-sessions-oford_at_batch-003_0/48099/0/7
> [batch-003:23921] jobdir: /tmp/openmpi-sessions-oford_at_batch-003_0/48099/0
> [batch-003:23921] top: openmpi-sessions-oford_at_batch-003_0
> [batch-003:23921] tmp: /tmp
> [batch-044:01101] procdir: /tmp/openmpi-sessions-oford_at_batch-044_0/48099/0/13
> [batch-044:01101] jobdir: /tmp/openmpi-sessions-oford_at_batch-044_0/48099/0
> [batch-044:01101] top: openmpi-sessions-oford_at_batch-044_0
> [batch-044:01101] tmp: /tmp
> [batch-039:02035] procdir: /tmp/openmpi-sessions-oford_at_batch-039_0/48099/0/42
> [batch-039:02035] jobdir: /tmp/openmpi-sessions-oford_at_batch-039_0/48099/0
> [batch-039:02035] top: openmpi-sessions-oford_at_batch-039_0
> [batch-039:02035] tmp: /tmp
> [batch-033:02863] procdir: /tmp/openmpi-sessions-oford_at_batch-033_0/48099/0/24
> [batch-033:02863] jobdir: /tmp/openmpi-sessions-oford_at_batch-033_0/48099/0
> [batch-033:02863] top: openmpi-sessions-oford_at_batch-033_0
> [batch-033:02863] tmp: /tmp
> [jac-8:26567] procdir: /tmp/openmpi-sessions-oford_at_jac-8_0/48099/0/18
> [jac-8:26567] jobdir: /tmp/openmpi-sessions-oford_at_jac-8_0/48099/0
> [jac-8:26567] top: openmpi-sessions-oford_at_jac-8_0
> [jac-8:26567] tmp: /tmp
> [batch-028:14068] procdir: /tmp/openmpi-sessions-oford_at_batch-028_0/48099/0/38
> [batch-028:14068] jobdir: /tmp/openmpi-sessions-oford_at_batch-028_0/48099/0
> [batch-028:14068] top: openmpi-sessions-oford_at_batch-028_0
> [batch-028:14068] tmp: /tmp
> [batch-019:32641] procdir: /tmp/openmpi-sessions-oford_at_batch-019_0/48099/0/48
> [batch-019:32641] jobdir: /tmp/openmpi-sessions-oford_at_batch-019_0/48099/0
> [batch-019:32641] top: openmpi-sessions-oford_at_batch-019_0
> [batch-019:32641] tmp: /tmp
> [batch-038:07716] procdir: /tmp/openmpi-sessions-oford_at_batch-038_0/48099/0/29
> [batch-038:07716] jobdir: /tmp/openmpi-sessions-oford_at_batch-038_0/48099/0
> [batch-038:07716] top: openmpi-sessions-oford_at_batch-038_0
> [batch-038:07716] tmp: /tmp
> [jac-2:05183] procdir: /tmp/openmpi-sessions-oford_at_jac-2_0/48099/0/22
> [jac-2:05183] jobdir: /tmp/openmpi-sessions-oford_at_jac-2_0/48099/0
> [jac-2:05183] top: openmpi-sessions-oford_at_jac-2_0
> [jac-2:05183] tmp: /tmp
> [batch-017:25718] procdir: /tmp/openmpi-sessions-oford_at_batch-017_0/48099/0/34
> [batch-017:25718] jobdir: /tmp/openmpi-sessions-oford_at_batch-017_0/48099/0
> [batch-017:25718] top: openmpi-sessions-oford_at_batch-017_0
> [batch-017:25718] tmp: /tmp
> [batch-026:09510] procdir: /tmp/openmpi-sessions-oford_at_batch-026_0/48099/0/25
> [batch-026:09510] jobdir: /tmp/openmpi-sessions-oford_at_batch-026_0/48099/0
> [batch-026:09510] top: openmpi-sessions-oford_at_batch-026_0
> [batch-026:09510] tmp: /tmp
> [batch-043:01668] procdir: /tmp/openmpi-sessions-oford_at_batch-043_0/48099/0/40
> [batch-043:01668] jobdir: /tmp/openmpi-sessions-oford_at_batch-043_0/48099/0
> [batch-043:01668] top: openmpi-sessions-oford_at_batch-043_0
> [batch-043:01668] tmp: /tmp
> [batch-008:17836] procdir: /tmp/openmpi-sessions-oford_at_batch-008_0/48099/0/41
> [batch-008:17836] jobdir: /tmp/openmpi-sessions-oford_at_batch-008_0/48099/0
> [batch-008:17836] top: openmpi-sessions-oford_at_batch-008_0
> [batch-008:17836] tmp: /tmp
> [jac-12:17562] procdir: /tmp/openmpi-sessions-oford_at_jac-12_0/48099/0/10
> [jac-12:17562] jobdir: /tmp/openmpi-sessions-oford_at_jac-12_0/48099/0
> [jac-12:17562] top: openmpi-sessions-oford_at_jac-12_0
> [jac-12:17562] tmp: /tmp
> [batch-005:03532] procdir: /tmp/openmpi-sessions-oford_at_batch-005_0/48099/0/36
> [batch-005:03532] jobdir: /tmp/openmpi-sessions-oford_at_batch-005_0/48099/0
> [batch-005:03532] top: openmpi-sessions-oford_at_batch-005_0
> [batch-005:03532] tmp: /tmp
> [batch-045:04261] procdir: /tmp/openmpi-sessions-oford_at_batch-045_0/48099/0/11
> [batch-045:04261] jobdir: /tmp/openmpi-sessions-oford_at_batch-045_0/48099/0
> [batch-045:04261] top: openmpi-sessions-oford_at_batch-045_0
> [batch-045:04261] tmp: /tmp
> [batch-001:22016] procdir: /tmp/openmpi-sessions-oford_at_batch-001_0/48099/0/39
> [batch-001:22016] jobdir: /tmp/openmpi-sessions-oford_at_batch-001_0/48099/0
> [batch-001:22016] top: openmpi-sessions-oford_at_batch-001_0
> [batch-001:22016] tmp: /tmp
> [batch-040:08361] procdir: /tmp/openmpi-sessions-oford_at_batch-040_0/48099/0/47
> [batch-040:08361] jobdir: /tmp/openmpi-sessions-oford_at_batch-040_0/48099/0
> [batch-040:08361] top: openmpi-sessions-oford_at_batch-040_0
> [batch-040:08361] tmp: /tmp
> [batch-007:05689] procdir: /tmp/openmpi-sessions-oford_at_batch-007_0/48099/0/43
> [batch-007:05689] jobdir: /tmp/openmpi-sessions-oford_at_batch-007_0/48099/0
> [batch-007:05689] top: openmpi-sessions-oford_at_batch-007_0
> [batch-007:05689] tmp: /tmp
> [batch-032:02059] procdir: /tmp/openmpi-sessions-oford_at_batch-032_0/48099/0/28
> [batch-032:02059] jobdir: /tmp/openmpi-sessions-oford_at_batch-032_0/48099/0
> [batch-032:02059] top: openmpi-sessions-oford_at_batch-032_0
> [batch-032:02059] tmp: /tmp
> [batch-031:01049] procdir: /tmp/openmpi-sessions-oford_at_batch-031_0/48099/0/45
> [batch-031:01049] jobdir: /tmp/openmpi-sessions-oford_at_batch-031_0/48099/0
> [batch-031:01049] top: openmpi-sessions-oford_at_batch-031_0
> [batch-031:01049] tmp: /tmp
> [batch-024:05301] procdir: /tmp/openmpi-sessions-oford_at_batch-024_0/48099/0/35
> [batch-024:05301] jobdir: /tmp/openmpi-sessions-oford_at_batch-024_0/48099/0
> [batch-024:05301] top: openmpi-sessions-oford_at_batch-024_0
> [batch-024:05301] tmp: /tmp
> [batch-034:24766] procdir: /tmp/openmpi-sessions-oford_at_batch-034_0/48099/0/46
> [batch-034:24766] jobdir: /tmp/openmpi-sessions-oford_at_batch-034_0/48099/0
> [batch-034:24766] top: openmpi-sessions-oford_at_batch-034_0
> [batch-034:24766] tmp: /tmp
> [jac-10:27707] procdir: /tmp/openmpi-sessions-oford_at_jac-10_0/48099/0/21
> [jac-10:27707] jobdir: /tmp/openmpi-sessions-oford_at_jac-10_0/48099/0
> [jac-10:27707] top: openmpi-sessions-oford_at_jac-10_0
> [jac-10:27707] tmp: /tmp
> [batch-014:30991] procdir: /tmp/openmpi-sessions-oford_at_batch-014_0/48099/0/30
> [batch-014:30991] jobdir: /tmp/openmpi-sessions-oford_at_batch-014_0/48099/0
> [batch-014:30991] top: openmpi-sessions-oford_at_batch-014_0
> [batch-014:30991] tmp: /tmp
> [jac-3:20114] procdir: /tmp/openmpi-sessions-oford_at_jac-3_0/48099/0/27
> [jac-3:20114] jobdir: /tmp/openmpi-sessions-oford_at_jac-3_0/48099/0
> [jac-3:20114] top: openmpi-sessions-oford_at_jac-3_0
> [jac-3:20114] tmp: /tmp
> [batch-015:01829] procdir: /tmp/openmpi-sessions-oford_at_batch-015_0/48099/0/9
> [batch-015:01829] jobdir: /tmp/openmpi-sessions-oford_at_batch-015_0/48099/0
> [batch-015:01829] top: openmpi-sessions-oford_at_batch-015_0
> [batch-015:01829] tmp: /tmp
> [batch-041:17727] procdir: /tmp/openmpi-sessions-oford_at_batch-041_0/48099/0/14
> [batch-041:17727] jobdir: /tmp/openmpi-sessions-oford_at_batch-041_0/48099/0
> [batch-041:17727] top: openmpi-sessions-oford_at_batch-041_0
> [batch-041:17727] tmp: /tmp
> [jac-13:26153] procdir: /tmp/openmpi-sessions-oford_at_jac-13_0/48099/0/26
> [jac-13:26153] jobdir: /tmp/openmpi-sessions-oford_at_jac-13_0/48099/0
> [jac-13:26153] top: openmpi-sessions-oford_at_jac-13_0
> [jac-13:26153] tmp: /tmp
> [batch-027:23746] procdir: /tmp/openmpi-sessions-oford_at_batch-027_0/48099/0/17
> [batch-027:23746] jobdir: /tmp/openmpi-sessions-oford_at_batch-027_0/48099/0
> [batch-027:23746] top: openmpi-sessions-oford_at_batch-027_0
> [batch-027:23746] tmp: /tmp
> [jac-11:15014] [[48099,0],0] node[0].name jac-11 daemon 0 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[1].name jac-5 daemon 1 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[2].name batch-020 daemon 2 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[3].name batch-037 daemon 3 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[4].name batch-042 daemon 4 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[5].name jac-9 daemon 5 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[6].name batch-004 daemon 6 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[7].name batch-003 daemon 7 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[8].name batch-046 daemon 8 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[9].name batch-015 daemon 9 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[10].name jac-12 daemon 10 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[11].name batch-045 daemon 11 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[12].name batch-006 daemon 12 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[13].name batch-044 daemon 13 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[14].name batch-041 daemon 14 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[15].name batch-025 daemon 15 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[16].name jac-6 daemon 16 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[17].name batch-027 daemon 17 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[18].name jac-8 daemon 18 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[19].name batch-023 daemon 19 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[20].name batch-036 daemon 20 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[21].name jac-10 daemon 21 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[22].name jac-2 daemon 22 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[23].name batch-022 daemon 23 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[24].name batch-033 daemon 24 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[25].name batch-026 daemon 25 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[26].name jac-13 daemon 26 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[27].name jac-3 daemon 27 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[28].name batch-032 daemon 28 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[29].name batch-038 daemon 29 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[30].name batch-014 daemon 30 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[31].name batch-030 daemon 31 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[32].name jac-7 daemon 32 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[33].name batch-013 daemon 33 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[34].name batch-017 daemon 34 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[35].name batch-024 daemon 35 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[36].name batch-005 daemon 36 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[37].name jac-4 daemon 37 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[38].name batch-028 daemon 38 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[39].name batch-001 daemon 39 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[40].name batch-043 daemon 40 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[41].name batch-008 daemon 41 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[42].name batch-039 daemon 42 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[43].name batch-007 daemon 43 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[44].name batch-018 daemon 44 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[45].name batch-031 daemon 45 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[46].name batch-034 daemon 46 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[47].name batch-040 daemon 47 arch ffca0200
> [jac-11:15014] [[48099,0],0] node[48].name batch-019 daemon 48 arch ffca0200
> [jac-11:15014] *** Process received signal ***
> [jac-11:15014] Signal: Segmentation fault (11)
> [jac-11:15014] Signal code: Address not mapped (1)
> [jac-11:15014] Failing at address: 0x302c5d70
> [jac-11:15014] [ 0] [0x9f540c]
> [jac-11:15014] [ 1] /home/oford/openmpi/lib/openmpi/mca_odls_default.so [0x52c3d4]
> [jac-11:15014] [ 2] /home/oford/openmpi/lib/libopen-rte.so.0 [0x22afe9]
> [jac-11:15014] [ 3] /home/oford/openmpi/lib/libopen-rte.so.0(orte_daemon_cmd_processor+0x2f6) [0x22cb86]
> [jac-11:15014] [ 4] /home/oford/openmpi/lib/libopen-pal.so.0 [0x129ba8]
> [jac-11:15014] [ 5] /home/oford/openmpi/lib/libopen-pal.so.0(opal_event_loop+0x27) [0x129e47]
> [jac-11:15014] [ 6] /home/oford/openmpi/lib/libopen-pal.so.0(opal_progress+0xce) [0x11d00e]
> [jac-11:15014] [ 7] /home/oford/openmpi/lib/libopen-rte.so.0(orte_plm_base_launch_apps+0x355) [0x23c5f5]
> [jac-11:15014] [ 8] /home/oford/openmpi/lib/openmpi/mca_plm_rsh.so [0x391d1b]
> [jac-11:15014] [ 9] mpirun [0x804a8f0]
> [jac-11:15014] [10] mpirun [0x8049ef6]
> [jac-11:15014] [11] /lib/libc.so.6(__libc_start_main+0xe5) [0x5b36e5]
> [jac-11:15014] [12] mpirun [0x8049e41]
> [jac-11:15014] *** End of error message ***
> [batch-041:17727] [[48099,0],14] routed:binomial: Connection to lifeline [[48099,0],0] lo[jac-6:04101] sess_dir_finalize: proc session dir not empty - leaving
> eaving
> batch-034:24766] sess_dir_finalize: proc session dir not empty - leaving
>
> [jac-12:17562] sess_dir_finalize: proc session dir not empty - leaving
> ng
>
>
> Package: Open MPI oford_at_jac-12 Distribution
> Open MPI: 1.4.1
> Open MPI SVN revision: r22421
> Open MPI release date: Jan 14, 2010
> Open RTE: 1.4.1
> Open RTE SVN revision: r22421
> Open RTE release date: Jan 14, 2010
> OPAL: 1.4.1
> OPAL SVN revision: r22421
> OPAL release date: Jan 14, 2010
> Ident string: 1.4.1
> Prefix: /home/oford/openmpi
> Configured architecture: i686-pc-linux-gnu
> Configure host: jac-12
> Configured by: oford
> Configured on: Thu Feb 25 22:46:23 GMT 2010
> Configure host: jac-12
> Built by: oford
> Built on: Thu Feb 25 23:03:10 GMT 2010
> Built host: jac-2
> C bindings: yes
> C++ bindings: yes
> Fortran77 bindings: yes (all)
> Fortran90 bindings: yes
> Fortran90 bindings size: small
> C compiler: gcc
> C compiler absolute: /usr/bin/gcc
> C++ compiler: g++
> C++ compiler absolute: /usr/bin/g++
> Fortran77 compiler: gfortran
> Fortran77 compiler abs: /usr/bin/gfortran
> Fortran90 compiler: gfortran
> Fortran90 compiler abs: /usr/bin/gfortran
> C profiling: yes
> C++ profiling: yes
> Fortran77 profiling: yes
> Fortran90 profiling: yes
> C++ exceptions: no
> Thread support: posix (mpi: no, progress: no)
> Sparse Groups: no
> Internal debug support: no
> MPI parameter check: runtime
> Memory profiling support: no
> Memory debugging support: no
> libltdl support: yes
> Heterogeneous support: no
> mpirun default --prefix: yes
> MPI I/O support: yes
> MPI_WTIME support: gettimeofday
> Symbol visibility support: yes
> FT Checkpoint support: no (checkpoint thread: no)
> MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.4.1)
> MCA memory: ptmalloc2 (MCA v2.0, API v2.0, Component v1.4.1)
> MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.4.1)
> MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.4.1)
> MCA carto: file (MCA v2.0, API v2.0, Component v1.4.1)
> MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.4.1)
> MCA timer: linux (MCA v2.0, API v2.0, Component v1.4.1)
> MCA installdirs: env (MCA v2.0, API v2.0, Component v1.4.1)
> MCA installdirs: config (MCA v2.0, API v2.0, Component v1.4.1)
> MCA dpm: orte (MCA v2.0, API v2.0, Component v1.4.1)
> MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.4.1)
> MCA allocator: basic (MCA v2.0, API v2.0, Component v1.4.1)
> MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: basic (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: inter (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: self (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: sm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: sync (MCA v2.0, API v2.0, Component v1.4.1)
> MCA coll: tuned (MCA v2.0, API v2.0, Component v1.4.1)
> MCA io: romio (MCA v2.0, API v2.0, Component v1.4.1)
> MCA mpool: fake (MCA v2.0, API v2.0, Component v1.4.1)
> MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.4.1)
> MCA mpool: sm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA pml: cm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA pml: csum (MCA v2.0, API v2.0, Component v1.4.1)
> MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.4.1)
> MCA pml: v (MCA v2.0, API v2.0, Component v1.4.1)
> MCA bml: r2 (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rcache: vma (MCA v2.0, API v2.0, Component v1.4.1)
> MCA btl: self (MCA v2.0, API v2.0, Component v1.4.1)
> MCA btl: sm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA btl: tcp (MCA v2.0, API v2.0, Component v1.4.1)
> MCA topo: unity (MCA v2.0, API v2.0, Component v1.4.1)
> MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.4.1)
> MCA osc: rdma (MCA v2.0, API v2.0, Component v1.4.1)
> MCA iof: hnp (MCA v2.0, API v2.0, Component v1.4.1)
> MCA iof: orted (MCA v2.0, API v2.0, Component v1.4.1)
> MCA iof: tool (MCA v2.0, API v2.0, Component v1.4.1)
> MCA oob: tcp (MCA v2.0, API v2.0, Component v1.4.1)
> MCA odls: default (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ras: slurm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.4.1)
> MCA rml: oob (MCA v2.0, API v2.0, Component v1.4.1)
> MCA routed: binomial (MCA v2.0, API v2.0, Component v1.4.1)
> MCA routed: direct (MCA v2.0, API v2.0, Component v1.4.1)
> MCA routed: linear (MCA v2.0, API v2.0, Component v1.4.1)
> MCA plm: rsh (MCA v2.0, API v2.0, Component v1.4.1)
> MCA plm: slurm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA filem: rsh (MCA v2.0, API v2.0, Component v1.4.1)
> MCA errmgr: default (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ess: env (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ess: hnp (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ess: singleton (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ess: slurm (MCA v2.0, API v2.0, Component v1.4.1)
> MCA ess: tool (MCA v2.0, API v2.0, Component v1.4.1)
> MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.4.1)
> MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.4.1)
>
>
> <config.log.gz>_______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users