Subject: Re: [MTT users] MTT Error on SLES11
From: Rafael Folco (rfolco_at_[hidden])
Date: 2009-04-08 16:05:30
Ethan, Jeff,
Sorry for the perl confusion, nevermind...
Below is attached the log and ini files, and also how I run MTT.
Thanks,
Rafael
client/mtt -d -v -p -f openmpi.ini 2>&1 |tee test.log
...
*** Run test phase starting
>> Test run [trivial]
Evaluating: trivial
Found a match! trivial [trivial
Evaluating: Simple
>> Running with [openmpi-1.2.8] / [1.2.8] / [openmpi-1.2.8]
Found MPI details: [mpi details: open mpi]
Using [mpi details: open mpi] with [MPI Install: openmpi-1.2.8]
Evaluating: # We can exit if the test passed or was skipped (i.e.,
there's
no need
# to cleanup).
if test "$MTT_TEST_RUN_RESULT" = "passed" -o "$MTT_TEST_RUN_RESULT" =
"skipped"; then
exit 0
fi
if test "$MTT_TEST_HOSTFILE" != ""; then
args="--hostfile $MTT_TEST_HOSTFILE"
elif test "$MTT_TEST_HOSTLIST" != ""; then
args="--host $MTT_TEST_HOSTLIST"
fi
orterun $args -np $MTT_TEST_NP --prefix $MTT_TEST_PREFIX
mtt_ompi_cleanup.pl
Got final exec: mpirun &if(&have_hostfile(), "&join("--hostfile ",
"&hostfile()")", "&if(&have_hostlist(), "&join("--host ",
"&hostlist()")", "")") -np &test_np() --mca btl openib,self --debug
--prefix &test_prefix() &test_executable() &test_argv()
chdir /tmp/ompi-core-testers/installs/dLS2/tests/trivial/test_get__trivial
Evaluating: require MTT::Test::Specify::Simple
Evaluating: $ret = &MTT::Test::Specify::Simple::Specify(@args)
Evaluating: &find_executables(".")
Got name: find_executables
Got args: "."
_do: $ret = MTT::Values::Functions::find_executables(".")
&find_executables got .
&find_exectuables returning: ./c_ring ./f77_ring ./f90_hello ./cxx_ring
./f77_hello ./cxx_hello ./f90_ring ./c_hello
*** ERROR: Module aborted: MTT::Test::Specify::Simple:Specify: Can't use
string ("8") as an ARRAY ref while "strict refs" in use at
/tmp/ompi-core-testers/lib/MTT/Values.pm line 75.
On Wed, 2009-04-08 at 14:15 -0400, Ethan Mallove wrote:
> On Wed, Apr/08/2009 11:36:05AM, Rafael Folco wrote:
> > Well, I took a look at /tmp/ompi-core-testers/lib/MTT/Values.pm line 75.
> >
> > This piece of code looks wrong to me:
> >
> > if ($#{@$ret} < 0) {
> >
> > $ret references an array
> > @$ret points to the first element of this array
> > $# returns the number of elements
> >
> > So this line is trying to count elements of the first element??! Doesn't
> > make sense. Correct me if I am wrong, what am I missing here ?
> >
> > "if ($#{$ret} < 0) {" would be correct, without @.
> >
> > I believe "strict refs" has been forced somewhere on SLES11... I also
> > tried on other distro and it works fine.
>
> How do perl -V differ between the two distros?
>
> I can not reproduce the error on a SLES 10 machine.
>
> Could you run MTT with the --debug option and send the output with the
> line 75 perl error? That might help me determine which INI param is
> responsible for the error.
>
> -Ethan
>
> >
> > Thanks,
> >
> > Rafael
> >
> > On Tue, 2009-04-07 at 15:53 -0300, Rafael Folco wrote:
> > > Hi,
> > >
> > > I'm trying to run MTT on SLES11, but I am getting an error message
> > > during the RUN phase and I can't figure out what is the problem.
> > >
> > > *** ERROR: Module aborted: MTT::Test::Specify::Simple:Specify: Can't use
> > > string ("183") as an ARRAY ref while "strict refs" in use at
> > > /tmp/ompi-core-testers/lib/MTT/Values.pm line 75.
> > >
> > > What I could see was that this error is nothing specific to any
> > > particular test, it happens at certain points during the RUN phase.
> > > Also, the BUILD phase has been completed successfully for all tests.
> > >
> > > Anybody have already seen this? Any thoughts ?
> > >
> > > Thanks in advance.
> > >
> > > Rafael
> > >
> > >
> >
> > --
> > Rafael Folco
> > OpenHPC / Test Lead
> > IBM Linux Technology Center
> > E-Mail: rfolco_at_[hidden]
> >
> > _______________________________________________
> > mtt-users mailing list
> > mtt-users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
-- Rafael Folco OpenHPC / Test Lead IBM Linux Technology Center E-Mail: rfolco_at_[hidden]
#======================================================================
# Overall configuration
#======================================================================
[MTT]
# OMPI Core: if you are not running in a scheduled environment and you
# have a fixed hostfile for what nodes you'll be running on, fill in
# the absolute pathname to it here. If you do not have a hostfile,
# leave it empty. Example:
# hostfile = /home/me/mtt-runs/mtt-hostfile
# This file will be parsed and will automatically set a valid value
# for &env_max_np() (it'll count the number of lines in the hostfile,
# adding slots/cpu counts if it finds them). The "hostfile" value is
# ignored if you are running in a recognized scheduled environment.
hostfile =
# OMPI Core: if you would rather list the hosts individually on the
# mpirun command line, list hosts here delimited by whitespace (if you
# have a hostfile listed above, this value will be ignored!). Hosts
# can optionally be suffixed with ":num", where "num" is an integer
# indicating how many processes may be started on that machine (if not
# specified, ":1" is assumed). The sum of all of these values is used
# for &env_max_np() at run time. Example (4 uniprocessors):
# hostlist = node1 node2 node3 node4
# Another example (4 2-way SMPs):
# hostlist = node1:2 node2:2 node3:2 node4:2
# The "hostlist" value is ignored if you are running in a scheduled
# environment or if you have specified a hostfile.
hostlist = cluster-ib-6 cluster-ib-7
# OMPI Core: if you are running in a scheduled environment and want to
# override the scheduler and set the maximum number of processes
# returned by &env_max_procs(), you can fill in an integer here.
max_np =
# OMPI Core: Output display preference; the default width at which MTT
# output will wrap.
textwrap = 76
# OMPI Core: After the timeout for a command has passed, wait this
# many additional seconds to drain all output, and then kill it with
# extreme prejiduce.
drain_timeout = 5
# OMPI Core: Whether this invocation of the client is a test of the
# client setup itself. Specifically, this value should be set to true
# (1) if you are testing your MTT client and/or INI file and do not
# want the results included in normal reporting in the MTT central
# results database. Results submitted in "trial" mode are not
# viewable (by default) on the central database, and are automatically
# deleted from the database after a short time period (e.g., a week).
# Setting this value to 1 is exactly equivalent to passing "--trial"
# on the MTT client command line. However, any value specified here
# in this INI file will override the "--trial" setting on the command
# line (i.e., if you set "trial = 0" here in the INI file, that will
# override and cancel the effect of "--trial" on the command line).
trial = 1
# OMPI Core: Set the scratch parameter here (if you do not want it to
# be automatically set to your current working directory). Setting
# this parameter accomplishes the same thing that the --scratch option
# does.
# scratch = &getenv("HOME")/mtt-scratch
# OMPI Core: Set local_username here if you would prefer to not have
# your local user ID in the MTT database
local_username = root
# OMPI Core: --force can be set here, instead of at the command line.
# Useful for a developer workspace in which it makes no sense to not
# use --force
force = 1
logfile = &getenv("HOME")/mtt.log
#======================================================================
# MPI run details
#======================================================================
[MPI Details: Open MPI]
exec = mpirun @hosts@ -np &test_np() --mca btl openib,self --debug --prefix &test_prefix() &test_executable() &test_argv()
# Yes, all these quotes are necessary. Don't mess with them!
hosts = &if(&have_hostfile(), "&join("--hostfile ", "&hostfile()")", \
"&if(&have_hostlist(), "&join("--host ", "&hostlist()")", "")")
# Example showing conditional substitution based on the MPI get
# section name (e.g., different versions of OMPI have different
# capabilities / bugs).
mca = &enumerate( \
"--mca btl sm,tcp,self_at_v1_1_mca_params@", \
"--mca btl tcp,self_at_v1_1_mca_params@")
# OMPI v1.1 cannot handle heterogeneous numbers of TCP or OpenIB
# interfaces within a single job. So restrict it to a finite number
# that will be the same across all processes in the job (adjust for
# your own site, of course -- this particular example is meaningless
# if all nodes at your site have a homogeneous type and number of
# network interfaces).
v1_1_mca_params = &if(&eq("&mpi_get_name()", "ompi-nightly-v1.1"), \
" --mca btl_tcp_if_include eth0 --mca oob_tcp_if_include eth0 --mca btl_openib_max_btls 1", "")
#----------------------------------------------------------------------
# WARNING: THIS DEFAULT after_each_exec STEP IS PRONE TO FAILURE!
# Given that part of what we are testing is ORTE itself, using orterun
# to launch something to cleanup can be problematic. We *HIGHLY*
# recommend that you replace the after_each_exec section default value
# below with something that your run-time system can performan
# natively. For example, putting "srun -N $SLURM_NNODES killall -9
# mpirun orted &test_executable()" works nicely on SLURM / Linux
# systems -- assuming that your MTT run has all nodes exclusively to
# itself (i.e., that the "killall" won't kill some legitimate jobs).
#----------------------------------------------------------------------
# A helper script is installed by the "OMPI" MPI Install module named
# "mtt_ompi_cleanup.pl". This script is orterun-able and will kill
# all rogue orteds on a node and whack any session directories.
# Invoke via orterun just to emphasize that it is not an MPI
# application. The helper script is installed in OMPI's bin dir, so
# it'll automatically be found in the path (because OMPI's bin dir is
# in the path).
after_each_exec = <<EOT
# We can exit if the test passed or was skipped (i.e., there's no need
# to cleanup).
if test "$MTT_TEST_RUN_RESULT" = "passed" -o "$MTT_TEST_RUN_RESULT" = "skipped"; then
exit 0
fi
if test "$MTT_TEST_HOSTFILE" != ""; then
args="--hostfile $MTT_TEST_HOSTFILE"
elif test "$MTT_TEST_HOSTLIST" != ""; then
args="--host $MTT_TEST_HOSTLIST"
fi
orterun $args -np $MTT_TEST_NP --prefix $MTT_TEST_PREFIX mtt_ompi_cleanup.pl
EOT
[MPI get: openmpi-1.2.8]
mpi_details = openmpi-1.2.8
alreadyinstalled_dir = /usr/lib64/mpi/gcc/openmpi
module = AlreadyInstalled
[MPI install: openmpi-1.2.8]
module = Analyze::OMPI
mpi_get = openmpi-1.2.8
#======================================================================
# Test get phase
#======================================================================
[Test get: trivial]
module = Trivial
#----------------------------------------------------------------------
[Test get: ibm]
module = SVN
svn_username = ******
svn_password = ******
svn_url = https://svn.open-mpi.org/svn/ompi-tests/trunk/ibm
svn_post_export = <<EOT
./autogen.sh
EOT
#----------------------------------------------------------------------
[Test get: onesided]
module = SVN
svn_username = ******
svn_password = ******
svn_url = https://svn.open-mpi.org/svn/ompi-tests/trunk/onesided
svn_post_export = <<EOT
./autogen.sh
EOT
#----------------------------------------------------------------------
[Test get: mpicxx]
module = SVN
svn_username = ******
svn_password = ******
svn_url = https://svn.open-mpi.org/svn/ompi-tests/trunk/cxx-test-suite
svn_post_export = <<EOT
./autogen.sh
EOT
#----------------------------------------------------------------------
[Test get: imb]
module = SVN
svn_username = ******
svn_password = ******
svn_url = https://svn.open-mpi.org/svn/ompi-tests/trunk/IMB_2.3
#----------------------------------------------------------------------
[Test get: netpipe]
module = SVN
svn_username = ******
svn_password = ******
svn_url = https://svn.open-mpi.org/svn/ompi-tests/trunk/NetPIPE_3.6.2
#======================================================================
# Test build phase
#======================================================================
[Test build: trivial]
test_get = trivial
save_stdout_on_success = 1
merge_stdout_stderr = 1
stderr_save_lines = 100
merge_stdout_stderr = 1
stderr_save_lines = 100
module = Trivial
#----------------------------------------------------------------------
[Test build: ibm]
test_get = ibm
save_stdout_on_success = 1
merge_stdout_stderr = 1
stderr_save_lines = 100
module = Shell
shell_build_command = <<EOT
./configure CC=mpicc CXX=mpic++ F77=mpif77
make
EOT
#----------------------------------------------------------------------
[Test build: onesided]
test_get = onesided
save_stdout_on_success = 1
merge_stdout_stderr = 1
stderr_save_lines = 100
# Have the onesided tests skip the OMPI 1.1 testing; MPI-2 one-sided
# just plain doesn't work there and won't be fixed.
skip_mpi_get = ompi-nightly-v1.1, ompi-nightly-trunk, ompi-nightly-v1.2
skip_mpi_install = ompi-nightly-v1.1, ompi-nightly-trunk, ompi-nightly-v1.2
# Can also have a skip_mpi_install for the same purpose (skip specific
# installs)
module = Shell
shell_build_command = <<EOT
./configure
make
EOT
#----------------------------------------------------------------------
[Test build: mpicxx]
test_get = mpicxx
save_stdout_on_success = 1
merge_stdout_stderr = 1
skip_mpi_get = ompi-nightly-v1.1, ompi-nightly-trunk, ompi-nightly-v1.2
skip_mpi_install = ompi-nightly-v1.1, ompi-nightly-trunk, ompi-nightly-v1.2
module = Shell
shell_build_command = <<EOT
./configure CC=mpicc CXX=mpic++
make
EOT
#----------------------------------------------------------------------
[Test build: imb]
test_get = imb
save_stdout_on_success = 1
merge_stdout_stderr = 1
stderr_save_lines = 100
skip_mpi_get = ompi-nightly-v1.1, ompi-nightly-trunk, ompi-nightly-v1.2
skip_mpi_install = ompi-nightly-v1.1, ompi-nightly-trunk, ompi-nightly-v1.2
module = Shell
shell_build_command = <<EOT
cd src
make clean IMB-MPI1
EOT
#----------------------------------------------------------------------
[Test build: netpipe]
test_get = netpipe
save_stdout_on_success = 1
merge_stdout_stderr = 1
stderr_save_lines = 100
skip_mpi_get = ompi-nightly-v1.1, ompi-nightly-trunk, ompi-nightly-v1.2
skip_mpi_install = ompi-nightly-v1.1, ompi-nightly-trunk, ompi-nightly-v1.2
module = Shell
shell_build_command = <<EOT
make mpi
EOT
#======================================================================
# Test Run phase
#======================================================================
[Test run: trivial]
test_build = trivial
pass = &and(&cmd_wifexited(), &eq(&cmd_wexitstatus(), 0))
timeout = &test_np()
save_stdout_on_pass = 1
merge_stdout_stderr = 1
stdout_save_lines = 100
np = &env_max_procs()
specify_module = Simple
simple_only:tests = &find_executables(".")
#----------------------------------------------------------------------
[Test run: ibm]
test_build = ibm
pass = &and(&cmd_wifexited(), &eq(&cmd_wexitstatus(), 0))
skipped = &and(&cmd_wifexited(), &eq(&cmd_wexitstatus(), 77))
timeout = &max(30, &multiply(10, &test_np()))
save_stdout_on_pass = 1
merge_stdout_stderr = 1
stdout_save_lines = 100
np = &env_max_procs()
specify_module = Simple
# Similar rationale to the intel test run section
simple_first:tests = &find_executables("collective", "communicator", \
"datatype", "dynamic", "environment", \
"group", "info", "io", "onesided", \
"pt2pt", "topology")
# Similar rationale to the intel test run section
simple_fail:tests = environment/abort environment/final
simple_fail:pass = &and(&cmd_wifexited(), &ne(&cmd_wexitstatus(), 0))
simple_fail:exclusive = 1
simple_fail:np = &env_max_procs()
#----------------------------------------------------------------------
[Test run: onesided]
test_build = onesided
pass = &and(&cmd_wifexited(), &eq(&cmd_wexitstatus(), 0))
timeout = &max(30, &multiply(10, &test_np()))
save_stdout_on_pass = 1
merge_stdout_stderr = 1
stdout_save_lines = 100
np = &if(>(&env_max_procs(), 0), &step(2, &max(2, &env_max_procs()), 2), 2)
specify_module = Simple
simple_pass:tests = &cat("run_list")
#----------------------------------------------------------------------
[Test run: mpicxx]
test_build = mpicxx
pass = &and(&cmd_wifexited(), &eq(&cmd_wexitstatus(), 0))
timeout = &max(30, &multiply(10, &test_np()))
save_stdout_on_pass = 1
merge_stdout_stderr = 1
argv = &if(&eq("&mpi_get_name()", "ompi-nightly-v1.1"), "-nothrow", "")
np = &env_max_procs()
specify_module = Simple
simple_pass:tests = src/mpi2c++_test
#----------------------------------------------------------------------
[Test run: imb correctness]
test_build = imb
pass = &and(&cmd_wifexited(), &eq(&cmd_wexitstatus(), 0))
timeout = &max(1800, &multiply(50, &test_np()))
save_stdout_on_pass = 1
merge_stdout_stderr = 1
stdout_save_lines = 100
np = &env_max_procs()
specify_module = Simple
simple_only:tests = src/IMB-MPI1
#----------------------------------------------------------------------
[Test run: imb performance]
test_build = imb
pass = &eq(&cmd_wexitstatus(), 0)
timeout = -1
save_stdout_on_pass = 1
# Ensure to leave this value as "-1", or performance results could be lost!
stdout_save_lines = -1
merge_stdout_stderr = 1
np = &env_max_procs()
argv = -npmin &test_np() &enumerate("PingPong", "PingPing", "Sendrecv", "Exchange", "Allreduce", "Reduce", "Reduce_scatter", "Allgather", "Allgatherv", "Alltoall", "Bcast", "Barrier")
specify_module = Simple
analyze_module = IMB
simple_pass:tests = src/IMB-MPI1
#----------------------------------------------------------------------
[Test run: netpipe]
test_build = netpipe
pass = &eq(&cmd_wexitstatus(), 0)
timeout = -1
save_stdout_on_pass = 1
# Ensure to leave this value as "-1", or performance results could be lost!
stdout_save_lines = -1
merge_stdout_stderr = 1
# NetPIPE is ping-pong only, so we only need 2 procs
np = 2
specify_module = Simple
analyze_module = NetPipe
simple_pass:tests = NPmpi
#======================================================================
# Reporter phase
#======================================================================
# This is a backup for while debugging MTT; it also writes results to
# a local text file
[Reporter: text file backup]
module = TextFile
textfile_filename = $phase-$section-$mpi_name-$mpi_version.txt
textfile_summary_header = <<EOT
hostname: &shell("hostname")
uname: &shell("uname -a")
who am i: &shell("who am i")
EOT
textfile_summary_footer =
textfile_detail_header =
textfile_detail_footer =
textfile_textwrap = 78