Subject: Re: [MTT users] unable to pull tests
From: Ethan Mallove (ethan.mallove_at_[hidden])
Date: 2007-11-14 18:25:30
Do you have access to the ompi-tests repository?
What happens if you do this command outside of MTT?
$ svn export https://svn.open-mpi.org/svn/ompi-tests/trunk/onesided
You could also try using "http", instead of "https" in
your svn_url.
-Ethan
On Wed, Nov/14/2007 02:04:27PM, Karol Mroz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello everyone...
>
> I've been trying to get MTT set up to run tests on the SCTP BTL Brad
> Penoff and I (over at UBC) have developed. While I'm able to obtain a
> nightly tarball, build and install the Open MPI middleware, I'm unable
> to pull any tests for the various svn repositories. Currently I've tried
> pulling IBM and Onesided tests as shown in the sample
> ompi-core-template.ini file.
>
> Here is the output I see from the console when running with --verbose:
> - ----------------------------------------------------------------------
> *** MTT: ./mtt -f ../samples/ompi-core-template-kmroz.ini --verbose
> *** Reporter initializing
> *** Reporter initialized
> *** MPI get phase starting
> >> MPI get: [mpi get: ompi-nightly-trunk]
> Checking for new MPI sources...
> No new MPI sources
> *** MPI get phase complete
> *** MPI install phase starting
> >> MPI install [mpi install: gcc warnings]
> Installing MPI: [ompi-nightly-trunk] / [1.3a1r16706] / [gcc warnings]...
> Completed MPI install successfully
> Installing MPI: [ompi-nightly-trunk] / [1.3a1r16682] / [gcc warnings]...
> Completed MPI install successfully
> Installing MPI: [ompi-nightly-trunk] / [1.3a1r16723] / [gcc warnings]...
> Completed MPI install successfully
> *** MPI install phase complete
> *** Test get phase starting
> >> Test get: [test get: onesided]
> Checking for new test sources...
> - ----------------------------------------------------------------------
>
> As you can see, MTT seems to hang on 'Checking for new test sources.'
>
> I will attach a copy of the .ini file in hopes that someone may be able
> to point me in the right direction.
>
> Thanks in advance.
>
>
> ompi-core-template-kmroz.ini:
> - -----------------------------------------------------------------------
> # Copyright (c) 2006-2007 Cisco Systems, Inc. All rights reserved.
> # Copyright (c) 2006-2007 Sun Microystems, Inc. All rights reserved.
> #
>
> # Template MTT configuration file for Open MPI core testers. The
> # intent for this template file is to establish at least some loose
> # guidelines for what Open MPI core testers should be running on a
> # nightly basis. This file is not intended to be an exhaustive sample
> # of all possible fields and values that MTT offers. Each site will
> # undoubtedly have to edit this template for their local needs (e.g.,
> # pick compilers to use, etc.), but this file provides a baseline set
> # of configurations that we intend you to run.
>
> # OMPI core members will need to edit some values in this file based
> # on your local testing environment. Look for comments with "OMPI
> # Core:" for instructions on what to change.
>
> # Note that this file is artificially longer than it really needs to
> # be -- a bunch of values are explicitly set here that are exactly
> # equivalent to their defaults. This is mainly because there is no
> # reliable form of documentation for this ini file yet, so the values
> # here comprise a good set of what options are settable (although it
> # is not a comprehensive set).
>
> # Also keep in mind that at the time of this writing, MTT is still
> # under active development and therefore the baselines established in
> # this file may change on a relatively frequent basis.
>
> # The guidelines are as follows:
> #
> # 1. Download and test nightly snapshot tarballs of at least one of
> # the following:
> # - the trunk (highest preference)
> # - release branches (highest preference is the most recent release
> # branch; lowest preference is the oldest release branch)
> # 2. Run all 4 correctness test suites from the ompi-tests SVN
> # - trivial, as many processes as possible
> # - intel tests with all_tests_no_perf, up to 64 processes
> # - IBM, as many processes as possible
> # - IMB, as many processes as possible
> # 3. Run with as many different components as possible
> # - PMLs (ob1, dr)
> # - BTLs (iterate through sm, tcp, whatever high speed network(s) you
> # have, etc. -- as relevant)
>
> #======================================================================
> # Overall configuration
> #======================================================================
>
> [MTT]
>
> # OMPI Core: if you are not running in a scheduled environment and you
> # have a fixed hostfile for what nodes you'll be running on, fill in
> # the absolute pathname to it here. If you do not have a hostfile,
> # leave it empty. Example:
> # hostfile = /home/me/mtt-runs/mtt-hostfile
> # This file will be parsed and will automatically set a valid value
> # for &env_max_np() (it'll count the number of lines in the hostfile,
> # adding slots/cpu counts if it finds them). The "hostfile" value is
> # ignored if you are running in a recognized scheduled environment.
> hostfile =
>
> # OMPI Core: if you would rather list the hosts individually on the
> # mpirun command line, list hosts here delimited by whitespace (if you
> # have a hostfile listed above, this value will be ignored!). Hosts
> # can optionally be suffixed with ":num", where "num" is an integer
> # indicating how many processes may be started on that machine (if not
> # specified, ":1" is assumed). The sum of all of these values is used
> # for &env_max_np() at run time. Example (4 uniprocessors):
> # hostlist = node1 node2 node3 node4
> # Another example (4 2-way SMPs):
> # hostlist = node1:2 node2:2 node3:2 node4:2
> # The "hostlist" value is ignored if you are running in a scheduled
> # environment or if you have specified a hostfile.
> hostlist = localhost
>
> # OMPI Core: if you are running in a scheduled environment and want to
> # override the scheduler and set the maximum number of processes
> # returned by &env_max_procs(), you can fill in an integer here.
> max_np = 2
>
> # OMPI Core: Output display preference; the default width at which MTT
> # output will wrap.
> textwrap = 76
>
> # OMPI Core: After the timeout for a command has passed, wait this
> # many additional seconds to drain all output, and then kill it with
> # extreme prejiduce.
> drain_timeout = 5
>
> # OMPI Core: Whether this invocation of the client is a test of the
> # client setup itself. Specifically, this value should be set to true
> # (1) if you are testing your MTT client and/or INI file and do not
> # want the results included in normal reporting in the MTT central
> # results database. Results submitted in "trial" mode are not
> # viewable (by default) on the central database, and are automatically
> # deleted from the database after a short time period (e.g., a week).
> # Setting this value to 1 is exactly equivalent to passing "--trial"
> # on the MTT client command line. However, any value specified here
> # in this INI file will override the "--trial" setting on the command
> # line (i.e., if you set "trial = 0" here in the INI file, that will
> # override and cancel the effect of "--trial" on the command line).
> # trial = 0
>
> # OMPI Core: Set the scratch parameter here (if you do not want it to
> # be automatically set to your current working directory). Setting
> # this parameter accomplishes the same thing that the --scratch option
> # does.
> # scratch = &getenv("HOME")/mtt-scratch
>
> # OMPI Core: Set local_username here if you would prefer to not have
> # your local user ID in the MTT database
> local_username = kmroz
>
> # OMPI Core: --force can be set here, instead of at the command line.
> # Useful for a developer workspace in which it makes no sense to not
> # use --force
> # force = 1
>
> # OMPI Core: Specify a list of sentinel files that MTT will regularly
> # check for. If these files exist, MTT will exit more-or-less
> # immediately (i.e., after the current test completes) and report all
> # of its results. This is a graceful mechanism to make MTT stop right
> # where it is but not lose any results.
> # terminate_files = &getenv("HOME")/mtt-stop,&scratch_root()/mtt-stop
>
> # OMPI Core: Specify a default description string that is used in the
> # absence of description strings in the MPI install, Test build, and
> # Test run sections. The intent of this field is to record variable
> # data that is outside the scope, but has effect on the software under
> # test (e.g., firmware version of a NIC). If no description string is
> # specified here and no description strings are specified below, the
> # description data field is left empty when reported.
> # description = NIC firmware: &system("get_nic_firmware_rev")
>
> # OMPI Core: Specify a logfile where you want all MTT output to be
> # sent in addition to stdout / stderr.
> # logfile = /tmp/my-logfile.txt
>
> # OMPI Core: If you have additional .pm files for your own funclets,
> # you can have a comma-delimited list of them here. Note that each
> # .pm file *must* be a package within the MTT::Values::Functions
> # namespace. For example, having a Cisco.pm file must include the
> # line:
> #
> # package MTT::Values::Functions::Cisco;
> #
> # If this file contains a perl function named foo, you can invoke this
> # functlet as &Cisco::foo(). Note that funclet files are loaded
> # almost immediately, so you can use them even for other field values
> # in the MTT section.
> # funclet_files = /path/to/file1.pm, /path/to/file2.pm
>
> #----------------------------------------------------------------------
>
> [Lock]
> # The only module available is the MTTLockServer, and requires running
> # the mtt-lock-server executable somewhere. You can leave this
> # section blank and there will be no locking.
> #module = MTTLockServer
> #mttlockserver_host = hostname where mtt-lock-server is running
> #mttlockserver_port = integer port number of the mtt-lock-server
>
> #======================================================================
> # MPI get phase
> #======================================================================
>
> [MPI get: ompi-nightly-trunk]
> mpi_details = Open MPI
>
> module = OMPI_Snapshot
> ompi_snapshot_url = http://www.open-mpi.org/nightly/trunk
>
> [MPI install: gcc warnings]
> mpi_get = ompi-nightly-trunk
> save_stdout_on_success = 1
> merge_stdout_stderr = 0
> bitness = 32
>
> module = OMPI
> ompi_vpath_mode = none
> # OMPI Core: This is a GNU make option; if you are not using GNU make,
> # you'll probably want to delete this field (i.e., leave it to its
> # default [empty] value).
> ompi_make_all_arguments = -j 4
> ompi_make_check = 1
> # OMPI Core: You will likely need to update these values for whatever
> # compiler you want to use. You can pass any configure flags that you
> # want, including those that change which compiler to use (e.g., CC=cc
> # CXX=CC F77=f77 FC=f90). Valid compiler names are: gnu, pgi, intel,
> # ibm, kai, absoft, pathscale, sun. If you have other compiler names
> # that you need, let us know. Note that the compiler_name flag is
> # solely for classifying test results; it does not automatically pass
> # values to configure to set the compiler.
> ompi_compiler_name = gnu
> ompi_compiler_version = &get_gcc_version()
> ompi_configure_arguments = CFLAGS=-pipe --enable-picky --enable-debug
>
> #----------------------------------------------------------------------
>
> #======================================================================
> # MPI run details
> #======================================================================
>
> [MPI Details: Open MPI]
>
> # MPI tests
> exec = mpirun @hosts@ -np &test_np() @mca@ --prefix &test_prefix()
> &test_executable() &test_argv()
>
> # ORTE tests
> exec:rte = &test_executable() --host &env_hosts() --prefix
> &test_prefix() &test_argv()
>
> hosts = &if(&have_hostfile(), "--hostfile " . &hostfile(), \
> &if(&have_hostlist(), "--host " . &hostlist(), ""))
>
> # Example showing conditional substitution based on the MPI get
> # section name (e.g., different versions of OMPI have different
> # capabilities / bugs).
> mca = &enumerate( \
> "--mca btl sm,tcp,self_at_v1_1_mca_params@", \
> "--mca btl tcp,self_at_v1_1_mca_params@")
>
> # Boolean indicating IB connectivity
> is_up = &check_ipoib_connectivity()
>
> # Figure out which mca's to use
> mca = <<EOT
> &perl('
>
> # Return cached mca, if we have it
> if (defined(@mca)) {
> return \@mca;
> }
>
> my @hosts = split /\s+|,/, hostlist_hosts();
>
> if (scalar(@hosts) < 2) {
> push(@mca, "--mca btl self,sm");
> } else {
> if ($ib_up) {
> push(@mca, "--mca btl self,udapl");
> } else {
> push(@mca, "--mca btl self,tcp");
> }
> }
> return \@mca;
> ')
> EOT
>
> # OMPI v1.1 cannot handle heterogeneous numbers of TCP or OpenIB
> # interfaces within a single job. So restrict it to a finite number
> # that will be the same across all processes in the job (adjust for
> # your own site, of course -- this particular example is meaningless
> # if all nodes at your site have a homogeneous type and number of
> # network interfaces).
> v1_1_mca_params = &if(&eq(&mpi_get_name(), "ompi-nightly-v1.1"), \
> " --mca btl_tcp_if_include eth0 --mca oob_tcp_if_include eth0 --mca
> btl_openib_max_btls 1", "")
>
> #----------------------------------------------------------------------
> # WARNING: THIS DEFAULT after_each_exec STEP IS PRONE TO FAILURE!
> # Given that part of what we are testing is ORTE itself, using orterun
> # to launch something to cleanup can be problematic. We *HIGHLY*
> # recommend that you replace the after_each_exec section default value
> # below with something that your run-time system can performan
> # natively. For example, putting "srun -N $SLURM_NNODES killall -9
> # mpirun orted &test_executable()" works nicely on SLURM / Linux
> # systems -- assuming that your MTT run has all nodes exclusively to
> # itself (i.e., that the "killall" won't kill some legitimate jobs).
> #----------------------------------------------------------------------
>
> # A helper script is installed by the "OMPI" MPI Install module named
> # "mtt_ompi_cleanup.pl". This script is orterun-able and will kill
> # all rogue orteds on a node and whack any session directories.
> # Invoke via orterun just to emphasize that it is not an MPI
> # application. The helper script is installed in OMPI's bin dir, so
> # it'll automatically be found in the path (because OMPI's bin dir is
> # in the path).
>
> after_each_exec = <<EOT
> # We can exit if the test passed or was skipped (i.e., there's no need
> # to cleanup).
> if test "$MTT_TEST_RUN_RESULT" = "passed" -o "$MTT_TEST_RUN_RESULT" =
> "skipped"; then
> exit 0
> fi
>
> if test "$MTT_TEST_HOSTFILE" != ""; then
> args="--hostfile $MTT_TEST_HOSTFILE"
> elif test "$MTT_TEST_HOSTLIST" != ""; then
> args="--host $MTT_TEST_HOSTLIST"
> fi
> orterun $args -np $MTT_TEST_NP --prefix $MTT_TEST_PREFIX mtt_ompi_cleanup.pl
> EOT
>
>
> #======================================================================
> # Test get phase
> #======================================================================
>
> #[Test get: ibm]
> #module = SVN
> #svn_url = https://svn.open-mpi.org/svn/ompi-tests/trunk/ibm
> #svn_post_export = <<EOT
> #./autogen.sh
> #EOT
>
> [Test get: onesided]
> module = SVN
> svn_url = https://svn.open-mpi.org/svn/ompi-tests/trunk/onesided
> svn_post_export = <<EOT
> ./autogen.sh
> EOT
>
> #----------------------------------------------------------------------
>
>
> #======================================================================
> # Test build phase
> #======================================================================
>
> [Test build: ibm]
> test_get = ibm
> save_stdout_on_success = 1
> merge_stdout_stderr = 1
> stderr_save_lines = 100
>
> module = Shell
> shell_build_command = <<EOT
> ./configure CC=mpicc CXX=mpic++ F77=mpif77
> make
> EOT
>
> #[Test build: onesided]
> #test_get = onesided
> #save_stdout_on_success = 1
> #merge_stdout_stderr = 1
> #stderr_save_lines = 100
> # Have the onesided tests skip the OMPI 1.1 testing; MPI-2 one-sided
> # just plain doesn't work there and won't be fixed.
> #skip_mpi_get = ompi-nightly-v1.1
> # Can also have a skip_mpi_install for the same purpose (skip specific
> # installs)
>
> #module = Shell
> #shell_build_command = <<EOT
> #./configure
> #make
> #EOT
>
> #----------------------------------------------------------------------
>
>
> #======================================================================
> # Test Run phase
> #======================================================================
>
> [Test run: ibm]
> test_build = ibm
> pass = &and(&cmd_wifexited(), &eq(&cmd_wexitstatus(), 0))
> skipped = &and(&cmd_wifexited(), &eq(&cmd_wexitstatus(), 77))
> timeout = &max(30, &multiply(10, &test_np()))
> save_stdout_on_pass = 1
> merge_stdout_stderr = 1
> stdout_save_lines = 100
> np = &env_max_procs()
>
> specify_module = Simple
> Similar rationale to the intel test run section
> simple_first:tests = &find_executables("collective", "communicator", \
> "datatype", "dynamic",
> "environment", \
> "group", "info", "io", "onesided", \
> "pt2pt", "topology")
>
> # Similar rationale to the intel test run section
> simple_fail:tests = environment/abort environment/final
> simple_fail:pass = &and(&cmd_wifexited(), &ne(&cmd_wexitstatus(), 0))
> simple_fail:exclusive = 1
> simple_fail:np = &env_max_procs()
>
> #[Test run: onesided]
> #test_build = onesided
> #pass = &and(&cmd_wifexited(), &eq(&cmd_wexitstatus(), 0))
> #timeout = &max(30, &multiply(10, &test_np()))
> #save_stdout_on_pass = 1
> #merge_stdout_stderr = 1
> #stdout_save_lines = 100
> #np = &if(>(&env_max_procs(), 0), &step(2, &max(2, &env_max_procs()),
> 2), 2)
>
> #specify_module = Simple
> #simple_pass:tests = &cat("run_list")
>
>
> #----------------------------------------------------------------------
>
>
> #======================================================================
> # Reporter phase
> #======================================================================
>
> #[Reporter: IU database]
> #module = MTTDatabase
>
> #mttdatabase_realm = OMPI
> #mttdatabase_url = https://www.open-mpi.org/mtt/submit/
> # OMPI Core: Change this to be the username and password for your
> # submit user. Get this from the OMPI MTT administrator.
> #mttdatabase_username = you must set this value
> #mttdatabase_password = you must set this value
> # OMPI Core: Change this to be some short string identifying your
> # cluster.
> #mttdatabase_platform = you must set this value
>
> #----------------------------------------------------------------------
>
> # This is a backup for while debugging MTT; it also writes results to
> # a local text file
>
> #[Reporter: text file backup]
> module = TextFile
>
> textfile_filename = $phase-$section-$mpi_name-$mpi_version.txt
>
> textfile_summary_header = <<EOT
> hostname: &shell("hostname")
> uname: &shell("uname -a")
> who am i: &shell("who am i")
> EOT
>
> #textfile_summary_footer =
> #textfile_detail_header =
> #textfile_detail_footer =
>
> textfile_textwrap = 78
>
> - --
> Karol Mroz
> kmroz_at_[hidden]
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFHO3Druoug78g/Mz8RAiReAJ0fl9txgv+VjI2St+c88dN9iR8oBQCfead+
> qfz2+ijSfk8GMvy+z3Lbi04=
> =SHO4
> -----END PGP SIGNATURE-----
> _______________________________________________
> mtt-users mailing list
> mtt-users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users