Subject: Re: [MTT users] test suites and np question
From: Kawashima, Takahiro (t-kawashima_at_[hidden])
Date: 2016-05-18 21:05:43
Jeff,
I hope we do in the future. But currently we don't have enough
machine time and direct Internet connectivity (especially from
testing machines).
What type of test do you expect? Building Open MPI binary and
running some short test programs on some x86-64 machines are
realistic if the connectivity problem is resolved.
But running many or long test programs on many SPARC machines
per GitHub pull request is not realistic for us.
(daily or weekly run may be realistic)
> Great!
>
> Will you be able to also do some continuous integration type testing (i.e., run some basic tests for each Github pull request)? Josh/IBM is going to post some information about their Jenkins/Github pull request setup shortly.
>
>
> > On May 18, 2016, at 9:50 AM, Kawashima, Takahiro <t-kawashima_at_[hidden]> wrote:
> >
> > Jeff,
> >
> > Thank you. It's very useful information.
> > I'll plan our run based on your information.
> >
> > Once we (Fujitsu) come to be able to run the test suites regularly,
> > we'll prepare to upload the reports to the server and push our test suites.
> >
> > Thanks,
> > Takahiro Kawashima,
> > MPI development team,
> > Fujitsu
> >
> >>> Fujitsu started to try MTT + ompi-tests on our machines.
> >>> With the sample .ini file, we wrote our .ini file and some
> >>> test suites are run.
> >>>
> >>> I have two questions.
> >>>
> >>> (a) There are many test suites (directories) in ompi-tests.
> >>> ibm, onesided, sun, ...
> >>> Which test suites should I use to participate in
> >>> OMPI MTT daily/weekly run?
> >>
> >> The general guidance is: run as many tests as you have resources for. Meaning: we'll take any testing you can give. :-)
> >>
> >> Have a look in ompi-tests:cisco/mtt/community/*.ini and cisco/mtt/usnic/*.ini. Those are the ini files I use every night for Cisco usNIC-specific testing and community-wide testing. You can see the results of them in the MTT community reporter:
> >>
> >> http://mtt.open-mpi.org/
> >>
> >> I generally aim for about 20-24 hours of testing. It's a little fuzzy, because Cisco's MTT will only fire for a given version (I'm currently testing the master, v1.10, and v2.x branches) if there were new commits that day (i.e., if there's a new nightly snapshot tarball since the last run).
> >>
> >> If you run too many tests such that your testing is more than 24 hours, then your resources quickly get behind and you're testing tarballs from days ago -- and that's not too useful.
> >>
> >>> (b) What is the recommended `np` value (number of processes)?
> >>> Should I use the largest `np` I can run?
> >>
> >> Yes, subject to what I mentioned above: you want to aim for a total of ~24 hours of testing so that you can start the next cycle with the next night's snapshot tarball.
> >>
> >> You can pack this in however you want -- do lots of small-np-value tests and a few large-np-value tests (just to sanity test large np values, etc.), etc.
> >>
> >> You can also take into account that little development is done on the weekends. For example, you might want to aim for ~24 hours of testing on Monday-Thursday evenings, and then aim for a 3-day run on Friday evening (because there might not be new tarballs generated over the weekend).
> >>
> >>> Does it depend on test suites?
> >>
> >> Yes. Some test suites have upper-bounds on the number of processes they can run. IIRC, the Intel test suite, for example, can only run up to 64 processes (because of some hard-coded array sizes) unless you use a specific -D to compile it (that increases the size of these arrays).