From: Josh Hursey (jjhursey_at_[hidden])
Date: 2006-10-25 10:37:31
The discussion started with the bug characteristics of v1.2 versus
the trunk.
It seemed from the call that IU was the only institution that can
asses this via MTT as noone else spoke up. Since people were
interested in seeing things that were breaking I suggested that I
start forwarding the IU internal MTT reports (run nightly and weekly)
to the testing_at_open-mpi.org. This was meet by Brain insisting that it
would result in "thousands" of emails to the development list. I
clarified that it is only 3 - 4 messages a day from IU. However if
all other institutions do this then it would be a bunch of email
(where 'a bunch' would still be less than 'thousands'). That's how we
got to a 'we need a single summary presented to the group' comment.
It should be noted that we brought up IU sending to the 'testing_at_open-
mpi.org' list as a bandaid until MTT could do it better.
This single summary can be email or a webpage that people can check.
Rich said that he would prefer a webpage, and noone else really had a
comment. That got us talking about the current summary page that MTT
generates. Tim M mentioned that the current website is difficult to
figure out how to get the answers you need. I agree, it is hard
[usability] for someone to go to the summary page and answer the
question "So what failed from IU last night, and how does that differ
from Yesterday -- e.g., what regressed and progressed yesterday at
IU?". The website is flexible enough to due it, but having a couple
of basic summary pages would be nice for basic users. What that
should look like we can discuss further.
The IU group really likes the emails that we currently generate. A
plain-text summary of the previous run. I posted copies on the MTT
bug tracker here:
http://svn.open-mpi.org/trac/mtt/ticket/61
Currently we have not put the work in to aggregate the runs, so for
each ini file that we run we get 1 email to the IU group. This is
fine for the moment, but as we add the rest of the clusters and
dimensions in the testing matrix we will need MTT to aggregate the
results for us and generate such an email.
So I think the general feel of the discussion is that we need the
following from MTT:
- A 'basic' summary page providing answers to some general
frequently asked queries. The current interface is too advanced for
the current users.
- A summary email [in plain-text preferably] similar to the one
that IU generated showing an aggregation of the previous nights
results for (a) all reporters (b) my institution [so I can track them
down and file bugs].
- 1 email a day on the previous nights testing results.
Some relevant bugs currently in existence:
http://svn.open-mpi.org/trac/mtt/ticket/92
http://svn.open-mpi.org/trac/mtt/ticket/61
http://svn.open-mpi.org/trac/mtt/ticket/94
The other concern is that given the frequency of testing as bugs
appear from the testing someone needs to make sure the bug tracker is
updated. I think the group is unclear about how this is done. Meaning
when a MTT identifies a test as failed whom is responsible for
putting the bug in the bug tracker?
The obvious solution is the institution that identified the bug.
[Warning: My opinion] But then that becomes unwieldy for IU since we
have a large testing matrix, and would need to commit someone to
doing this everyday (and it may take all day to properly track a set
of bugs). Also this kind of punishes an institution for testing more
instead of providing incentive to test.
------ Page Break -- Context switch ------
In case you all want to know what we are doing here at IU. I attached
to this email our planed MTT testing matrix. Currently we have BigRed
and Odin running the complete matrix less the BLACS tests. Wotan and
Thor will come online as we get more resources to support them.
In order to do such a complex testing matrix we have various .ini
files that we use. And since some of the dimensions in the matrix are
large we break some of the tests into a couple .ini files that are
submitted concurrently to have them run in a reasonable time.
Questions? Thoughts?
-- Josh
On Oct 25, 2006, at 8:37 AM, Jeff Squyres wrote:
> Looking over Len's minutes from yesterday, I see that there was a
> bunch of discussion about MTT on the OMPI teleconf yesterday, but
> neither Ethan nor I were there to be a part of it. :-\
>
> I couldn't make much sense from Len's minutes:
>
> -----
> - having some trouble with MTT config, so will try to look more
> closely at some of these failures
> - instead of e-mails sending them to the testing at MTT list
> - plenty of internal IU e-mail, better to have one summary e-mail
> each day
> - cannot send a summary
> - send to mtt list and digest it
> - or you can just file bugs
> - can't use mtt web site to get the info
> -----
>
> What is IU requesting? Who can't use the MTT web site to get info?
> What info are you trying to get / why can't you get it?
>
> Should we have a teleconf about MTT stuff?
>
> I'm on travel and unavailable all today, but have time tomorrow
> (Thurs).
>
> --
> Jeff Squyres
> Server Virtualization Business Unit
> Cisco Systems
>
> _______________________________________________
> mtt-users mailing list
> mtt-users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users