On Tue, Apr 14, 2009 at 11:50 PM, Ethan Mallove <ethan.mallove_at_[hidden]>wrote:
> On Tue, Apr/14/2009 09:27:14PM, Mike Dubman wrote:
> > On Tue, Apr 14, 2009 at 5:04 PM, Jeff Squyres <jsquyres_at_[hidden]>
> > On Apr 13, 2009, at 2:08 PM, Mike Dubman wrote:
> > Hello Ethan,
> > Sorry for joining the discussion late... I was on travel last week
> > that always makes me waaay behind on my INBOX. *:-(
> > On Mon, Apr 13, 2009 at 5:44 PM, Ethan Mallove <
> > wrote:
> > Will this translate to something like
> > lib/MTT/Reporter/GoogleDatabase.pm? *If we are to move away from
> > current MTT Postgres database, we want to be able to submit
> results to
> > both the current MTT database and the new Google database during
> > transition period. Having a GoogleDatabase.pm would make this
> > I think we should keep both storage options: current postgress and
> > datastore. The mtt changes will be minor to support datastore.
> > Due that fact that google appengine API (as well as datastore API)
> > be python or java only, we will create external scripts to
> > datastore objects:
> > Ah, good point (python/java not perl). *But I think that
> > lib/MTT/Reporter/GoogleDataStore.pm could still be a good thing --
> > have invested a lot of time/effort into getting our particular mtt
> > clients setup just the way we want them, setting up INI files,
> > submitting to batch schedulers, etc.
> > A GoogleDataStore.pm reporter could well fork/exec a python/java
> > executable to do the actual communication/storing of the data,
> > *More below.
> > completely agree, once we have external python/java/cobol scripts to
> > manipulate GDS objects, we should wrap it by perl and call from MTT in
> > same way like it works today for submitting to the postgress.
> > *
> > The mtt will dump test results in xml format. Then, we provide two
> > python (or java?) scripts:
> > mtt-results-submit-to-datastore.py - script will be called at the
> > of mtt run and will read xml files, create objects and save to
> > datastore
> > Could be pretty easy to have a Reporter/GDS.pm (I keep making that
> > filename shorter, don't I? :-) ) that simply invokes the
> > mtt-result-submit-to-datastore.pt script on the xml that it dumped
> > that particular test.
> > Specifically: I do like having partial results submitted while my
> > tests are running. *Cisco's testing cycle is about 24 hours, but
> > of tests are finishing all the time, so it's good to see those
> > without having to wait the full 24 hours before anything shows up.
> > guess that's my only comment on the idea of having a script that
> > traverses the MTT scratch to find / submit everything -- I'd prefer
> > we kept the same Reporter idea and used an underlying .py script to
> > submit results as they become ready.
> > Is this do-able?
> > sounds good, we should introduce some guid (like pid) for mtt session,
> > where all mtt results generated by this session will be referring to
> > guid.* Later we use this guid to submit partial results as they become
> > ready and connect it to the appropriate mtt session object (see
> > mtt-results-query.py - sample script to query datastore and
> > some simple visual/tabular reports. It will serve as tutorial for
> > howto access mtt data from scripts for reporting.
> > Later, we add another script to replace php web frontend. It will
> > hosted on google appengine machines and will provide web viewer
> > mtt results. (same way like index.php does today)
> > Sounds good.
> > > * * *b. mtt_save_to_db.py - script which will go over mtt
> > dir, find
> > > * * *all xml files generated for every mtt phase, parse it and
> > to
> > > * * *datastore, preserving test results relations,i.e. all test
> > results will
> > > * * *be grouped by mtt general info: mpi version, name, date,
> > >
> > > * * *c. same script can scan, parse and save from xml files
> > generated by
> > > * * *wrapper scripts for non mtt based executions (fluent, ..)
> > I'm confused here. *Can't MTT be outfitted to report results of a
> > Fluent run?
> > I think we can enhance mtt to be not only mpi testing platform,
> > also to serve as mpi benchmarking platform. We can use datastore
> > keep mpi-based benchmarking results in the same manner like mtt
> > for testing results. (no changes to mtt required for that, it is
> > a side effect of using datastore to keep data of any type)
> > I think that Ethan was asking was: can't MTT run Fluent and then use
> > normal Reporter mechanism to report the results into whatever
> > data store we have? *(postgres or GDS)
> > ahhh, okie, i see.
> > Correct me if Im wrong, the current mtt implementation allows
> > way of executing mpi test:
> > /path/to/mpirun <mpirun options> <test>
> > Many mpi based applications have embedded MPI libraries and
> > way to start it, one should set env variable to point to desired mpi
> > installation or pass it as cmd line argument, for example:
> > for fluent:
> > export OPENMPI_ROOT=/path/to/openmpi
> > fluent <cmd line args>
> We'd probably want a special "MPI details" INI section to run Fluent,
> [MPI Details: Fluent]
> exec = fluent @fluent_args@
> > for pamcrash:
> > pamworld -np 2 -mpidir=/path/to/openmpi/dir ....
> Ditto for pamcrash.
> > Im not sure if it is possible to express that execution semantic in
> > ini file. Please suggest.
> > So far, it seems that such executions can be handled externally from
> > but using same object model.
> MTT supports the following INI parameters:
> * setenv
> * prepend_path
> * env_module
> * env_importer
> > *
> > I can see the value of both sides -- a) using the MTT client as the
> > gateway to *all* data storage, or b) making MTT but one (possibly of
> > many) tools that can write into the GDS. *a) certainly is more
> > attractive towards having a common data format back in GDS such that
> > single web tool is capable of reporting from the data and being able
> > make conherent sense out of the data (vs. 3rd party tools that put
> > back in GDS that may not be in exactly the same format / layout and
> > therefore our web reporter may not be able to make sense out of the
> > and report on it).
> > I think that having a Reporter/GDS.pm that system()'s the back-end
> > python script gives the best of both worlds -- the MTT client can
> > [continue to] submit results in the normal way, but there's also a
> > standalone script that can submit results from external tool runs
> > manually running Fluent, parsing the results, and submitting to our
> > GDS). *And hopefully the back-end python script will enforce a
> > structure to the data that is submitted so that all tools -- MTT and
> > 3rd party tools -- adhere to the same format and the reporter can
> > therefore report on it coherently.
> > agree. (a) is a preferred form. (b) can be used for tools that cannot
> > called from mtt.
> > *
> > For the attachment...
> > I can "sorta read" python, but I'm not familiar with its intricacies
> > its internal APIs.
> > - models.py: looks good. *I don't know if *all* the fields we have
> > listed here; it looks fairly short to me. *Did you attempt to
> > all of the fields we submit through the various phases in Reporter
> > there, or did you intentionally leave some out? *(I honestly haven't
> > checked; it just "feels short" to me compared to our SQL schema).
> > I listed only some of the fields in every object representing specific
> > test result source (called phase in mtt language). This is because
> > test result source object is derived from python provided db.Expando
> > class. This gives us great flexibility, like adding dynamic attributes
> > every objects, for example:
> > obj = new MttBuildPhaseResult()
> > obj.my_favorite_dynamic_key = "hello"
> > obj.my_another_dynamic_key = 7
> > So, we can have all phase attributes in the phase object without
> > it in the *sql schema way*. Also we can query object model by these
> > dynamic keys.
> > *
> It looks like model.py doesn't have the daisy chain of inheritance
> that the SQL schema requires.
> Shouldn't RunTestPhase back-reference the MPIInstallPhase,
> TestBuildPhase, and TestSession phase? E.g., we might need to look at
> the configure arguments that are keyed to a given test run.
> you are right, will add it to the model. Every phase object will have a
> reference to other relevant phase objects, i.e.
RunTestPhase -> MPIInstallPhase
RunTestPhase -> TestBuildPhase
*Phase -> TestSession
sounds good? Will go over sql schema and try to track additional relations.
> > --> meta question: is it in the zen of GDS to not have too many
> > fields like you would in SQL? *I.e., if you want to do an operation
> > GDS that you
> > as far as it seems now, gds creates indexes automatically and also
> > provides API to define indexes manually.
> > would typically use an SQL index field for, is the idea that you
> > do a map/reduce to select the data instead of an index field?
> > yep. seems correct.
> > *
> > - start_datastore.sh: hmm. *This script seems to imply that the
> > datastore is *local*! *Don't we have to HTTP submit the results to
> > Google? *More specifically: what is dev_appserver.py? *Is that,
> > perchance, just a local proxy agent that will end up submitting our
> > to $datastore_path, which actually resides at Google? *Do we have to
> > a specific google username/URL to submit (and query) results?
> > You need to download google`s sdk (dev_appserver.py is a part of it).
> > order to develop for gds you* run your code inside sdk locally, and
> > feel comfortable with it - you upload it to the google cluster. In
> > to run attached example, you need to download sdk, put it in the
> > dir hierarchy:
> > somedir/sdk
> > somedir/vbench-dev
> > and run start_datastore.sh, which will run local instance of GDS on
> > machine.Then in another shell you need to run vbech-dev.py, which
> > simulates mtt client accessing GDS, storing some objects in according
> > proposed models and then running some sql-like quires to fetch and
> > manipulate results.
> > see
> > - there's no comments in vbench-dev.py -- can you explain what's
> > on in there? *Can you explain how we would use these scripts?
> > This is a mtt simulator, it implements google appengine API to receive
> > HTTP requests and call appropriate callbacks. (there is a map of
> > urls to callbacks).
> > The main callback (which intercepts http GET requests to specific URL)
> > runs the test code which creates objects defined in models.py, groups
> > test results into MTTSession and they run some queries to access
> > previously created objects.
> > The real mtt client will use URL pointing to MTT python code running
> > google`s cluster, and use near same code to create/query/manipulate
> > objects defined in models.py.
> > *
> > - it *looks* like these scripts are for storing data out in the GDS.
> > *Have you looked at the querying side? *Do we know that storing data
> > the form you listed in models.py are easily retrievable in the ways
> > we want? *E.g., can you mock up queries that resemble the queries we
> > currently have in our web-based query system today, just to show
> > storing the data in this way will actually allow us to do the kinds
> > queries that we want to do?
> > I think vbench-dev.py shows some querying capabilities for stored
> > there are many ways to query objects by object CLASS and Attributes.
> > see many examples here:
> > see
> > for more querying examples we can use.
> > *
> > In short: I think I'm missing much of the back-story / rationale of
> > the scripts in your tarball work / are to be used.
> > BTW -- if it's useful to have a teleconference about this kind of
> > I can host a WebEx meeting. *WebEx has local dialins around the
> > including Israel...
> > sure, what about next week?
> > *
> > regards
> > Mike
> > --
> > Jeff Squyres
> > Cisco Systems
> > References
> > Visible links
> > . mailto:jsquyres_at_[hidden]
> > . mailto:ethan.mallove_at_[hidden]
> > . http://submit-to-datastore.pt/
> > .
> > .
> > _______________________________________________
> > mtt-devel mailing list
> > mtt-devel_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel
> mtt-devel mailing list