On Mon, Apr 13, 2009 at 5:44 PM, Ethan Mallove <email@example.com>
Will this translate to something like
lib/MTT/Reporter/GoogleDatabase.pm? If we are to move away from the
current MTT Postgres database, we want to be able to submit results to
both the current MTT database and the new Google database during the
transition period. Having a GoogleDatabase.pm would make this easier.
I think we should keep both storage options: current postgress and datastore. The mtt changes will be minor to support datastore.
Due that fact that google appengine API (as well as datastore API) can be python or java only, we will create external scripts to manipulate datastore objects:
The mtt will dump test results in xml format. Then, we provide two python (or java?) scripts:
mtt-results-submit-to-datastore.py - script will be called at the end of mtt run and will read xml files, create objects and save to datastore
mtt-results-query.py - sample script to query datastore and generate some simple visual/tabular reports. It will serve as tutorial for howto access mtt data from scripts for reporting.
Later, we add another script to replace php web frontend. It will be hosted on google appengine machines and will provide web viewer for mtt results. (same way like index.php does today)
I'm confused here. Can't MTT be outfitted to report results of a
> b. mtt_save_to_db.py - script which will go over mtt scratch dir, find
> all xml files generated for every mtt phase, parse it and save to
> datastore, preserving test results relations,i.e. all test results will
> be grouped by mtt general info: mpi version, name, date, ....
> c. same script can scan, parse and save from xml files generated by
> wrapper scripts for non mtt based executions (fluent, ..)
I think we can enhance mtt to be not only mpi testing platform, but also to serve as mpi benchmarking platform. We can use datastore to keep mpi-based benchmarking results in the same manner like mtt does for testing results. (no changes to mtt required for that, it is just a side effect of using datastore to keep data of any type)
I don't see the models.py attachment.
> d. mtt_query_db.py script will be provided with basic query capabilities
> over proposed datastore object model. Most users will prefer writing
> custom sql-like select queries for fetching results.
> 3. Important notes:
> a. The single mtt client execution generates many result files, every
> generated file represents test phase. This file contains test results
> and can be characterized as a set of attributes with its values. Every
> test phase has its own attributes which are differ for different phases.
> For example: attributes for TestBuild phase has keys "compiler_name,
> compiler_version", the MPIInstall phase has attributes: prefix_dir,
> arch, ....
> Hence, most of the datastore objects representing phases of MTT* are
> derived from "db.Expando" model, which allows having dynamic attributes
> for its derived sub-classes.
> The attached is archive with a simple test for using datastore for mtt.
> Please see models.py file with proposed object model and comment.
I just sent original email with attachment, tell me if you want me to send it again.