Open MPI logo

MTT Devel Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all MTT Devel mailing list

Subject: Re: [MTT devel] MTToGDS
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2010-02-04 13:56:16


Somehow I *TOTALLY* missed this email from late September. YOW! :-(

I am therefore just now trying to get up to speed.

Many thanks, Mike! This stuff looks awesome, and I'm excited to start using it.

I have a few questions...

1. It looks like we have a separate username/password login system for the appspot to submit MTT results. Is there a reason we're not using regular google accounts for authentication? I thought that was part of the strength of appspot -- users just got regular google account, and we could add them to a whitelist in our appspot app. That way, Google handles all the account maintenance stuff for us (username, email address, password, ...etc.). And all we do is ship off a "is this username/password combo valid?" kind of query to Google during login/authentication (or use their cookie-based authentication scheme thingy so that our app never even sees the password, which is better).

2. I think that I have been submitting some mpi install and test build results to the DB. How do I see these results in the db?

3. It looks like the ClusterInfo.pm stuff makes a bunch of assumptions about the target node (like how to launch on the target node via ssh, what commands and devices are available over there, etc.). Can we generalize that a bit?

4. The name of the Reporter module is MTTGDS.pm -- can we rename to GDS.pm? (the "MTT" is redundant, IMHO)

5. It looks like MTTGDS.pm caches all of its results to submit during Finalize. Is there a reason for not submitting during the MTT run?

6. I'm trying some simple runs right now with an "already installed" MTT kind of setup. I've successfully gotten through mpi get, mpi install, and test get, but when I run the test build phase (which builds the "trivial" section ok), I get the following during MTTGDS finalize:

*** ERROR: Module aborted: MTT::Reporter::MTTGDS:Finalize: Runtime Error at
    /home/jsquyres/svn/mtt/lib/MTT/Reporter/MTTGDS.pm line 772, <HANDLE>
    line 80.

Any idea how that can happen / how to fix it?

On Sep 29, 2009, at 3:21 PM, Mike Dubman wrote:

> Hello guys and gals,
>
> We have completed development and testing of Google DataStore support in MTT and are glad to submit it for community tests.
>
>
>
> New Files:
>
> ========
>
> The following new files were added to support GDS inside MTT:
>
> 1. client/bquery.pl
>
> Perl-based GDS client, provides basic DB querying/fetching capabilities. It creates resultset (files in YAML format) from user-provided sql-like query
>
> 2. client/breport.pl
>
> Perl-based report tool, creates excel reports from yaml files, generated by bquery.pl tool.
>
> 3. client/custom_launchers/
>
> For brave only: custom launchers for non-standard HPC, mpi-based applications
>
> 4. lib/MTT/Reporter/MTTGDS.pm
>
> GDS Reporter, saves mtt results to GDS (see samples/gds-demo.ini for configuration examples)
>
> 5. lib/MTT/Utils/ClusterInfo.pm
>
> Helper library to gather node hw/sw configuration information which is saved in GDS together with tests results.
>
> 6. New TestResults analyzers for HPC applications:
>
> lib/MTT/test/Analyze/Performance/Fluent.pm
>
> lib/MTT/test/Analyze/Performance/HPCC.pm
>
> lib/MTT/test/Analyze/Performance/HPLGDS.pm
>
> lib/MTT/test/Analyze/Performance/OpenFoam.pm
>
> lib/MTT/test/Analyze/Performance/PamCrash.pm
>
> 7. samples/gds-demo.ini
>
> Example of howto configure GDS in MTT and run bquery/breport tools at the end of MTT session
>
> 8. server/gds/
>
> GDS backend part, which is running at Google and providing Object to YAML, YAML to Object translation service as well as helper code for bquery.pl DB client.
>
> 9. docs/gds/
>
> Various documentation
>
>
>
> Known Issues and Limitations:
>
> ======================
>
> * lib/MTT/Utils/ClusterInfo.pm uses "sudo" command to gather node`s hardware information.
>
> * When using client/custom_launchers/ to run tests, it is impossible to kill the test application when timeout reached.
>
>
>
> How to start using MTToGDS:
>
> ======================
>
> * Contact Jeff to provide you with GDS login/password which is needed for querying/saving to DB (http://open-mpi-mtt.appspot.com)
>
> * See samples/gds-demo.ini for configuration examples as well as for DB querying and reports generation.
>
> * Read Google GQL syntax documentation and use it with bquery.pl in order to query&fetch objects from GDB.
>
> * The following perl modules are required for all MTToGDS components:
> libYAML
> YAML::Syck
> YAML::XS
>
> for breport:
> GD::Graph
> Spreadsheet::WriteExcel
>
> You can install it on linux systems with yum as following:
> yum install perl-libyaml perl-YAML-Syck perl-YAML-XS perl-GD-Graph perl-Spreadsheet-WriteExcel
>
> Special Thanks to:
>
> ==============
>
> Igor Ivanov, Andrew Senin, Alexander Alekhin from Argus-Cv.com for they contribution in developing and testing of this feature!
>
> Regards
>
> Mike
>
> _______________________________________________
> mtt-devel mailing list
> mtt-devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel

-- 
Jeff Squyres
jsquyres_at_[hidden]