Subject: Re: [MTT users] To run MTT on our cluster
From: Jaison Mulerikkal (j.mulerikkal_at_[hidden])
Date: 2014-06-12 17:49:41
Hi Ralph,
You are pointing in the right direction.
If you can grant us "read access to ompi-tests without granting write permissions to the entire OMPI repo” will be perfect. We got better confidence in you than anyone else with choosing and maintaining those tests! We shall dumb the results to a our own folder and analyse using some in-house custom tools.
Please let me know how can I get read access to those ompi-tests.
Jaison Mulerikkal, PhD
New Zealand eScience Infrastructure
The University of Auckland
e: j.mulerikkal_at_[hidden]
p: +64 9 923 2194 (internal ext: 82194)
w: www.jaison.me
On 13/06/2014, at 3:02 am, Ralph Castain <rhc_at_[hidden]> wrote:
> Hi Jaison
>
> Sounds to me like reporting the results into our database wouldn't be particularly useful for either side. We don't have great visualization support for what you have in mind - we just show failures etc, and the reporting tool is really geared towards testing of nightly tarballs. I suspect you'd be disappointed in it for this purpose.
>
> It sounds to me like your best solution would be to use MTT to run your tests, but tell it to dump the results into a file for you to examine or post-process with a custom tool as the output format is pretty machine-friendly.
>
> As for the tests themselves, I think I'd leave that up to you. As Jeff said, we just collected these from various places and there isn't anything "proprietary" about them. I will admit that we did (a) integrate them into MTT a bit, and (b) do a fair amount of cleanup as some of the tests were just plain wrong. We haven't pushed any of those corrections back to the source, so we *might* have something more correct? Hard to say as I don't believe we have ever gone back to the source and checked for updates on their side.
>
> If you want access to them, I'd have to defer to Jeff to see if we could perhaps grant read access to ompi-tests without granting write permissions to the entire OMPI repo. I don't think that would be appropriate in this situation.
>
> HTH
> Ralph
>
>
> On Jun 10, 2014, at 4:40 PM, Jaison Mulerikkal <j.mulerikkal_at_[hidden]> wrote:
>
>> Hi Ralph and Jeff,
>>
>> Thanks for our replies and your willingness to help.
>>
>> Our intentions are as follows:
>>
>> There are lots or customers/users that use different variants of MPI and specifically OpenMPI in our clusters. They run various interesting things with MPI/OpenMPI. Sometimes they end up with some issues as they run their programs. We spend lot of time to figure out what their problems are. Sometimes it could be because of the way we have set up our MPI modules.
>>
>> So we need a reliable set of MPI tests/programs to run nightly or at some frequency to see that all our MPI settings are okay. I thought MTT will be a good tool to do that. I assume that you have the right set of testing programs and a way to present the results.
>>
>> My manager thinks that he don’t mind sending the results of this particular set of tests to an external server (as yours), as long as we got access to those test results and we can watch it on some visualisation tool, list, etc.
>>
>> We don't intend to test nightly Open MPI tarballs. We are using easy-build to install our libraries/apps. I think it may conflict with easy-build workflow. Or is there any way to integrate it with easy-build?
>>
>> We would like to get access to a comprehensive list of tests, that can verify the MPI/OpenMPI health of our systems.
>>
>> Hope this clarification helps. Now what do you suggest?
>>
>> Jaison Mulerikkal, PhD
>> New Zealand eScience Infrastructure
>> The University of Auckland
>> e: j.mulerikkal_at_[hidden]
>> p: +64 9 923 2194 (internal ext: 82194)
>> w: www.jaison.me
>>
>>
>>
>>
>>
>> On 11/06/2014, at 11:19 am, Ralph Castain <rhc_at_[hidden]> wrote:
>>
>>> I had a chance to think about this some more, and I'm wondering if this falls into some new category of "member". Personally, I would really welcome getting the MTT results from another site as it would expand coverage of various environments, something we are beginning to lack. So even if Jaison et al aren't going to contribute code, contributing test result data is a valuable contribution in itself.
>>>
>>> I think this fits to your question about testing nightly OMPI tarballs. If Jaison can run tests with our tarballs and tell us the results, and that meets his needs to characterize/validate their cluster, then it seems like both sides win.
>>>
>>>
>>> On Jun 10, 2014, at 7:46 AM, Jeff Squyres (jsquyres) <jsquyres_at_[hidden]> wrote:
>>>
>>>> Sorry for the delay in replying -- I was at the MPI Fourm last week, and my available time for email was limited.
>>>>
>>>> So I think there's three issues on the table:
>>>>
>>>> 1. Access to the closed ompi-tests SVN repo
>>>> 2. Access to some example MTT .ini files
>>>> 3. Submitting data to mtt.open-mpi.org
>>>>
>>>> Forgive me -- as you'll see below, you're raising a corner case that we have not previously encountered...
>>>>
>>>> Can I ask what your intentions are? We typically haven't given access to #1 or #3 except to developer organizations in the Open MPI community itself (signing the IP agreement is the first step). This is not to say that we won't -- it's just that you happen to be the first organization who has expressed a more-than-passing interest in #1 and #3 :-), but you do not appear to be interested in becoming a contributor or member Open MPI organization. We haven't really thought about this case before.
>>>>
>>>> I'd like to reiterate two points:
>>>>
>>>> 1. You don't need to sign the Open MPI IP agreement to use MTT; MTT is open source software that anyone can freely use under the BSD license.
>>>> 2. Most of the tests in the ompi-tests repo are publicly available; we just don't have redistribution rights (which is why our repo is closed). We have augmented these publicly-available test suites with a bunch of home-grown tests, too, but I don't know if they're generally useful outside of Open MPI (i.e., some test for specific Open MPI behavior).
>>>>
>>>> Also, I don't think I was specific about this before, but submitting data to mtt.open-mpi.org is intended for tracking the nightly development tarballs. It's not meant as a general MTT database hosting platform for individual organizations' internal testing.
>>>>
>>>> So I guess my questions are:
>>>>
>>>> 1. Do you really want/need access to our SVN repo, or do you just want/need a list of the test suites that you can download from the internet yourself?
>>>>
>>>> 2. Do you intend to test nightly Open MPI tarballs?
>>>>
>>>> Thanks for your patience in helping us think through the issues; this is what happens when you happen to be first. :-)
>>>>
>>>>
>>>>
>>>> On Jun 2, 2014, at 8:22 PM, Ralph Castain <rhc_at_[hidden]> wrote:
>>>>
>>>>>
>>>>> On Jun 2, 2014, at 4:48 PM, Jaison Mulerikkal <j.mulerikkal_at_[hidden]> wrote:
>>>>>
>>>>>> Hi Ralph and others,
>>>>>>
>>>>>> Thanks for your replies. Very useful and encouraging.
>>>>>>
>>>>>> I have posted the signed copy of Corporate Contributor License Agreement to the address you have given.
>>>>>>
>>>>>> Hope you can give us access to the tests-repo now.
>>>>>
>>>>> I'll have to let Jeff do this - he's off to the MPI Forum, and so it may not happen for a few days
>>>>>
>>>>>> Would like to get the configuration file (.ini) that you use and to get a copy of “runmtt.sh” that you run. That will help me understand how it is actually used in cluster.
>>>>>
>>>>> See attached
>>>>>
>>>>>>
>>>>>> I will think about setting up a database for ourselves. After talking to my manager, it seems that he is not fuzzy about using your external database, especially in the beginning. So we may begin with just doing that and later setup something for ourselves. Thanks again for your help.
>>>>>
>>>>> No problem - my .ini is setup to report to our database.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Jaison Mulerikkal, PhD
>>>>>> New Zealand eScience Infrastructure
>>>>>> The University of Auckland
>>>>>> e: j.mulerikkal_at_[hidden]
>>>>>> p: +64 9 923 2194 (internal ext: 82194)
>>>>>> w: www.jaison.me
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 29/05/2014, at 5:04 pm, Ralph Castain <rhc_at_[hidden]> wrote:
>>>>>>
>>>>>>>
>>>>>>> On May 28, 2014, at 8:29 PM, Jaison Mulerikkal <j.mulerikkal_at_[hidden]> wrote:
>>>>>>>
>>>>>>>> Hi Ralph,
>>>>>>>
>>>>>>> Hello - and welcome!
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks for your previous clarification.
>>>>>>>>
>>>>>>>> At our organisation (www.nesi.org.nz) we would like to use MTT for MPI heath check of our clusters.
>>>>>>>
>>>>>>> I actually know some people there! Small world - almost came to work there a few years ago, and have a standing promise to come visit :-)
>>>>>>>
>>>>>>>> It it okay use MTT, legally, if we sign up to the following agreement.
>>>>>>>> http://www.open-mpi.org/community/contribute/corporate.php. Whom shall I send this doc?
>>>>>>>
>>>>>>> There should be an address on the form - it gets sent to:
>>>>>>>
>>>>>>> Center for Research in Extreme Scale Technologies (CREST)
>>>>>>> ATTN: Andrew Lumsdaine,
>>>>>>> Open MPI 3rd Party Contribution Agreements
>>>>>>> 2719 E. 10th Street
>>>>>>> Bloomington, IN 47408
>>>>>>> USA
>>>>>>>
>>>>>>> Let us know once you send it and we can proceed - we trust your word that it is "in the mail".
>>>>>>>
>>>>>>>>
>>>>>>>> If that is done, how can we install MTT in our clusters? Can you please explain it to me step by step? Or is there any elaborate documentation to get info on it? I have gone through the MTT wiki. It doesn’t explicitly speak about how MTT can be deployed in a stand-alone cluster like ours.
>>>>>>>
>>>>>>> Actually, all you really have to do is clone the git repo for it - all the software is there, and it doesn't require any more installation then setting up a .ini file with the proper configuration. Several examples are included in the repo. I can also share mine with you as it does pretty much what you describe. I have a simple "runmtt.sh" script that executes the tests for me, but it is totally trivial and easily shared. The repo is here:
>>>>>>>
>>>>>>> https://github.com/open-mpi/mtt
>>>>>>>
>>>>>>> You can access it any time you like - we don't need the agreement for MTT itself (only for the tests).
>>>>>>>
>>>>>>>>
>>>>>>>> We are using Easy-Build to install MPI and other libraries. So we may skip those steps in MTT.We just need to run the tests and collect health reports. From what I understand from the documentation, I have to fetch MPI tests from your svn server each time and run it on our machines.
>>>>>>>
>>>>>>> You actually don't have to do that - you can clone the ompi-tests repo once, and just run the tests from there. It gets updated infrequently, but you can always just do an "svn up" to ensure you have the latest copies. Again, that is an option in your .ini file.
>>>>>>>
>>>>>>>> Can we collect the results internally and use MTT Reviewer internally to assess it?
>>>>>>>
>>>>>>> Certainly - setting up the database is a bit of a challenge, but once done you can go that route if you like. I'm not the best for those questions, but I'm sure Jeff would be happy to help. He's on travel at the moment, but expected back shortly.
>>>>>>>
>>>>>>>> Or does it go to a central server outside our organisation?
>>>>>>>
>>>>>>> We encourage people to submit the results to the central MTT repo so we can all see them - helps us to maintain the code. However, some places prefer not to do so for firewall or security purposes, and that's just fine too.
>>>>>>>
>>>>>>>>
>>>>>>>> Can you please help us through these queries? Thanks in advance.
>>>>>>>
>>>>>>> We are happy to help get you up-and-running! Please don't hesitate to ask, and I'll send my files along as soon as you are ready.
>>>>>>>
>>>>>>> Ralph
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Jaison Mulerikkal, PhD
>>>>>>>> New Zealand eScience Infrastructure
>>>>>>>> The University of Auckland
>>>>>>>> e: j.mulerikkal_at_[hidden]
>>>>>>>> p: +64 9 923 2194 (internal ext: 82194)
>>>>>>>> w: www.jaison.me
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 23/05/2014, at 3:49 pm, Ralph Castain <rhc_at_[hidden]> wrote:
>>>>>>>>
>>>>>>>>> I do it all the time myself - the ini file contains this:
>>>>>>>>>
>>>>>>>>> #======================================================================
>>>>>>>>> # MPI get phase
>>>>>>>>> #======================================================================
>>>>>>>>>
>>>>>>>>> #----------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> [MPI get: My Installation]
>>>>>>>>> mpi_details = Open MPI
>>>>>>>>> # Leave this parameter blank to
>>>>>>>>> # have MTT search your path for an MPI
>>>>>>>>> # alreadyinstalled_dir = /your/installation
>>>>>>>>> module = AlreadyInstalled
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #----------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> [MPI install: My Installation]
>>>>>>>>> mpi_get = My Installation
>>>>>>>>> module = Analyze::OMPI
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Everything else proceeds the same
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On May 22, 2014, at 8:14 PM, Jaison Mulerikkal <j.mulerikkal_at_[hidden]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am just referring to the following previous query at MTT forums, because I too has a similar requirement.
>>>>>>>>>>
>>>>>>>>>> -----------------------
>>>>>>>>>> I want to skip the MPI Get and Install phases, and simply point MTT to
>>>>>>>>>> my pre-built/installed Open MPI. Can this be done
>>>>>>>>>>
>>>>>>>>>> http://www.open-mpi.org/community/lists/mtt-users/2006/06/0013.php
>>>>>>>>>> ———————————
>>>>>>>>>>
>>>>>>>>>> The reply mentioned that this functionality was in your TODO list. Have you implemented it in anyway at all?
>>>>>>>>>>
>>>>>>>>>> We are running a HPC cluster and we use easy-build to make our modules (including MPI) available to our users. So we may not want to Get, Install, Build steps at MTT, but the Run section to test the stability of the systems. Is it possible to do?
>>>>>>>>>>
>>>>>>>>>> Thanking you in advance,
>>>>>>>>>>
>>>>>>>>>> Jaison Mulerikkal
>>>>>>>>>> New Zealand eScience Infrastructure
>>>>>>>>>> The University of Auckland
>>>>>>>>>> e: j.mulerikkal_at_[hidden]
>>>>>>>>>> p: +64 9 923 2194 (internal ext: 82194)
>>>>>>>>>> w: www.jaison.me
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> mtt-users mailing list
>>>>>>>>>> mtt-users_at_[hidden]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> mtt-users mailing list
>>>>>>>>> mtt-users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> mtt-users mailing list
>>>>>>>> mtt-users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> mtt-users mailing list
>>>>>>> mtt-users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>
>>>>>> _______________________________________________
>>>>>> mtt-users mailing list
>>>>>> mtt-users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>
>>>>>
>>>>> On Jun 2, 2014, at 4:48 PM, Jaison Mulerikkal <j.mulerikkal_at_[hidden]> wrote:
>>>>>
>>>>>> Hi Ralph and others,
>>>>>>
>>>>>> Thanks for your replies. Very useful and encouraging.
>>>>>>
>>>>>> I have posted the signed copy of Corporate Contributor License Agreement to the address you have given.
>>>>>>
>>>>>> Hope you can give us access to the tests-repo now.
>>>>>
>>>>> I'll have to let Jeff do this - he's off to the MPI Forum, and so it may not happen for a few days
>>>>>
>>>>>> Would like to get the configuration file (.ini) that you use and to get a copy of “runmtt.sh” that you run. That will help me understand how it is actually used in cluster.
>>>>>
>>>>> See attached
>>>>>
>>>>>>
>>>>>> I will think about setting up a database for ourselves. After talking to my manager, it seems that he is not fuzzy about using your external database, especially in the beginning. So we may begin with just doing that and later setup something for ourselves. Thanks again for your help.
>>>>>
>>>>> No problem - my .ini is setup to report to our database.
>>>>>
>>>>> <bend-local.ini>
>>>>> <1_Warning.txt>
>>>>> <2_Warning.txt>
>>>>>
>>>>>>
>>>>>> Jaison Mulerikkal, PhD
>>>>>> New Zealand eScience Infrastructure
>>>>>> The University of Auckland
>>>>>> e: j.mulerikkal_at_[hidden]
>>>>>> p: +64 9 923 2194 (internal ext: 82194)
>>>>>> w: www.jaison.me
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 29/05/2014, at 5:04 pm, Ralph Castain <rhc_at_[hidden]> wrote:
>>>>>>
>>>>>>>
>>>>>>> On May 28, 2014, at 8:29 PM, Jaison Mulerikkal <j.mulerikkal_at_[hidden]> wrote:
>>>>>>>
>>>>>>>> Hi Ralph,
>>>>>>>
>>>>>>> Hello - and welcome!
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks for your previous clarification.
>>>>>>>>
>>>>>>>> At our organisation (www.nesi.org.nz) we would like to use MTT for MPI heath check of our clusters.
>>>>>>>
>>>>>>> I actually know some people there! Small world - almost came to work there a few years ago, and have a standing promise to come visit :-)
>>>>>>>
>>>>>>>> It it okay use MTT, legally, if we sign up to the following agreement.
>>>>>>>> http://www.open-mpi.org/community/contribute/corporate.php. Whom shall I send this doc?
>>>>>>>
>>>>>>> There should be an address on the form - it gets sent to:
>>>>>>>
>>>>>>> Center for Research in Extreme Scale Technologies (CREST)
>>>>>>> ATTN: Andrew Lumsdaine,
>>>>>>> Open MPI 3rd Party Contribution Agreements
>>>>>>> 2719 E. 10th Street
>>>>>>> Bloomington, IN 47408
>>>>>>> USA
>>>>>>>
>>>>>>> Let us know once you send it and we can proceed - we trust your word that it is "in the mail".
>>>>>>>
>>>>>>>>
>>>>>>>> If that is done, how can we install MTT in our clusters? Can you please explain it to me step by step? Or is there any elaborate documentation to get info on it? I have gone through the MTT wiki. It doesn’t explicitly speak about how MTT can be deployed in a stand-alone cluster like ours.
>>>>>>>
>>>>>>> Actually, all you really have to do is clone the git repo for it - all the software is there, and it doesn't require any more installation then setting up a .ini file with the proper configuration. Several examples are included in the repo. I can also share mine with you as it does pretty much what you describe. I have a simple "runmtt.sh" script that executes the tests for me, but it is totally trivial and easily shared. The repo is here:
>>>>>>>
>>>>>>> https://github.com/open-mpi/mtt
>>>>>>>
>>>>>>> You can access it any time you like - we don't need the agreement for MTT itself (only for the tests).
>>>>>>>
>>>>>>>>
>>>>>>>> We are using Easy-Build to install MPI and other libraries. So we may skip those steps in MTT.We just need to run the tests and collect health reports. From what I understand from the documentation, I have to fetch MPI tests from your svn server each time and run it on our machines.
>>>>>>>
>>>>>>> You actually don't have to do that - you can clone the ompi-tests repo once, and just run the tests from there. It gets updated infrequently, but you can always just do an "svn up" to ensure you have the latest copies. Again, that is an option in your .ini file.
>>>>>>>
>>>>>>>> Can we collect the results internally and use MTT Reviewer internally to assess it?
>>>>>>>
>>>>>>> Certainly - setting up the database is a bit of a challenge, but once done you can go that route if you like. I'm not the best for those questions, but I'm sure Jeff would be happy to help. He's on travel at the moment, but expected back shortly.
>>>>>>>
>>>>>>>> Or does it go to a central server outside our organisation?
>>>>>>>
>>>>>>> We encourage people to submit the results to the central MTT repo so we can all see them - helps us to maintain the code. However, some places prefer not to do so for firewall or security purposes, and that's just fine too.
>>>>>>>
>>>>>>>>
>>>>>>>> Can you please help us through these queries? Thanks in advance.
>>>>>>>
>>>>>>> We are happy to help get you up-and-running! Please don't hesitate to ask, and I'll send my files along as soon as you are ready.
>>>>>>>
>>>>>>> Ralph
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Jaison Mulerikkal, PhD
>>>>>>>> New Zealand eScience Infrastructure
>>>>>>>> The University of Auckland
>>>>>>>> e: j.mulerikkal_at_[hidden]
>>>>>>>> p: +64 9 923 2194 (internal ext: 82194)
>>>>>>>> w: www.jaison.me
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 23/05/2014, at 3:49 pm, Ralph Castain <rhc_at_[hidden]> wrote:
>>>>>>>>
>>>>>>>>> I do it all the time myself - the ini file contains this:
>>>>>>>>>
>>>>>>>>> #======================================================================
>>>>>>>>> # MPI get phase
>>>>>>>>> #======================================================================
>>>>>>>>>
>>>>>>>>> #----------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> [MPI get: My Installation]
>>>>>>>>> mpi_details = Open MPI
>>>>>>>>> # Leave this parameter blank to
>>>>>>>>> # have MTT search your path for an MPI
>>>>>>>>> # alreadyinstalled_dir = /your/installation
>>>>>>>>> module = AlreadyInstalled
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #----------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> [MPI install: My Installation]
>>>>>>>>> mpi_get = My Installation
>>>>>>>>> module = Analyze::OMPI
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Everything else proceeds the same
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On May 22, 2014, at 8:14 PM, Jaison Mulerikkal <j.mulerikkal_at_[hidden]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am just referring to the following previous query at MTT forums, because I too has a similar requirement.
>>>>>>>>>>
>>>>>>>>>> -----------------------
>>>>>>>>>> I want to skip the MPI Get and Install phases, and simply point MTT to
>>>>>>>>>> my pre-built/installed Open MPI. Can this be done
>>>>>>>>>>
>>>>>>>>>> http://www.open-mpi.org/community/lists/mtt-users/2006/06/0013.php
>>>>>>>>>> ———————————
>>>>>>>>>>
>>>>>>>>>> The reply mentioned that this functionality was in your TODO list. Have you implemented it in anyway at all?
>>>>>>>>>>
>>>>>>>>>> We are running a HPC cluster and we use easy-build to make our modules (including MPI) available to our users. So we may not want to Get, Install, Build steps at MTT, but the Run section to test the stability of the systems. Is it possible to do?
>>>>>>>>>>
>>>>>>>>>> Thanking you in advance,
>>>>>>>>>>
>>>>>>>>>> Jaison Mulerikkal
>>>>>>>>>> New Zealand eScience Infrastructure
>>>>>>>>>> The University of Auckland
>>>>>>>>>> e: j.mulerikkal_at_[hidden]
>>>>>>>>>> p: +64 9 923 2194 (internal ext: 82194)
>>>>>>>>>> w: www.jaison.me
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> mtt-users mailing list
>>>>>>>>>> mtt-users_at_[hidden]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> mtt-users mailing list
>>>>>>>>> mtt-users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> mtt-users mailing list
>>>>>>>> mtt-users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> mtt-users mailing list
>>>>>>> mtt-users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>
>>>>>> _______________________________________________
>>>>>> mtt-users mailing list
>>>>>> mtt-users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>
>>>>> _______________________________________________
>>>>> mtt-users mailing list
>>>>> mtt-users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>
>>>>
>>>> --
>>>> Jeff Squyres
>>>> jsquyres_at_[hidden]
>>>> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
>>>>
>>>> _______________________________________________
>>>> mtt-users mailing list
>>>> mtt-users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>
>>> _______________________________________________
>>> mtt-users mailing list
>>> mtt-users_at_[hidden]
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>> Link to this post: http://www.open-mpi.org/community/lists/mtt-users/2014/06/0811.php
>>
>> _______________________________________________
>> mtt-users mailing list
>> mtt-users_at_[hidden]
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>> Link to this post: http://www.open-mpi.org/community/lists/mtt-users/2014/06/0813.php
>
> _______________________________________________
> mtt-users mailing list
> mtt-users_at_[hidden]
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
> Link to this post: http://www.open-mpi.org/community/lists/mtt-users/2014/06/0814.php