On Mar 19, 2009, at 10:51 AM, Mike Dubman wrote:
> I think we can switch to desired framework (datastore+mapreduce)
> gradually in the background:
> Here is a short battle plan:
> 1. create datastore (google`s or similar)
> 2. design datastore layout (what to keep, how to keep, objects &
> 3. create cmd line tool to submit results into datastore
> 4. integrate (3) into mtt
> 5. Milestone: we have tool to submit run results into two DBs
> (currents & datastore)
Agreed -- this is very do-able.
> 6. Create mpi-aware cmd line tool to query submitted results. Tool
> should allow query and fetch selected results.
> 7. Milestone: we have cmd line tool to query performance results.
> This tool can be used by community to play with custom scripts for
> fetching results and generating custom reports.
> 8. here we can collect 3rd party/contributed scripts to create
> various visual reports based on perf results.
> what do you think?
> I think we can provide some dark forces here to perform most of the
Awesome! I can say that if this stuff becomes available, Cisco will
start "double submitting" -- do the currently-official postgres db
(i.e., same as today), and to the new/experimental datastore.
> Will it be possible to host datastore on openmpi.org and open access
> to it?
I think we have 2 options here:
1. Google's datastore/app engine. That requires signing up for a
Google Apps account with Google Engine access. Josh has one of these
(anyone can get a Google Apps account; as I understand it, you have to
apply for Google Engine access and approval can take a looooong time
-- Josh just got approved after nearly a year). Josh -- could we use
your account, perchance? (I'm not sure if this is Josh's main/
personal Google account, or a generic account he created)
2. Hadoop. This is the open source project that is modeled off
Google's papers that they published about map/reduce. We'd have to
host the hadoop data store somewhere (e.g., IU), but it benefits from
having multiple machines to store data, such as a data farm. I do not
believe that IU has such a resource.
There are definite similarities between the two choices, but I believe
the APIs are different -- so we have to code for one or the other.
I think I would prefer #1 in order to take care of the hosting issue.
If we get past the proof-of-concept stage, I'm guessing it'll be
pretty easy to get the funding to get a real Google Apps account (it's
$50/user/year -- darn cheap).