Open MPI logo

MTT Devel Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all MTT Devel mailing list

Subject: Re: [MTT devel] GSOC application
From: Ethan Mallove (ethan.mallove_at_[hidden])
Date: 2009-03-23 12:05:30


On Mon, Mar/23/2009 03:53:53PM, Mike Dubman wrote:
> I'm playing with google datastore now and will send some proposal and
> thoughts.
>
> On Mon, Mar 23, 2009 at 2:33 PM, Jeff Squyres <jsquyres_at_[hidden]> wrote:
>
> Yes, I think you're right -- making a "schema" for the datastore might
> be quite easy. *I'm on travel all this week and likely won't be able to
> look into this stuff -- can you guys post a proposal and we can dive
> into it from that angle?
>
> On Mar 22, 2009, at 6:48 AM, Mike Dubman wrote:
>
> Hello guys,
>
> I`m not sure if we should preserve current DB schema, from one simple
> reason - datastore is an object oriented storage and have different
> rules and techniques then rdbms.
> The basic storage unit in the datastore is an object which can be
> saved, loaded and queried.
> (hadoop is based on the same principles, but open source.)
>
> It seems that DB model for mtt over datastore should not be complex at
> all. The current mtt db schema is mostly optimized for specific
> queries dictated by web UI. Datastore creates indexes automatically,
> based on submitted queries history.
>
> I suggest we discuss/exchange db layout proposals by emails and when
> we get to some general understanding how it should look like - we
> switch to telepresence.
>
> Also, It seems not problem at all to get datastore access for existing
> gmail account. You get 500MB quota for storage. It takes 5min to start
> using it.
>
> Here is some short info for datastore API:
> - howto submit data model to datastore
> - howto save, load, query
>
> http://code.google.com/appengine/docs/python/gettingstarted/usingdatastore.html
>
> please comment.

Do we have a monthly cost estimate for this project? We will exceed
the free quota of CPU/bandwidth/storage/email, and get billed
(depending on how efficient our App is):

  http://code.google.com/appengine/docs/billing.html

The biggest concern would be the Stored Data cost, because I like how
we can now archive lots and lots of test results. I do not have
permission to access /var/lib/pgsql/data, but weren't we at or near
100 GBs recently? The bandwidth charge would seem to be pretty
nominal. We could upload/download 100GB/mo. for just $10/mo. and I am
not sure if we approach 100GB. CPU Time is a a mystery number to me.

  -------------------+---------------------+----------
  Resource | Unit | Unit cost
  -------------------+---------------------+----------
  Outgoing Bandwidth | gigabytes | $0.12
  Incoming Bandwidth | gigabytes | $0.10
  CPU Time | CPU hours | $0.10
  Stored Data | gigabytes per month | $0.15
  Recipients Emailed | recipients | $0.0001
  -------------------+---------------------+----------

Would we itemize the MTT bill on a per user basis? E.g., orgs that
use MTT more, would have to pay more?

-Ethan

>
> Thanks
>
> Mike
>
> On Fri, Mar 20, 2009 at 5:38 PM, Jeff Squyres <jsquyres_at_[hidden]>
> wrote:
> On Mar 20, 2009, at 10:42 AM, Josh Hursey wrote:
>
> Yeah I think this sounds like a good way to move forward with this
> work. The database schema is pretty complex. If you need help on the
> database side of things let me know.
>
> To get started, would it be useful to have a meeting over the phone/
> telepresence to design the datastore layout? This gives us an
> opportunity to start from a blank slate with regards to the
> datastore, so it may be useful brainstorm a bit beforehand.
>
> Yes, it probably would. *My understanding of hadoop (which is very
> highlevel) is that just dump everything in without too much concern
> about the structure / "schema". *But I could be wrong on that.
>
> The Google Apps account is under my personal Google account,
> so I am
> reluctant to use it. I think the reason it took so long for me, was
> because when I originally signed up it was in limited beta. I think
> the approval time is much shorter now (maybe a day?), and we can make
> an openmpi or mtt account that we can use.
>
> With regard to Hadoop, I don't think that IU has a set of machines
> that would work, but I can ask around. We could always try Hadoop on
> a single machine if people wanted to play around with data querying/
> storage.
>
> I don't have a strong preference either way, but Google Apps may
> provide us with a lower overhead solution for the long run even
> though it costs $$.
>
> It looks like there is a set that you can use for free. *When you go
> over one of several metrics (CPU hours/day, storage, bandwidth in,
> bandwidth out, etc.), then you have to start paying. *But even with
> that, the costs look *quite* reasonable and should be easily covered
> by the combined Open MPI organizations (I'm talking hundreds of
> dollars here, not tens of thousands).
>
> --
> Jeff Squyres
> Cisco Systems
>
> _______________________________________________
> mtt-devel mailing list
> mtt-devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel
>
> _______________________________________________
> mtt-devel mailing list
> mtt-devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel
>
> --
> Jeff Squyres
> Cisco Systems
>
> _______________________________________________
> mtt-devel mailing list
> mtt-devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel
>
> References
>
> Visible links
> . mailto:jsquyres_at_[hidden]
> . http://code.google.com/appengine/docs/python/gettingstarted/usingdatastore.html
> . mailto:jsquyres_at_[hidden]
> . mailto:mtt-devel_at_[hidden]
> . http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel
> . mailto:mtt-devel_at_[hidden]
> . http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel
> . mailto:mtt-devel_at_[hidden]
> . http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel

> _______________________________________________
> mtt-devel mailing list
> mtt-devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-devel