On 21 Sep 2010, at 09:54, Mikael Lavoie wrote:
> Sorry, but i get lost in what i wanna do, i have build a small home cluster with Pelican_HPC, that user openMPI, and i was trying to find a way to get a multithreaded program work in a multiprocess way without taking the time to learn MPI. And my vision was a sort of wrapper that take C posix app src code, and convert it from pthread to a multiprocessMPI app. But the problem is the remote memory access, that will only be implemented in MPI 3.0(for what i've read of it).
> So, after 12 hour of intensive reading about MPI and POSIX, the best way to deal with my problem(running a C pthreaded app in my cluster) is to convert the src in a SPMD way.
> I didn't mentionned that basicly, my prog open huge text file, take each string and process it through lot's of cryptographic iteration and then save the result in an output.out like file.
> So i will need to make the master process split the input file and then send them as input for the worker process.
> But if you or someone else know a kind of interpretor like program to run a multithreaded C program and convert it logically to a master/worker multiprocess MPI that will be sended by ssh to the interpreter on the worker side and then lunched.
> This is what i've tried to explain in the last msg. A dream for the hobyist that want to get the full power of a night-time cluster, without having to learn all the MPI syntax and structure.
> If it doesn't exist, this would be a really great tool i think.
> Thank you for your reply, but i think i have answered my question alone... No Pain, No Gain...
What you are thinking of is I believe something more like ScaleMP or Mosix, neither of which I have first-hand experience of. It's a hard problem to solve and I don't believe there is any general solution available.
It sounds like your application would be a fairly easy conversion to MPI but to do that you will need to re-code areas of your application, it almost sounds like you could get away with just using MPI_Init, MPI_Scatter and MPI_Gather. Typically you would use the head-node to launch the job but not do any computation, rank 0 in the job would then do the marshalling of data and all ranks would be started simultaneously, you'll find this easier than having one single-rank job spawn more ranks as required.
Ashley Pittman, Bath, UK.
Padb - A parallel job inspection tool for cluster computing