After reading Anthony's question again, I am not sure now that we are having the same problem, but we might. In any case, the attached example programs trigger the issue of running out of pipes. I don't see how orted could, even if it was reused. There is only a very limited number of processes running at any given time. Once slave terminates, how would it still have open pipes? Shouldn't the total number of open files, or pipes, be very limited in this situation? And yet, after maybe 20 or so iterations in master.c, orted complains about running out of pipes.
a while back in January of this year, a user (Anthony Thevenin) had the problem of running out of open pipes when he tried to use MPI_Comm_spawn a few times. As I the thread his started in the mailing list archives and have just joined the mailing list myself, I unfortunately can't reply to the thread. "The thread was titled: Doing a lot of spawns does not work with ompi 1.3 BUT works with ompi 1.2.7".
The discussion stopped without really presenting a solution. Is the issue brought up by Anthony fixed? We are running into the same problem.