Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] sg riddle (was: EXTERNAL: Re: Can you set the gid of the processes created by mpirun?)
From: Reuti (reuti_at_[hidden])
Date: 2011-09-15 13:00:46


Am 15.09.2011 um 16:31 schrieb Blosch, Edwin L:

> I think one has to quote everything after the gid, sg 25000 "ls -l" should work.
>
> I tried something similar to the orted wrapper but also couldn't get it to work, but I didn't have time yet to dig into it. Might be worth trying it with both quotes:
> #!/bin/sh
> exec sg 25000 "orted ${@}"

Thx for the reply.

It looks like it depends on the Version of `sg`. On Scientific Linux I *must* use the group name, no go with the group id:

$ sg 537 ls
unknown group: 537

On openSUSE you can use either of them.

The version on Scientific Linux accepts any number of arguments, but all except the first will be ignored:

$ sg my_group "echo" "1" "2" "3"

(Yes, an empty line only) while on openSUSE:

$ sg 24000 "echo" "1" "2" "3"
$ sg: Too many arguments.
Try `sg --help' or `sg --usage' for more information.

==

This is now going away from Open MPI but anyway:

What is indeed working is: sg 25000 "ls -l"

But it still fails for:

$ sg 24000 "./tt.sh ${@}"
sg: Too many arguments.

I works again if I use * in place of a @, but this will change how the bash will assemble the arguments:

$ set "11 12" "21 22"
$ echo $1
11 12
$ echo $2
21 22
$ sg 24000 "./tt.sh ${*}"
12
11

What I expect is:

$ ./tt.sh "${@}"
21 22
11 12

While the script tt.sh lists just:

#!/bin/sh
echo $2
echo $1

-- Reuti

> Also, Ralph,
>
> I think I skipped one step of logic in asking you yesterday to consider adding the capability to pass a multi-word argument to orte_launch_agent. This itself seems like it could have uses, so I like the idea. But what I'd like to have, in the end, is the possibility to launch all processes with the same group or gid (for the situation where all hosts share the same passwd/group database), without relying on any launcher other than rsh/ssh. I think it should be possible. The best way to do this might not be the idea of using sg, so perhaps this should be discussed first before you spend time adding multi-word support for the orte_launch_agent MCA parameter. But if you do it I will certainly test it.
>
> Thanks again
>
>
> Ed
>
> -----Original Message-----
> From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]] On Behalf Of Reuti
> Sent: Thursday, September 15, 2011 4:37 AM
> To: Open MPI Users
> Subject: Re: [OMPI users] EXTERNAL: Re: Can you set the gid of the processes created by mpirun?
>
> Am 15.09.2011 um 01:15 schrieb Blosch, Edwin L:
>
>> I would appreciate trying to fix the multi-word argument to orte_launch_agent, unless you've already fixed it in a later series in which case I could update. My workaround with the setuid applied to a copied executable doesn't work for other applications that run on our clusters ( I mean their job scripts would also have to be modified similarly).
>
> I played around with `sg` and it looks like having some pitfalls:
>
> $ sg 25000 ls -l
>
> The -l will go to `sg`, even double quotes fo " and ' don't help and there is no --:
>
> $ sg 2600 ls "'-l'"
>
> gives an error too.
>
> (I was about to suggest an orted wrapper doing:
>
> #!/bin/sh
> exec sg 25000 orted ${@}"
>
> but it failed.)
>
>
>> I think OpenMPI should be able to launch all processes with the same uid and gid even when the user has launched the job while using one of his secondary groups. Whether it should be a change request or bug (unexpected, counter-intuitive), I'll leave that to your judgment.
>
> Yes, but for now the "username" is used, not the "uid" - right? So it could have a different uid/gid on the machines, but with the new feature they must be the same. Okay, in a cluster they are most likely unique across all machines anyway. But just to note as a side effect.
>
> -- Reuti
>
>
>> Thank you again for the support
>>
>>
>>
>> From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]] On Behalf Of Ralph Castain
>> Sent: Wednesday, September 14, 2011 5:10 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] EXTERNAL: Re: Can you set the gid of the processes created by mpirun?
>>
>> Hi Ed
>>
>> I finally had some time to dig around a little. I believe we could make this work if I fix the multi-word cmd line param issue. Either "newgrp" or "sg" should resolve the problem - both are user-level cmds.
>>
>> Up to you - let me know if you want me to pursue this.
>> Ralph
>>
>>
>> On Sep 14, 2011, at 3:31 PM, Blosch, Edwin L wrote:
>>
>>
>> Thank you - I did pursue this kind of workaround, and it worked, but you'll be happy to know that nothing had to be owned by root.
>>
>> ASIDE ----
>> Just to remind: The job script is a shell script that invokes mpirun; the job script itself is run with as the correct user, but the group id may be changed to whatever the user requests of the job-scheduling system. I think it may not be uncommon to have jobs that request a specific Unix group, for many reasons, but in our case the group is an input for the scheduler's prioritization policy.
>>
>> Outcome: rank 0 runs user:group2 as the user requested, but the launched child processes run user:group1 where group 1 is the user's primary group. The peculiarity of this application is that each of the processes writes a file to disk, so the resulting group ownership of rank 0 files is group2, but the group ownership of all other ranks' files is group1. That was the original problem I'm trying to work around.
>> --- END ASIDE
>>
>> Fortunately for me, there is another peculiarity of this application -- the executable gets copied out to /tmp (local space) on each of the hosts to be used in the job. We found this helped prevent some crashes during test phases where an executable gets overwritten while in use. Definitely a special behavior. But as a result of this peculiarity, the mpirun command ends up launching the copied executable, and I took advantage of that.
>>
>> I had the job script do chown user:group2 on the copied executables and then chmod 6711, and then I observed that the child processes ran as user:group2, same as the rank 0 process, so the files they created had the desired group ownership.
>>
>> I will explore Reuti's guidance as well.
>>
>> Thank you
>>
>>
>> From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]] On Behalf Of Randall Svancara
>> Sent: Wednesday, September 14, 2011 3:07 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] EXTERNAL: Re: Can you set the gid of the processes created by mpirun?
>>
>> You could set the setuid bit on the application and chown it to root?? It is about as secure as anything else that has been described thus far. As a system admin, I cringe at the thought of anything that would allow something to run as someone else, so there would have to be a pretty good justification for such unique use case as yours.
>>
>> Randall
>>
>> On Wed, Sep 14, 2011 at 12:56 PM, Reuti <reuti_at_[hidden]> wrote:
>> Am 14.09.2011 um 19:02 schrieb Blosch, Edwin L:
>>
>>> Thanks for trying.
>>>
>>> Do you feel that this is an impossible request without the assistance of some process running as root, for example, as Reuti mentioned, the daemons of a job scheduler? Or are you saying it will just not be as straightforward as calling setgid as you had hoped?
>>>
>>> Also, do you think there is a way I could make use of the sg command below? Perhaps there is a way to have the rsh/ssh launcher start the application processes with a command like 'sg <group> <executable name>'?
>>
>> What about a half-tight integration (or call it: classic tight integration), i.e. no recompilation necessary?
>>
>> - setup your mpiexec call in the jobscript to use a plain rsh for the remote startup (no path given): -mca plm_rsh_agent rsh
>>
>> - the PE of SGE needs the argument -catch_rsh in start_proc_args and the supplied script in $SGE_ROOT/mpi/startmpi.sh
>>
>> (SGE will create a symbolic link in $TMPDIR therein [which will be called first this way] to the rsh-wrapper in $SGE_ROOT/mpi [pitfall: some applications need a -V to be added in the lines woth "qrsh", i.e. "qrsh -inherit -V ..." to send all environment variables to the slaves])
>>
>> - what is your setting of qrsh_daemon/qrsh_command in `qconf -sconf`? This will then be used finally to reach the node and should be builtin or point to the SGE supplied rsh/rshd (no rshd necessary to install, no rshd is running all the time, no rshd will be started by xinet.d or alike)
>>
>> - like you do already: switch off the built-in SGE starter in your mpiexec call: -mca plm_rsh_disable_qrsh 1
>>
>> -- Reuti
>>
>> PS: To avoid misunderstandings: you could also set "-mca plm_rsh_agent foobar" and in $SGE_ROOT/mpi/startmpi.sh you change it to create a symbolic link called "foobar " in $TMPDIR. It's just a name at this stage of startup.
>>
>>
>>> Ed
>>>
>>>
>>> NAME
>>> sg - execute command as different group ID
>>>
>>> SYNOPSIS
>>> sg [-] [group [-c ] command]
>>>
>>> DESCRIPTION
>>> The sg command works similar to newgrp but accepts a command. The
>>> command will be executed with the /bin/sh shell. With most shells you
>>> may run sg from, you need to enclose multi-word commands in quotes.
>>> Another difference between newgrp and sg is that some shells treat
>>> newgrp specially, replacing themselves with a new instance of a shell
>>> that newgrp creates. This doesn't happen with sg, so upon exit from a
>>> sg command you are returned to your previous group ID.
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]] On Behalf Of Ralph Castain
>>> Sent: Wednesday, September 14, 2011 11:33 AM
>>> To: Open MPI Users
>>> Subject: Re: [OMPI users] EXTERNAL: Re: Can you set the gid of the processes created by mpirun?
>>>
>>>
>>> On Sep 14, 2011, at 9:39 AM, Blosch, Edwin L wrote:
>>>
>>>> Thanks, Ralph,
>>>>
>>>> I get the failure messages, unfortunately:
>>>>
>>>> setgid FAILED
>>>> setgid FAILED
>>>> setgid FAILED
>>>>
>>>> I actually had attempted to call setgid from within the application previously, which looks similar to what you've done, but it failed. That was when I initiated the post to the mailing list. My conclusion, a guess really, was that Linux would not let me setgid from within my program because I was not root.
>>>
>>> I was afraid of that - the documentation seemed to indicate that would be the case, but I figured it was worth a quick try. Sorry I can't be of help.
>>>
>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]] On Behalf Of Ralph Castain
>>>> Sent: Wednesday, September 14, 2011 8:15 AM
>>>> To: Open MPI Users
>>>> Subject: Re: [OMPI users] EXTERNAL: Re: Can you set the gid of the processes created by mpirun?
>>>>
>>>> The attached should set the gid of the remote daemons (and their children) to the gid of mpirun. No cmd line option or anything is required - it will just always do it.
>>>>
>>>> Would you mind giving it a try?
>>>>
>>>> Please let me know if/how it works.
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>> --
>> Randall Svancara
>> http://knowyourlinux.com/
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users