Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] [OMPI svn] svn:open-mpi r21513
From: George Bosilca (bosilca_at_[hidden])
Date: 2009-06-24 18:05:31


Just for the sake of it. A funy command line to try:

[bosilca_at_dancer ~]$ mpirun --mca routed_base_verbose 0 --leave-session-
attached -np 1 --mca orte_launch_agent "orted --mca
routed_base_verbose 1" uptime

[node03:22355] [[14661,0],1] routed_linear: init routes for daemon job
[14661,0]
        hnp_uri 960823296.0;tcp://192.168.1.254:58135;tcp://192.168.0.2:58135
  18:02:59 up 26 days, 17:41, 0 users, load average: 0.97, 0.50, 0.53
[bosilca_at_dancer ~]$ [node03:22355] [[14661,0],1]
routed_linear_get([[14661,0],0]) --> [[14661,0],0]
[node03:22355] [[14661,0],1] routed_linear: init routes for daemon job
[14661,0]
        hnp_uri 960823296.0;tcp://192.168.1.254:58135;tcp://192.168.0.2:58135
[node03:22355] [[14661,0],1] routed_linear_get([[14661,0],0]) -->
[[14661,0],0]
[node03:22355] [[14661,0],1] routed_linear_get([[14661,0],0]) -->
[[14661,0],0]
[node03:22355] [[14661,0],1] routed_linear_get([[14661,0],0]) -->
[[14661,0],0]

This set the routed_base_verbose to zero for the HNP, and to 1 for
everybody else. As you can see from the output the orted output routed
information which means it correctly interpreted the multiword argument.

   george.

On Jun 24, 2009, at 17:52 , George Bosilca wrote:

>
> On Jun 24, 2009, at 17:41 , Jeff Squyres wrote:
>
>> -----
>> [14:38] svbu-mpi:~/svn/ompi/orte % mpirun --mca plm_base_verbose
>> 100 --leave-session-attached -np 1 --mca orte_launch_agent "$bogus/
>> bin/orted -s" uptime
>> ...lots of output...
>> srun --nodes=1 --ntasks=1 --kill-on-bad-exit --nodelist=svbu-
>> mpi062 /home/jsquyres/bogus/bin/orted -s -mca ess slurm -mca
>> orte_ess_jobid 3195142144 -mca orte_ess_vpid 1 -mca
>> orte_ess_num_procs 2 --hnp-uri "3195142144.0;tcp://
>> 172.29.218.140:34489;tcp://10.10.20.250:34489;tcp://
>> 10.10.30.250:34489;tcp://192.168.183.1:34489;tcp://
>> 192.168.184.1:34489" -mca orte_nodelist svbu-mpi062 --mca
>> plm_base_verbose 100 --mca orte_launch_agent "/home/jsquyres/bogus/
>> bin/orted -s"
>> ...
>> -----
>>
>> and it hangs, because the argv[0]
>>
>> "/home/jsquyres/bogus/bin/orted -s"
>>
>> (including the quotes!) cannot be exec'ed.
>
> OK so maybe the -s option was a bad example (it's the one I use
> regularly). It block the orted, you will have to log on each node,
> attach with gdb to the orted, and release them by doing a "set
> orted_spin_flag=0".
>
> george.
>
>>
>>
>>
>>
>> On Jun 24, 2009, at 5:15 PM, George Bosilca wrote:
>>
>>> I can't guarantee this for all PLM but I can confirm that rsh and
>>> slurm (1.3.12) works well with this.
>>>
>>> We try with and without Open MPI, and the outcome is the same.
>>>
>>> [bosilca_at_dancer c]$ srun -n 4 echo "1 2 3 4 5 it works"
>>> 1 2 3 4 5 it works
>>> 1 2 3 4 5 it works
>>> 1 2 3 4 5 it works
>>> 1 2 3 4 5 it works
>>>
>>> [bosilca_at_dancer c]$ srun -N 2 -c 2 mpirun --mca plm slurm --mca
>>> orte_launch_agent "orted -s" --mca plm_rsh_tree_spawn 1 --bynode --
>>> mca
>>> pml ob1 --mca orte_daemon_spin 0 ./hello
>>> Hello, world, I am 0 of 2 on node03
>>> Hello, world, I am 1 of 2 on node04
>>>
>>> *after releasing the orted from their spin.
>>>
>>> In fact what I find strange is the old behavior. Dropping arguments
>>> without even letting the user know about it, is certainly not a
>>> desirable approach.
>>>
>>> george.
>>>
>>> On Jun 24, 2009, at 16:15 , Ralph Castain wrote:
>>>
>>> > Yo George
>>> >
>>> > This commit is going to break non-rsh launchers. While it is true
>>> > that the rsh launcher may handle multi-word options by putting
>>> them
>>> > in quotes, we specifically avoided it here because it breaks
>>> SLURM,
>>> > Torque, and others.
>>> >
>>> > This is why we specifically put the inclusion of multi-word
>>> options
>>> > in the rsh plm module, and not here. Would you please move it back
>>> > there?
>>> >
>>> > Thanks
>>> > Ralph
>>> >
>>> >
>>> > On Wed, Jun 24, 2009 at 1:51 PM, <bosilca_at_[hidden]> wrote:
>>> > Author: bosilca
>>> > Date: 2009-06-24 15:51:52 EDT (Wed, 24 Jun 2009)
>>> > New Revision: 21513
>>> > URL: https://svn.open-mpi.org/trac/ompi/changeset/21513
>>> >
>>> > Log:
>>> > When we get a report from an orted about its state, don't use the
>>> > sender of
>>> > the message to update the structures, but instead use the
>>> > information from
>>> > the URI. The reason is that even the launch report messages can
>>> get
>>> > routed.
>>> >
>>> > Deal with the orted_cmd_line in a single location.
>>> >
>>> > Text files modified:
>>> > trunk/orte/mca/plm/base/plm_base_launch_support.c | 69 +++++
>>> ++++
>>> > ++++++++++++++----------------
>>> > 1 files changed, 41 insertions(+), 28 deletions(-)
>>> >
>>> > Modified: trunk/orte/mca/plm/base/plm_base_launch_support.c
>>> > =
>>> > =
>>> > =
>>> > =
>>> > =
>>> > =
>>> > =
>>> > =
>>> >
>>> =
>>> =
>>> ====================================================================
>>> > --- trunk/orte/mca/plm/base/plm_base_launch_support.c (original)
>>> > +++ trunk/orte/mca/plm/base/plm_base_launch_support.c 2009-06-24
>>> > 15:51:52 EDT (Wed, 24 Jun 2009)
>>> > @@ -433,7 +433,8 @@
>>> > {
>>> > orte_message_event_t *mev = (orte_message_event_t*)data;
>>> > opal_buffer_t *buffer = mev->buffer;
>>> > - char *rml_uri;
>>> > + orte_process_name_t peer;
>>> > + char *rml_uri = NULL;
>>> > int rc, idx;
>>> > int32_t arch;
>>> > orte_node_t **nodes;
>>> > @@ -442,19 +443,11 @@
>>> > int64_t setupsec, setupusec;
>>> > int64_t startsec, startusec;
>>> >
>>> > - OPAL_OUTPUT_VERBOSE((5, orte_plm_globals.output,
>>> > - "%s plm:base:orted_report_launch from
>>> > daemon %s",
>>> > - ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
>>> > - ORTE_NAME_PRINT(&mev->sender)));
>>> > -
>>> > /* see if we need to timestamp this receipt */
>>> > if (orte_timing) {
>>> > gettimeofday(&recvtime, NULL);
>>> > }
>>> >
>>> > - /* update state */
>>> > - pdatorted[mev->sender.vpid]->state = ORTE_PROC_STATE_RUNNING;
>>> > -
>>> > /* unpack its contact info */
>>> > idx = 1;
>>> > if (ORTE_SUCCESS != (rc = opal_dss.unpack(buffer, &rml_uri,
>>> > &idx, OPAL_STRING))) {
>>> > @@ -466,13 +459,26 @@
>>> > /* set the contact info into the hash table */
>>> > if (ORTE_SUCCESS != (rc =
>>> orte_rml.set_contact_info(rml_uri))) {
>>> > ORTE_ERROR_LOG(rc);
>>> > - free(rml_uri);
>>> > orted_failed_launch = true;
>>> > goto CLEANUP;
>>> > }
>>> > - /* lookup and record this daemon's contact info */
>>> > - pdatorted[mev->sender.vpid]->rml_uri = strdup(rml_uri);
>>> > - free(rml_uri);
>>> > +
>>> > + rc = orte_rml_base_parse_uris(rml_uri, &peer, NULL );
>>> > + if( ORTE_SUCCESS != rc ) {
>>> > + ORTE_ERROR_LOG(rc);
>>> > + orted_failed_launch = true;
>>> > + goto CLEANUP;
>>> > + }
>>> > +
>>> > + OPAL_OUTPUT_VERBOSE((5, orte_plm_globals.output,
>>> > + "%s plm:base:orted_report_launch from
>>> > daemon %s via %s",
>>> > + ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
>>> > + ORTE_NAME_PRINT(&peer),
>>> > + ORTE_NAME_PRINT(&mev->sender)));
>>> > +
>>> > + /* update state and record for this daemon contact info */
>>> > + pdatorted[peer.vpid]->state = ORTE_PROC_STATE_RUNNING;
>>> > + pdatorted[peer.vpid]->rml_uri = rml_uri;
>>> >
>>> > /* get the remote arch */
>>> > idx = 1;
>>> > @@ -555,31 +561,33 @@
>>> >
>>> > /* lookup the node */
>>> > nodes = (orte_node_t**)orte_node_pool->addr;
>>> > - if (NULL == nodes[mev->sender.vpid]) {
>>> > + if (NULL == nodes[peer.vpid]) {
>>> > ORTE_ERROR_LOG(ORTE_ERR_NOT_FOUND);
>>> > orted_failed_launch = true;
>>> > goto CLEANUP;
>>> > }
>>> > /* store the arch */
>>> > - nodes[mev->sender.vpid]->arch = arch;
>>> > + nodes[peer.vpid]->arch = arch;
>>> >
>>> > /* if a tree-launch is underway, send the cmd back */
>>> > if (NULL != orte_tree_launch_cmd) {
>>> > - orte_rml.send_buffer(&mev->sender, orte_tree_launch_cmd,
>>> > ORTE_RML_TAG_DAEMON, 0);
>>> > + orte_rml.send_buffer(&peer, orte_tree_launch_cmd,
>>> > ORTE_RML_TAG_DAEMON, 0);
>>> > }
>>> >
>>> > CLEANUP:
>>> >
>>> > OPAL_OUTPUT_VERBOSE((5, orte_plm_globals.output,
>>> > - "%s plm:base:orted_report_launch %s for
>>> > daemon %s at contact %s",
>>> > + "%s plm:base:orted_report_launch %s for
>>> > daemon %s (via %s) at contact %s",
>>> > ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
>>> > orted_failed_launch ? "failed" :
>>> "completed",
>>> > - ORTE_NAME_PRINT(&mev->sender),
>>> > pdatorted[mev->sender.vpid]->rml_uri));
>>> > + ORTE_NAME_PRINT(&peer),
>>> > + ORTE_NAME_PRINT(&mev->sender),
>>> > pdatorted[peer.vpid]->rml_uri));
>>> >
>>> > /* release the message */
>>> > OBJ_RELEASE(mev);
>>> >
>>> > if (orted_failed_launch) {
>>> > + if( NULL != rml_uri ) free(rml_uri);
>>> > orte_errmgr.incomplete_start(ORTE_PROC_MY_NAME->jobid,
>>> > ORTE_ERROR_DEFAULT_EXIT_CODE);
>>> > } else {
>>> > orted_num_callback++;
>>> > @@ -1133,18 +1141,23 @@
>>> > * being sure to "purge" any that would cause problems
>>> > * on backend nodes
>>> > */
>>> > - if (ORTE_PROC_IS_HNP) {
>>> > + if (ORTE_PROC_IS_HNP || ORTE_PROC_IS_DAEMON) {
>>> > cnt = opal_argv_count(orted_cmd_line);
>>> > for (i=0; i < cnt; i+=3) {
>>> > - /* if the specified option is more than one word, we
>>> > don't
>>> > - * have a generic way of passing it as some
>>> > environments ignore
>>> > - * any quotes we add, while others don't - so we
>>> ignore
>>> > any
>>> > - * such options. In most cases, this won't be a
>>> problem
>>> > as
>>> > - * they typically only apply to things of interest to
>>> > the HNP.
>>> > - * Individual environments can add these back into
>>> the
>>> > cmd line
>>> > - * as they know if it can be supported
>>> > - */
>>> > - if (NULL != strchr(orted_cmd_line[i+2], ' ')) {
>>> > + /* in the rsh environment, we can append multi-word
>>> > arguments
>>> > + * by enclosing them in quotes. Check for any
>>> multi-word
>>> > + * mca params passed to mpirun and include them
>>> > + */
>>> > + if (NULL != strchr(orted_cmd_line[i+2], ' ')) {
>>> > + char* param;
>>> > +
>>> > + /* must add quotes around it */
>>> > + asprintf(&param, "\"%s\"", orted_cmd_line[i+2]);
>>> > + /* now pass it along */
>>> > + opal_argv_append(argc, argv, orted_cmd_line[i]);
>>> > + opal_argv_append(argc, argv, orted_cmd_line[i
>>> +1]);
>>> > + opal_argv_append(argc, argv, param);
>>> > + free(param);
>>> > continue;
>>> > }
>>> > /* The daemon will attempt to open the PLM on the
>>> remote
>>> > _______________________________________________
>>> > svn mailing list
>>> > svn_at_[hidden]
>>> > http://www.open-mpi.org/mailman/listinfo.cgi/svn
>>> >
>>> > _______________________________________________
>>> > devel mailing list
>>> > devel_at_[hidden]
>>> > http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>
>>
>> --
>> Jeff Squyres
>> Cisco Systems
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel