On Jul 17, 2007, at 10:54 AM, Shai Venter wrote:
> I know you guys are busy. Any attention to my questions will be
> mostly appreciated.
Glad to help; sorry I didn't get to this yesterday.
> Is there a FAQ for mtt? May be some of my Q have been answered before.
Most of the information on MTT is currently on the MTT wiki:
We have a paper being published about MTT in the Euro PVM/MPI
conference in the beginning of October. The plan is to release MTT
as an open source package at the conference (to including having a
web site for it).
> I intend to run mtt and test some performance over Infiniband. My
> setup is a 2-Dell uni-core machines (i.e.: sw160,sw170) running
> SLES10.0. Each host has an Infiniband HCA Card installed. Each HCA
> Card has 2 Physical ports which are assigned unique IPs (i.e.:
> 126.96.36.199,188.8.131.52 and 184.108.40.206,220.127.116.11 respectively)
> Ports are connected back-to-back (port1 <-> port1 and port2 <-> port2)
> Q #1: Can I override INI file value for hostlist in command line?
> If yes, please provide example.
Yes. You can override any INI file value on the command line. For
./mtt ... field=value
I *believe* that these field=value tokens must be last on the command
line, after all --flags, etc.
Check out the ./mtt --help message for a list of all the command line
> Q #2: In your opinion, what should I specify to hostlist in order
> to run mpi jobs over my Infiniband fabric?
> Is it hostlist = sw160 sw170
> Is it hostlist = 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52
Using either the names or IP addresses is fine; MTT doesn't really
differentiate between them. However, if you want to specify that you
want to start 2 MPI procs on each, then you need to either list the
name/IP twice or use the :num_procs notation, like this:
hostlist = node1:2 node2:2 node3:2 node4:2
Did you look at the ompi-core-template.ini file in the samples
directory? It has a lot of comments in it explaining the fields, etc.
> Q #3: How do I determine which Interfaces mpi uses?
In the Cisco setup, I use MCA parameters to specifically force which
interfaces to use. For example, in my MPI Details section, I have
[MPI Details: Open MPI]
exec = mpirun -np &test_np() @mca@ --mca btl_tcp_if_include ib0 --mca
oob_tcp_if_include ib0 --prefix &test_prefix() &test_executable()
mca = &enumerate( \
"--mca btl sm,tcp,self @common_params@", \
"--mca btl tcp,self @common_params@", \
"--mca btl sm,openib,self @common_params@", \
"--mca btl openib,self @common_params@", \
"--mca mpi_leave_pinned 1 --mca btl openib,self
"--mca mpi_leave_pinned_pipeline 1 --mca btl openib,self
"--mca btl_openib_use_eager_rdma 0 --mca btl openib,self
"--mca btl_openib_use_srq 1 --mca btl openib,self
"--mca mpi_leave_pinned 1 --mca btl sm,openib,self
common_params = --mca btl_openib_max_btls 1
So I'm using --mca btl ... to force specifically which OMPI BTLs to use.
In short -- MTT doesn't know/care what interfaces you use. It just
lets you set exactly which command lines you use to execute MPI jobs.
> Q #4: How can I determine max num of processes for my setup? In the
> case of hostlist = sw160 sw170 , mtt will evaluate max_np to 2.
> In the case of hostlist = 184.108.40.206 220.127.116.11 18.104.22.168
> 22.214.171.124 , max_np will result to 4.
Right. Because in the first case, you listed each host once, so MTT
assumes that you only want one process per host. In the 2nd case,
you list each host twice, so MTT assumes that you want two processes
> Q #5: I what terms can I ask mtt to use a local scratch directory
> on one of the host s hard drive as oppose to some shared scratch
> folder on Network file system.
We don't really have a good solution for this at the moment; the
issue is that we need to have the target MPI installed and available
on all the nodes that you want to run. So we took the easy way out
and just have a single scratch that spans all nodes (usually a
network filesystem). A better solution would be to have a local
scratch and a global scratch; building the MPI could take place in
the local scratch and the install could go to the global scratch.
But we never got around to implementing that... patches would be