Subject: Re: [MTT users] Can not find my testing results in OMPI MTT DB
From: Ethan Mallove (ethan.mallove_at_[hidden])
Date: 2008-05-21 14:36:14
On Wed, May/21/2008 06:46:06PM, Pavel Shamis (Pasha) wrote:
> I sent it directly to your email. Please check.
> Thanks,
> Pasha
Got it. Thanks. It's a PHP memory overload issue.
(Apparently I didn't look far back enough in the httpd
error_logs.) See below.
>
> Ethan Mallove wrote:
> > On Wed, May/21/2008 05:19:44PM, Pavel Shamis (Pasha) wrote:
> >
> >> Jeff Squyres wrote:
> >>
> >>> Are we running into http max memory problems or http max upload size
> >>> problems again?
> >>>
> >> I guess it is some server side issue, you need to check the
> >> /var/log/httpd/* log on the server.
> >>
> >
> > The only thing I found in the httpd logs
> > (/var/log/httpd/www.open-mpi.org/error_log*) was this PHP
> > warning, which I don't think would result in lost results:
> >
> > PHP Warning: array_shift(): The argument should be an array in .../submit/index.php on line 1683
> >
> > I haven't received any emailed Postgres errors either. When
> > were these results submitted? I searched for "mellanox" over
> > the past four days. It seem the results aren't buried in
> > here, because I see no test run failures ...
> >
> > http://www.open-mpi.org/mtt/index.php?do_redir=659
> >
> > I'm assuming you're running with two Reporter INI sections:
> > Textfile and MTTDatabase? Can you send some MTT client
> > --verbose/--debug output from the below runs?
> >
> > Thanks,
> > Ethan
> >
> >
> >>> On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote:
> >>>
> >>>
> >>>> Hi,
> >>>>
> >>>> Here is test result from my last mtt run:
> >>>> +-------------+----------------+------+------+----------+------+
> >>>> | Phase | Section | Pass | Fail | Time out | Skip |
> >>>> +-------------+----------------+------+------+----------+------+
> >>>> | MPI install | ompi/gcc | 1 | 0 | 0 | 0 |
> >>>> | MPI install | ompi/intel-9.0 | 1 | 0 | 0 | 0 |
> >>>> | Test Build | trivial | 1 | 0 | 0 | 0 |
> >>>> | Test Build | trivial | 1 | 0 | 0 | 0 |
> >>>> | Test Build | intel-suite | 1 | 0 | 0 | 0 |
> >>>> | Test Build | intel-suite | 1 | 0 | 0 | 0 |
> >>>> | Test Build | imb | 1 | 0 | 0 | 0 |
> >>>> | Test Build | imb | 1 | 0 | 0 | 0 |
> >>>> | Test Build | presta | 1 | 0 | 0 | 0 |
> >>>> | Test Build | presta | 1 | 0 | 0 | 0 |
> >>>> | Test Build | osu_benchmarks | 1 | 0 | 0 | 0 |
> >>>> | Test Build | osu_benchmarks | 1 | 0 | 0 | 0 |
> >>>> | Test Build | netpipe | 1 | 0 | 0 | 0 |
> >>>> | Test Build | netpipe | 1 | 0 | 0 | 0 |
> >>>> | Test Run | trivial | 64 | 0 | 0 | 0 |
> >>>> | Test Run | trivial | 64 | 0 | 0 | 0 |
> >>>> | Test Run | intel-suite | 3179 | 165 | 400 | 0 |
> >>>> | Test Run | intel-suite | 492 | 0 | 0 | 0 |
> >>>> +-------------+----------------+------+------+----------+------+
> >>>>
> >>>> In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found the follow
> >>>> "test run" results:
> >>>> | Test Run | trivial | 64 | 0 | 0 | 0 |
> >>>> | Test Run | trivial | 64 | 0 | 0 | 0 |
> >>>> | Test Run | intel-suite | 492 | 0 | 0 | 0 |
> >>>>
> >>>> And I can not find this one:
> >>>> | Test Run | intel-suite | 3179 | 165 | 400 | 0 |
Some missing results are in mttdb_debug_file.16.txt (and
17.txt), which are the largest .txt files of the bunch. 8
variants isn't that much, but maybe it causes a problem when
there's lots of stderr/stdout? I'm surprised
submit/index.php barfs on files this size:
$ ls -l
...
-rw-r--r-- 1 em162155 staff 956567 May 21 14:21 mttdb_debug_file.16.inc.gz
-rw-r--r-- 1 em162155 staff 9603132 May 21 14:09 mttdb_debug_file.16.txt
...
$ client/mtt-submit -h www.open-mpi.org -f mttdb_debug_file.16.txt -z -u sun -p sun4sun -d
LWP::UserAgent::new: ()
LWP::UserAgent::proxy: http
Filelist: $VAR1 = 'mttdb_debug_file.16.txt';
LWP::MediaTypes::read_media_types: Reading media types from /ws/ompi-tools/lib/perl5/5.8.8/LWP/media.types
LWP::MediaTypes::read_media_types: Reading media types from /usr/perl5/site_perl/5.8.4/LWP/media.types
LWP::MediaTypes::read_media_types: Reading media types from /home/em162155/.mime.types
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: POST http://www.open-mpi.org/mtt/submit/index.php
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::UserAgent::request: Simple response: OK
$ tail -f /var/log/httpd/www.open-mpi.org/error_log | grep -w submit
...
[client 192.18.128.5] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 14 bytes) in /nfs/rontok/xraid/data/osl/www/www.open-mpi.org/mtt/submit/index.php on line 1559
...
We'll have to somehow be more efficient on these loops.
E.g., line 1559:
foreach (array_keys($_POST) as $k) {
Maybe if we broke $_POST up into multiple parts (e.g.,
$_POST_1, $_POST_2, ... $_POST_N)? Maybe we could do
something more efficient than array_keys here? I'm not sure.
The only workaround on the client side would be to break up
the runs. Maybe do a single MPI Install at a time? Do
ompi/gcc then ompi/intel-9.0 as seperate invocations of the
MTT client.
Sorry :-(
-Ethan
> >>>>
> >>>> From the log I see that all tests results were submitted successfully.
> >>>>
> >>>> Can you please check ?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Pasha
> >>>> _______________________________________________
> >>>> mtt-users mailing list
> >>>> mtt-users_at_[hidden]
> >>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
> >>>>
> >>>
> >> _______________________________________________
> >> mtt-users mailing list
> >> mtt-users_at_[hidden]
> >> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
> >>
> >
> >
>
> _______________________________________________
> mtt-users mailing list
> mtt-users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users