Subject: Re: [MTT users] Can not find my testing results in OMPI MTT DB
From: Pavel Shamis (Pasha) (pasha_at_[hidden])
Date: 2008-05-21 14:53:11
Oops, in the "MTT server side problem" we discussed other issue.
But anyway I did not see the problem on my server after the upgrade :)
Pasha
Pavel Shamis (Pasha) wrote:
> I had similar problem on my server. I upgraded the server to latest
> trunk and the problem disappear.
> (see "MTT server side problem" thread).
>
> Pasha
>
> Jeff Squyres wrote:
>
>> FWIW: I think we have at least one open ticket on this issue (break up
>> submits so that we don't overflow PHP and/or apache).
>>
>> On May 21, 2008, at 2:36 PM, Ethan Mallove wrote:
>>
>>
>>> On Wed, May/21/2008 06:46:06PM, Pavel Shamis (Pasha) wrote:
>>>
>>>> I sent it directly to your email. Please check.
>>>> Thanks,
>>>> Pasha
>>>>
>>> Got it. Thanks. It's a PHP memory overload issue.
>>> (Apparently I didn't look far back enough in the httpd
>>> error_logs.) See below.
>>>
>>>
>>>> Ethan Mallove wrote:
>>>>
>>>>> On Wed, May/21/2008 05:19:44PM, Pavel Shamis (Pasha) wrote:
>>>>>
>>>>>
>>>>>> Jeff Squyres wrote:
>>>>>>
>>>>>>
>>>>>>> Are we running into http max memory problems or http max upload size
>>>>>>> problems again?
>>>>>>>
>>>>>>>
>>>>>> I guess it is some server side issue, you need to check the
>>>>>> /var/log/httpd/* log on the server.
>>>>>>
>>>>>>
>>>>> The only thing I found in the httpd logs
>>>>> (/var/log/httpd/www.open-mpi.org/error_log*) was this PHP
>>>>> warning, which I don't think would result in lost results:
>>>>>
>>>>> PHP Warning: array_shift(): The argument should be an array in
>>>>> .../submit/index.php on line 1683
>>>>>
>>>>> I haven't received any emailed Postgres errors either. When
>>>>> were these results submitted? I searched for "mellanox" over
>>>>> the past four days. It seem the results aren't buried in
>>>>> here, because I see no test run failures ...
>>>>>
>>>>> http://www.open-mpi.org/mtt/index.php?do_redir=659
>>>>>
>>>>> I'm assuming you're running with two Reporter INI sections:
>>>>> Textfile and MTTDatabase? Can you send some MTT client
>>>>> --verbose/--debug output from the below runs?
>>>>>
>>>>> Thanks,
>>>>> Ethan
>>>>>
>>>>>
>>>>>
>>>>>>> On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Here is test result from my last mtt run:
>>>>>>>> +-------------+----------------+------+------+----------+------+
>>>>>>>> | Phase | Section | Pass | Fail | Time out | Skip |
>>>>>>>> +-------------+----------------+------+------+----------+------+
>>>>>>>> | MPI install | ompi/gcc | 1 | 0 | 0 | 0 |
>>>>>>>> | MPI install | ompi/intel-9.0 | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | trivial | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | trivial | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | intel-suite | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | intel-suite | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | imb | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | imb | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | presta | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | presta | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | osu_benchmarks | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | osu_benchmarks | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | netpipe | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Build | netpipe | 1 | 0 | 0 | 0 |
>>>>>>>> | Test Run | trivial | 64 | 0 | 0 | 0 |
>>>>>>>> | Test Run | trivial | 64 | 0 | 0 | 0 |
>>>>>>>> | Test Run | intel-suite | 3179 | 165 | 400 | 0 |
>>>>>>>> | Test Run | intel-suite | 492 | 0 | 0 | 0 |
>>>>>>>> +-------------+----------------+------+------+----------+------+
>>>>>>>>
>>>>>>>> In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found the follow
>>>>>>>> "test run" results:
>>>>>>>> | Test Run | trivial | 64 | 0 | 0 | 0 |
>>>>>>>> | Test Run | trivial | 64 | 0 | 0 | 0 |
>>>>>>>> | Test Run | intel-suite | 492 | 0 | 0 | 0 |
>>>>>>>>
>>>>>>>> And I can not find this one:
>>>>>>>> | Test Run | intel-suite | 3179 | 165 | 400 | 0 |
>>>>>>>>
>>> Some missing results are in mttdb_debug_file.16.txt (and
>>> 17.txt), which are the largest .txt files of the bunch. 8
>>> variants isn't that much, but maybe it causes a problem when
>>> there's lots of stderr/stdout? I'm surprised
>>> submit/index.php barfs on files this size:
>>>
>>> $ ls -l
>>> ...
>>> -rw-r--r-- 1 em162155 staff 956567 May 21 14:21
>>> mttdb_debug_file.16.inc.gz
>>> -rw-r--r-- 1 em162155 staff 9603132 May 21 14:09
>>> mttdb_debug_file.16.txt
>>> ...
>>>
>>> $ client/mtt-submit -h www.open-mpi.org -f mttdb_debug_file.16.txt
>>> -z -u sun -p sun4sun -d
>>> LWP::UserAgent::new: ()
>>> LWP::UserAgent::proxy: http
>>>
>>> Filelist: $VAR1 = 'mttdb_debug_file.16.txt';
>>> LWP::MediaTypes::read_media_types: Reading media types from
>>> /ws/ompi-tools/lib/perl5/5.8.8/LWP/media.types
>>> LWP::MediaTypes::read_media_types: Reading media types from
>>> /usr/perl5/site_perl/5.8.4/LWP/media.types
>>> LWP::MediaTypes::read_media_types: Reading media types from
>>> /home/em162155/.mime.types
>>> LWP::UserAgent::request: ()
>>> LWP::UserAgent::send_request: POST
>>> http://www.open-mpi.org/mtt/submit/index.php
>>> LWP::UserAgent::_need_proxy: Not proxied
>>> LWP::Protocol::http::request: ()
>>> LWP::UserAgent::request: Simple response: OK
>>>
>>> $ tail -f /var/log/httpd/www.open-mpi.org/error_log | grep -w submit
>>> ...
>>> [client 192.18.128.5] PHP Fatal error: Allowed memory size of
>>> 33554432 bytes exhausted (tried to allocate 14 bytes) in
>>> /nfs/rontok/xraid/data/osl/www/www.open-mpi.org/mtt/submit/index.php
>>> on line 1559
>>> ...
>>>
>>> We'll have to somehow be more efficient on these loops.
>>> E.g., line 1559:
>>>
>>> foreach (array_keys($_POST) as $k) {
>>>
>>> Maybe if we broke $_POST up into multiple parts (e.g.,
>>> $_POST_1, $_POST_2, ... $_POST_N)? Maybe we could do
>>> something more efficient than array_keys here? I'm not sure.
>>>
>>> The only workaround on the client side would be to break up
>>> the runs. Maybe do a single MPI Install at a time? Do
>>> ompi/gcc then ompi/intel-9.0 as seperate invocations of the
>>> MTT client.
>>>
>>> Sorry :-(
>>>
>>> -Ethan
>>>
>>>
>>>
>>>>>>>> From the log I see that all tests results were submitted
>>>>>>>> successfully.
>>>>>>>>
>>>>>>>> Can you please check ?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Pasha
>>>>>>>> _______________________________________________
>>>>>>>> mtt-users mailing list
>>>>>>>> mtt-users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>>
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> mtt-users mailing list
>>>>>> mtt-users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>
>>>>>>
>>>>>
>>>> _______________________________________________
>>>> mtt-users mailing list
>>>> mtt-users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>
>>
>
> _______________________________________________
> mtt-users mailing list
> mtt-users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>
>