Subject: Re: [MTT users] Can not find my testing results in OMPI MTT DB
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-07-08 16:11:46
...and/or we could finally make the client capable of breaking up
large sets of submit data into multiple submits to the server. :-)
I unfortunately have no cycles to work on this, but it shouldn't be
*too* hard to do... E.g., the client can loop over submitting N
results at a time until all results have been submitted.
On Jul 8, 2008, at 4:07 PM, Ethan Mallove wrote:
> I see a bunch of these errors in the httpd logs today:
>
> PHP Fatal error: Allowed memory size of 33554432
>
> Could you send your INI file? One way around this issue is
> to submit fewer test results at one time by breaking up your
> large MTT run into multiple runs. If you're running on the
> trunk, this parameter will submit results after each run.
>
> -Ethan
>
>
> On Tue, Jul/08/2008 10:22:47AM, Pavel Shamis (Pasha) wrote:
>> Hi All,
>> I still see that some time MTT (http://www.open-mpi.org/mtt) "lose"
>> my
>> testing results, in the local log I see:
>>
>> ### Test progress: 474 of 474 section tests complete (100%)
>> Submitting to MTTDatabase...
>> MTTDatabase client trying proxy: / Default (none)
>> MTTDatabase proxy successful / not 500
>> MTTDatabase response is a success
>> MTTDatabase client got response:
>> *** WARNING: MTTDatabase client did not get a serial; phases will be
>> isolated from each other in the reports
>> MTTDatabase client submit complete
>>
>> And I can not find these results in DB.
>> Is it any progress with this issue ?
>>
>>
>> Regards.
>> Pasha
>>
>> Ethan Mallove wrote:
>>> On Wed, May/21/2008 09:53:11PM, Pavel Shamis (Pasha) wrote:
>>>
>>>> Oops, in the "MTT server side problem" we discussed other issue.
>>>>
>>>> But anyway I did not see the problem on my server after the
>>>> upgrade :)
>>>>
>>>
>>>
>>> We took *some* steps to alleviate the PHP memory overload
>>> problem (e.g., r668, and then r1119), but evidently there's
>>> more work to do :-)
>>>
>>>
>>>> Pasha
>>>>
>>>> Pavel Shamis (Pasha) wrote:
>>>>
>>>>> I had similar problem on my server. I upgraded the server to
>>>>> latest
>>>>> trunk and the problem disappear.
>>>>> (see "MTT server side problem" thread).
>>>>>
>>>>> Pasha
>>>>>
>>>>> Jeff Squyres wrote:
>>>>>
>>>>>> FWIW: I think we have at least one open ticket on this issue
>>>>>> (break up
>>>>>> submits so that we don't overflow PHP and/or apache).
>>>>>>
>>>
>>> https://svn.open-mpi.org/trac/mtt/ticket/221
>>>
>>> -Ethan
>>>
>>>
>>>>>> On May 21, 2008, at 2:36 PM, Ethan Mallove wrote:
>>>>>>
>>>>>>
>>>>>>> On Wed, May/21/2008 06:46:06PM, Pavel Shamis (Pasha) wrote:
>>>>>>>
>>>>>>>> I sent it directly to your email. Please check.
>>>>>>>> Thanks,
>>>>>>>> Pasha
>>>>>>>>
>>>>>>> Got it. Thanks. It's a PHP memory overload issue.
>>>>>>> (Apparently I didn't look far back enough in the httpd
>>>>>>> error_logs.) See below.
>>>>>>>
>>>>>>>
>>>>>>>> Ethan Mallove wrote:
>>>>>>>>
>>>>>>>>> On Wed, May/21/2008 05:19:44PM, Pavel Shamis (Pasha) wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Jeff Squyres wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Are we running into http max memory problems or http max
>>>>>>>>>>> upload size
>>>>>>>>>>> problems again?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I guess it is some server side issue, you need to check the
>>>>>>>>>> /var/log/httpd/* log on the server.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> The only thing I found in the httpd logs
>>>>>>>>> (/var/log/httpd/www.open-mpi.org/error_log*) was this PHP
>>>>>>>>> warning, which I don't think would result in lost results:
>>>>>>>>>
>>>>>>>>> PHP Warning: array_shift(): The argument should be an array
>>>>>>>>> in
>>>>>>>>> .../submit/index.php on line 1683
>>>>>>>>>
>>>>>>>>> I haven't received any emailed Postgres errors either. When
>>>>>>>>> were these results submitted? I searched for "mellanox" over
>>>>>>>>> the past four days. It seem the results aren't buried in
>>>>>>>>> here, because I see no test run failures ...
>>>>>>>>>
>>>>>>>>> http://www.open-mpi.org/mtt/index.php?do_redir=659
>>>>>>>>>
>>>>>>>>> I'm assuming you're running with two Reporter INI sections:
>>>>>>>>> Textfile and MTTDatabase? Can you send some MTT client
>>>>>>>>> --verbose/--debug output from the below runs?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Ethan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> On May 21, 2008, at 5:28 AM, Pavel Shamis (Pasha) wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Here is test result from my last mtt run:
>>>>>>>>>>>> +-------------+----------------+------+------+----------
>>>>>>>>>>>> +------+
>>>>>>>>>>>> | Phase | Section | Pass | Fail | Time out |
>>>>>>>>>>>> Skip |
>>>>>>>>>>>> +-------------+----------------+------+------+----------
>>>>>>>>>>>> +------+
>>>>>>>>>>>> | MPI install | ompi/gcc | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | MPI install | ompi/intel-9.0 | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | trivial | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | trivial | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | intel-suite | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | intel-suite | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | imb | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | imb | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | presta | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | presta | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | osu_benchmarks | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | osu_benchmarks | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | netpipe | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Build | netpipe | 1 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Run | trivial | 64 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Run | trivial | 64 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Run | intel-suite | 3179 | 165 | 400 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Run | intel-suite | 492 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> +-------------+----------------+------+------+----------
>>>>>>>>>>>> +------+
>>>>>>>>>>>>
>>>>>>>>>>>> In the OMPI MTT DB (http://www.open-mpi.org/mtt) I found
>>>>>>>>>>>> the follow
>>>>>>>>>>>> "test run" results:
>>>>>>>>>>>> | Test Run | trivial | 64 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Run | trivial | 64 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>> | Test Run | intel-suite | 492 | 0 | 0 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>>
>>>>>>>>>>>> And I can not find this one:
>>>>>>>>>>>> | Test Run | intel-suite | 3179 | 165 | 400 |
>>>>>>>>>>>> 0 |
>>>>>>>>>>>>
>>>>>>> Some missing results are in mttdb_debug_file.16.txt (and
>>>>>>> 17.txt), which are the largest .txt files of the bunch. 8
>>>>>>> variants isn't that much, but maybe it causes a problem when
>>>>>>> there's lots of stderr/stdout? I'm surprised
>>>>>>> submit/index.php barfs on files this size:
>>>>>>>
>>>>>>> $ ls -l
>>>>>>> ...
>>>>>>> -rw-r--r-- 1 em162155 staff 956567 May 21 14:21
>>>>>>> mttdb_debug_file.16.inc.gz
>>>>>>> -rw-r--r-- 1 em162155 staff 9603132 May 21 14:09
>>>>>>> mttdb_debug_file.16.txt
>>>>>>> ...
>>>>>>>
>>>>>>> $ client/mtt-submit -h www.open-mpi.org -f mttdb_debug_file.
>>>>>>> 16.txt -z
>>>>>>> -u sun -p sun4sun -d
>>>>>>> LWP::UserAgent::new: ()
>>>>>>> LWP::UserAgent::proxy: http
>>>>>>>
>>>>>>> Filelist: $VAR1 = 'mttdb_debug_file.16.txt';
>>>>>>> LWP::MediaTypes::read_media_types: Reading media types from
>>>>>>> /ws/ompi-tools/lib/perl5/5.8.8/LWP/media.types
>>>>>>> LWP::MediaTypes::read_media_types: Reading media types from
>>>>>>> /usr/perl5/site_perl/5.8.4/LWP/media.types
>>>>>>> LWP::MediaTypes::read_media_types: Reading media types from
>>>>>>> /home/em162155/.mime.types
>>>>>>> LWP::UserAgent::request: ()
>>>>>>> LWP::UserAgent::send_request: POST
>>>>>>> http://www.open-mpi.org/mtt/submit/index.php
>>>>>>> LWP::UserAgent::_need_proxy: Not proxied
>>>>>>> LWP::Protocol::http::request: ()
>>>>>>> LWP::UserAgent::request: Simple response: OK
>>>>>>>
>>>>>>> $ tail -f /var/log/httpd/www.open-mpi.org/error_log | grep -w
>>>>>>> submit
>>>>>>> ...
>>>>>>> [client 192.18.128.5] PHP Fatal error: Allowed memory size of
>>>>>>> 33554432 bytes exhausted (tried to allocate 14 bytes) in
>>>>>>> /nfs/rontok/xraid/data/osl/www/www.open-mpi.org/mtt/submit/index.php
>>>>>>> on line 1559
>>>>>>> ...
>>>>>>>
>>>>>>> We'll have to somehow be more efficient on these loops.
>>>>>>> E.g., line 1559:
>>>>>>>
>>>>>>> foreach (array_keys($_POST) as $k) {
>>>>>>>
>>>>>>> Maybe if we broke $_POST up into multiple parts (e.g.,
>>>>>>> $_POST_1, $_POST_2, ... $_POST_N)? Maybe we could do
>>>>>>> something more efficient than array_keys here? I'm not sure.
>>>>>>>
>>>>>>> The only workaround on the client side would be to break up
>>>>>>> the runs. Maybe do a single MPI Install at a time? Do
>>>>>>> ompi/gcc then ompi/intel-9.0 as seperate invocations of the
>>>>>>> MTT client.
>>>>>>>
>>>>>>> Sorry :-(
>>>>>>>
>>>>>>> -Ethan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>>>>> From the log I see that all tests results were submitted
>>>>>>>>>>>> successfully.
>>>>>>>>>>>>
>>>>>>>>>>>> Can you please check ?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Pasha
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> mtt-users mailing list
>>>>>>>>>>>> mtt-users_at_[hidden]
>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> mtt-users mailing list
>>>>>>>>>> mtt-users_at_[hidden]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> mtt-users mailing list
>>>>>>>> mtt-users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> mtt-users mailing list
>>>>> mtt-users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>
>>>>>
>>>
>>>
>>
>> _______________________________________________
>> mtt-users mailing list
>> mtt-users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
-- Jeff Squyres Cisco Systems