Subject: Re: [MTT users] MTT server side problem
From: Josh Hursey (jjhursey_at_[hidden])
Date: 2008-05-07 21:24:16
Pasha,
I'm looking at the patch a bit closer and even though at a high level
the do_pg_connect, do_pg_query, simple_select, and select functions do
the same thing the versions in submit/index.php have some additional
error handling mechanisms that the ones in database.inc do not have.
Specifically they send email when the functions fail with messages
indicating what failed so corrections can be made.
So though I agree that we should unify the functionality I cannot
recommend this patch since it will result in losing useful error
handling functionality. Maybe there is another way to clean this up to
preserve the error reporting.
-- Josh
On May 7, 2008, at 11:56 AM, Pavel Shamis (Pasha) wrote:
> Hi Josh,
> I had the original problem with some old revision from trunk.
> Today I updated the server to latest revision from trunk + the patch
> and everything looks good.
>
> Can I commit the patch ?
>
> Pasha
>
>
> Ethan Mallove wrote:
>> On Wed, May/07/2008 06:04:07PM, Pavel Shamis (Pasha) wrote:
>>
>>> Hi Josh.
>>>
>>>> Looking at the patch I'm a little bit conserned. The
>>>> "get_table_fields()" is, as you mentioned, no longer used so
>>>> should be removed. However the other functions are critical to
>>>> the submission script particularly 'do_pg_connect' which opens
>>>> the connection to the backend database.
>>>>
>>> All the functions are implemented in $topdir/database.inc file.
>>> And the "database.inc" implementation is better because it use
>>> password and username from config.ini. The original
>>> implementation from submit/index use
>>> hardcoded values defined in the file.
>>>
>>>> Are you using the current development trunk (mtt/trunk) or the
>>>> stable release branch (mtt/branches/ompi-core-testers)?
>>>>
>>> trunk
>>>
>>>> Can you send us the error messages that you were receiving?
>>>>
>>> 1. On client side I see ""*** WARNING: MTTDatabase client did not
>>> get a serial"
>>> As result of the error some of MTT results is not visible via the
>>> web reporter
>>> 2. On server side I found follow error message:
>>> [client 10.4.3.214] PHP Fatal error: Allowed memory size of
>>> 33554432 bytes exhausted (tried to allocate 23592960
>>> bytes) in /.autodirect/swgwork/MTT/mtt/submit/index.php(79) :
>>> eval()'d code on line 77515
>>> [Mon May 05 19:26:05 2008] [notice] caught SIGTERM, shutting down
>>> [Mon May 05 19:30:54 2008] [notice] suEXEC mechanism enabled
>>> (wrapper: /usr/sbin/suexec)
>>> [Mon May 05 19:30:54 2008] [notice] Digest: generating secret for
>>> digest authentication ...
>>> [Mon May 05 19:30:54 2008] [notice] Digest: done
>>> [Mon May 05 19:30:54 2008] [notice] LDAP: Built with OpenLDAP LDAP
>>> SDK
>>> [Mon May 05 19:30:54 2008] [notice] LDAP: SSL support unavailable
>>> My memory limit in php.ini file was set on 256MB !
>>>
>>
>>
>> Looks like PHP is actually using a 32MB limit ("Allowed
>> memory size of 33554432 ..."). Does a (Apache?) daemon need
>> to be restarted for the php.ini file to take effect? To
>> check your settings, this little PHP script will print an
>> HTML page of all the active system settings (search on
>> "memory_limit").
>> <?php
>> phpinfo();
>> ?>
>>
>> -Ethan
>>
>>
>>
>>> Regards,
>>> Pasha
>>>
>>>
>>>> Cheers,
>>>> Josh
>>>>
>>>> On May 7, 2008, at 4:49 AM, Pavel Shamis (Pasha) wrote:
>>>>
>>>>
>>>>> Hi,
>>>>> I upgraded the server side (the mtt is still running , so don't
>>>>> know if the problem was resolved)
>>>>> During upgrade I had some problem with the submit/index.php
>>>>> script, it had some duplicated functions and some of them were
>>>>> broken.
>>>>> Please review the attached patch.
>>>>>
>>>>> Pasha
>>>>>
>>>>> Ethan Mallove wrote:
>>>>>
>>>>>> On Tue, May/06/2008 06:29:33PM, Pavel Shamis (Pasha) wrote:
>>>>>>
>>>>>>
>>>>>>>> I'm not sure which cron jobs you're referring to. Do you
>>>>>>>> mean these?
>>>>>>>>
>>>>>>>> https://svn.open-mpi.org/trac/mtt/browser/trunk/server/php/cron
>>>>>>>>
>>>>>>>>
>>>>>>> I talked about this one: https://svn.open-mpi.org/trac/mtt/wiki/ServerMaintenance
>>>>>>>
>>>>>>>
>>>>>> I'm guessing you would only be concerned with the below
>>>>>> periodic-maintenance.pl script, which just runs
>>>>>> ANALYZE/VACUUM queries. I think you can start that up
>>>>>> whenever you want (and it should optimize the Reporter).
>>>>>>
>>>>>> https://svn.open-mpi.org/trac/mtt/browser/trunk/server/sql/cron/periodic-maintenance.pl
>>>>>>
>>>>>> -Ethan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> The only thing there are the regular
>>>>>>>> mtt-results_at_[hidden] email alerts and some out-of-date
>>>>>>>> DB monitoring junk. You can ignore that stuff.
>>>>>>>>
>>>>>>>> Josh, are there some nightly (DB
>>>>>>>> pruning/cleaning/vacuuming?) cron jobs that Pasha should be
>>>>>>>> running?
>>>>>>>>
>>>>>>>> -Ethan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Ethan Mallove wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi Pasha,
>>>>>>>>>>
>>>>>>>>>> I thought this issue was solved in r1119 (see below). Do you
>>>>>>>>>> have the latest mtt/server scripts?
>>>>>>>>>>
>>>>>>>>>> https://svn.open-mpi.org/trac/mtt/changeset/1119/trunk/server/php/submit
>>>>>>>>>>
>>>>>>>>>> -Ethan
>>>>>>>>>>
>>>>>>>>>> On Tue, May/06/2008 03:26:43PM, Pavel Shamis (Pasha) wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> About the issue:
>>>>>>>>>>> 1. On client side I see ""*** WARNING: MTTDatabase client
>>>>>>>>>>> did not get a serial"
>>>>>>>>>>> As result of the error some of MTT results is not visible
>>>>>>>>>>> via the web reporter
>>>>>>>>>>> 2. On server side I found follow error message:
>>>>>>>>>>> [client 10.4.3.214] PHP Fatal error: Allowed memory size
>>>>>>>>>>> of 33554432 bytes exhausted (tried to allocate 23592960
>>>>>>>>>>> bytes) in /.autodirect/swgwork/MTT/mtt/submit/
>>>>>>>>>>> index.php(79) : eval()'d code on line 77515
>>>>>>>>>>> [Mon May 05 19:26:05 2008] [notice] caught SIGTERM,
>>>>>>>>>>> shutting down
>>>>>>>>>>> [Mon May 05 19:30:54 2008] [notice] suEXEC mechanism
>>>>>>>>>>> enabled (wrapper: /usr/sbin/suexec)
>>>>>>>>>>> [Mon May 05 19:30:54 2008] [notice] Digest: generating
>>>>>>>>>>> secret for digest authentication ...
>>>>>>>>>>> [Mon May 05 19:30:54 2008] [notice] Digest: done
>>>>>>>>>>> [Mon May 05 19:30:54 2008] [notice] LDAP: Built with
>>>>>>>>>>> OpenLDAP LDAP SDK
>>>>>>>>>>> [Mon May 05 19:30:54 2008] [notice] LDAP: SSL support
>>>>>>>>>>> unavailable
>>>>>>>>>>> My memory limit in php.ini file was set on 256MB !
>>>>>>>>>>>
>>>>>>>>>>> Any ideas ?
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Pavel Shamis (Pasha)
>>>>>>>>>>> Mellanox Technologies
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> mtt-users mailing list
>>>>>>>>>>> mtt-users_at_[hidden]
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Pavel Shamis (Pasha)
>>>>>>>>> Mellanox Technologies
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> mtt-users mailing list
>>>>>>>>> mtt-users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>>>
>>>>>>>>>
>>>>>>> --
>>>>>>> Pavel Shamis (Pasha)
>>>>>>> Mellanox Technologies
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> mtt-users mailing list
>>>>>>> mtt-users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>> --
>>>>> Pavel Shamis (Pasha)
>>>>> Mellanox Technologies
>>>>>
>>>>> Index: submit/index.php
>>>>> =
>>>>> ==================================================================
>>>>> --- submit/index.php (revision 1200)
>>>>> +++ submit/index.php (working copy)
>>>>> @@ -1,6 +1,7 @@
>>>>> <?php
>>>>> # Copyright (c) 2006 Cisco Systems, Inc. All rights
>>>>> reserved.
>>>>> # Copyright (c) 2006-2007 Sun Microsystems, Inc. All rights
>>>>> reserved.
>>>>> +# Copyright (c) 2008 Mellanox Technologies. All rights
>>>>> reserved.
>>>>>
>>>>> #
>>>>> #
>>>>> @@ -24,8 +25,7 @@ if (file_exists("$topdir/config.inc")) {
>>>>> ini_set("memory_limit", "32M");
>>>>>
>>>>> $topdir = '..';
>>>>> -$ompi_home = '/l/osl/www/doc/www.open-mpi.org';
>>>>> -include_once("$ompi_home/dbpassword.inc");
>>>>> +include_once("$topdir/database.inc");
>>>>> include_once("$topdir/reporter.inc");
>>>>>
>>>>>
>>>>> @@ -1465,60 +1465,6 @@ function get_table_indexes($table_name,
>>>>> return simple_select($sql_cmd);
>>>>> }
>>>>>
>>>>> -# Function used to determine which _POST fields
>>>>> -# to INSERT. Prevent non-existent fields from being
>>>>> -# INSERTed
>>>>> -function get_table_fields($table_name) {
>>>>> -
>>>>> - global $dbname;
>>>>> - global $id;
>>>>> -
>>>>> - # These indexes are special in that they link phases
>>>>> - # together and hence, can and do show up in _POST
>>>>> - if ($table_name == "test_build")
>>>>> - $special_indexes = array("mpi_install$id");
>>>>> - elseif ($table_name == "test_run")
>>>>> - $special_indexes = array("test_build$id");
>>>>> -
>>>>> - # Crude way to tell whether a field is an index
>>>>> - $is_not_index_clause =
>>>>> - "\n\t (table_name = '$table_name' AND NOT " .
>>>>> - "\n\t (data_type = 'integer' AND " .
>>>>> - "\n\t column_name ~ '_id$' AND " .
>>>>> - "\n\t table_catalog = '$dbname'))";
>>>>> -
>>>>> - $is_special_index_clause =
>>>>> - "\n\t (table_name = '$table_name' AND " .
>>>>> - "\n\t (column_name = '$special_indexes[0]'))";
>>>>> -
>>>>> - $is_index_columns = array(
>>>>> - "column_name",
>>>>> - "data_type",
>>>>> - "column_default");
>>>>> -
>>>>> - $sql_cmd = "\n SELECT " . join(",",$is_index_columns) .
>>>>> - "\n\t FROM information_schema.columns WHERE " .
>>>>> - "\n\t " .
>>>>> - $is_not_index_clause . " OR " .
>>>>> - $is_special_index_clause . ';';
>>>>> -
>>>>> - do_pg_connect();
>>>>> -
>>>>> - # This table will be easier to manage if it's
>>>>> - # keyed by column, instead of index
>>>>> - $tmp = array();
>>>>> - $arr = array();
>>>>> - $arr = select($sql_cmd);
>>>>> -
>>>>> - foreach ($is_index_columns as $col) {
>>>>> - $tmp[$col] = array();
>>>>> - for ($i = 0; $i < sizeof($arr); $i++) {
>>>>> - $tmp[$col][] = $arr[$i][$col];
>>>>> - }
>>>>> - }
>>>>> - return $tmp;
>>>>> -}
>>>>> -
>>>>> #
>>>>> # Useful display of all the parameters posted.
>>>>> # Warning: this function could explode the memory footprint
>>>>> causing the
>>>>> @@ -1622,90 +1568,6 @@ function is_null_($var) {
>>>>>
>>>>> ######################################################################
>>>>>
>>>>> -function do_pg_connect() {
>>>>> -
>>>>> - global $dbname;
>>>>> - global $user;
>>>>> - global $pass;
>>>>> - global $pgsql_conn;
>>>>> - static $connected = false;
>>>>> -
>>>>> - if (!$connected) {
>>>>> - $pgsql_conn =
>>>>> - pg_connect("host=localhost port=5432 dbname=$dbname
>>>>> user=$user password=$pass");
>>>>> -
>>>>> - # Exit if we cannot connect
>>>>> - if (!$pgsql_conn) {
>>>>> - mtt_abort("\nCould not connect to the $dbname
>>>>> database; " .
>>>>> - "submit this run later.");
>>>>> - }
>>>>> - else {
>>>>> - $connected = true;
>>>>> -
>>>>> - # Serialize all transactions by default - Safer
>>>>> that way.
>>>>> - do_pg_query("SET SESSION CHARACTERISTICS AS
>>>>> TRANSACTION ISOLATION LEVEL SERIALIZABLE", false);
>>>>> - }
>>>>> -
>>>>> - # pg_set_error_verbosity($pgsql_conn,
>>>>> PGSQL_ERRORS_VERBOSE); # PHP 5 needed
>>>>> - # pg_trace($_ENV['HOME'] . "/pgsql.trace", 'w',
>>>>> $pgsql_conn);
>>>>> - # debug("\npg_options: " .
>>>>> var_export(pg_options($pgsql_conn)));
>>>>> - }
>>>>> -}
>>>>> -
>>>>> -function do_pg_query($cmd, $silent) {
>>>>> - $db_res = null;
>>>>> -
>>>>> - do_pg_connect();
>>>>> -
>>>>> - debug("\nSQL: $cmd\n");
>>>>> - if (! ($db_res = pg_query($cmd))) {
>>>>> - $out = "\nSQL QUERY: " . $cmd .
>>>>> - "\nSQL ERROR: " . pg_last_error() .
>>>>> - "\nSQL ERROR: " . pg_result_error();
>>>>> -
>>>>> - # Some errors are unsurprising, allow for silence in
>>>>> - # such cases
>>>>> - if (! $silent) {
>>>>> - mtt_error($out);
>>>>> - mtt_send_mail($out);
>>>>> - }
>>>>> - }
>>>>> - debug("\nDatabase rows affected: " .
>>>>> pg_affected_rows($db_res) . "\n");
>>>>> -
>>>>> - return $db_res;
>>>>> -}
>>>>> -
>>>>> -# Fetch scalar value
>>>>> -function select_scalar($cmd) {
>>>>> -
>>>>> - $set = array();
>>>>> - $set = simple_select($cmd);
>>>>> - return array_shift($set);
>>>>> -}
>>>>> -
>>>>> -# Fetch 1D array
>>>>> -function simple_select($cmd) {
>>>>> -
>>>>> - do_pg_connect();
>>>>> -
>>>>> - $rows = null;
>>>>> -
>>>>> - debug("\nSQL: $cmd\n");
>>>>> - if (! ($result = pg_query($cmd))) {
>>>>> - $out = "\nSQL QUERY: " . $cmd .
>>>>> - "\nSQL ERROR: " . pg_last_error() .
>>>>> - "\nSQL ERROR: " . pg_result_error();
>>>>> - mtt_error($out);
>>>>> - mtt_send_mail($out);
>>>>> - }
>>>>> - $max = pg_num_rows($result);
>>>>> - for ($i = 0; $i < $max; ++$i) {
>>>>> - $row = pg_fetch_array($result, $i, PGSQL_NUM);
>>>>> - $rows[] = $row[0];
>>>>> - }
>>>>> - return $rows;
>>>>> -}
>>>>> -
>>>>> # Fetch an associative hash (column name => value)
>>>>> function associative_select($cmd) {
>>>>>
>>>>> @@ -1722,21 +1584,6 @@ function associative_select($cmd) {
>>>>> return pg_fetch_array($result);
>>>>> }
>>>>>
>>>>> -# Fetch 2D array
>>>>> -function select($cmd) {
>>>>> - do_pg_connect();
>>>>> -
>>>>> - debug("\nSQL: $cmd\n");
>>>>> - if (! ($result = pg_query($cmd))) {
>>>>> - $out = "\nSQL QUERY: " . $cmd .
>>>>> - "\nSQL ERROR: " . pg_last_error() .
>>>>> - "\nSQL ERROR: " . pg_result_error();
>>>>> - mtt_error($out);
>>>>> - mtt_send_mail($out);
>>>>> - }
>>>>> - return pg_fetch_all($result);
>>>>> -}
>>>>> -
>>>>> ######################################################################
>>>>>
>>>>> # Function for reporting errors back to the client
>>>>>
>>>>
>>> --
>>> Pavel Shamis (Pasha)
>>> Mellanox Technologies
>>>
>>> _______________________________________________
>>> mtt-users mailing list
>>> mtt-users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/mtt-users
>>>
>>
>>
>
>
> --
> Pavel Shamis (Pasha)
> Mellanox Technologies