Author Topic: Timeouts with opening a site (e.g. people lookup) (Read 4913 times)

porter · « **on:** September 09, 2006, 09:48:04 pm »

A bit tired and very much tired with how the SLOW response or no response from the AO people lookup almost stops my 0.2.8 today I edited Bot.php and changed the get_site function:

Code: [Select]

        /*
        Gets a URL
        */
        function get_site($url)
        {
                $fp = fopen($url, "r");
                if (!$fp) {
                        $content = ""; // could not connect
                } else {
                        stream_set_blocking($fp, FALSE);
                        stream_set_timeout($fp, 15);
                        $content = "";

                        while ($buffer = fgets($fp, 1024) && stream_get_meta_data($fp)<>'timed_out')
                        {
                                $content .= $buffer;
                        }
                        if (stream_get_meta_data($fp)=='timed_out') $content='';
                }

                return $content;
        }

I'm sure this is not perfect but it seemed to help me today. (I have not tested this more than a few minutes, though...)

I do not know if just setting that 15 second timeout would result in fgets($fp, 1024)===FALSE in the while loop so I include the "&& stream_get_meta_data($fp)<>'timed_out'"

Maybe this is something that has already been handled in the development branch in which case I suppose this belongs in 0.2.x support.

Please correct me if all of this is just bullshit and the server simply happened to behave better after I had done this...

[edit:] Of course, now I get no whereis results at all and have no way to know whether it is because the 15 sec timeout is too short, the code is bugged, or the people lookup is still dead. At least it does not hang the bot...

jjones666 · « **Reply #1 on:** September 10, 2006, 02:14:57 am »

I rigged up a "debug" thing to the whois cache so it tells me if something requested hits the lookup table or not.

Noticed that whenever someone enters the private group, whois cache is queried twice, plus an .xml download is also initiated (necessary or not). Just wondered if anyone had any idea why this is?

I did check all of my modules associated with private group have been changed to use the lookup features of the cache.

-jj-

Khalem · « **Reply #2 on:** September 10, 2006, 03:29:38 pm »

0.3 already solves this and the code could be adapted.

Here is the 0.3 code for reference

Code: [Select]

<?php
 /*
 Gets a URL
 Heavily inspired by Shadowgod's AOXML class and examples from php.net comments.
 */
 function get_site($url, $server_timeout = 5, $read_timeout = 10)
 {
  /*
  Parse the URL so we can use it in our raw socket request
  */
  $get_url = parse_url($url);

  /*
  Open the socket
  */
  $fd = fsockopen($get_url[host], 80, $errno, $errstr, $server_timeout);

  /*
  Make sure the socket was created successfully
  */
  if (!$fd)
  {
   $return["error"] = true;
   $return["errordesc"] = "Errno: $errno Errstr: $errstr";
   return $return;
  }
  else
  {
   /*
   Set the timeout to prevent it from hanging the bot
   */
   socket_set_timeout($fd,$read_timeout);

   /*
   Send the HTTP request
   */
   fputs($fd, "GET $get_url[path] HTTP/1.0\r\n");
   fputs($fd, "Host: $get_url[host]\r\n");
   fputs($fd, "Connection: Close\r\n");
   fputs($fd, "User-Agent: BeBot/$bot_version\r\n\r\n");

   /*
   Check if the server is giving us what we wanted
   */
   $http_response = fgets($fd);

   $results = stream_get_meta_data($fd);
   /*
   Make sure we haven't timed out while waiting for a responce
   */
   if ($results[timed_out] == 1)
   {
    $return["error"] = true;
    $return["errordesc"] = "Timed out while reading from $get_url[host]";
    $return["content"] = "";
    fclose($fd);
    return $return;
   }

   if ( ereg("200 OK", $http_response) )
   {
    $return["error"] = false;
    $return["content"] = "";
    
    /*
    Read the contents
    */
    while (!feof($fd))
    {
     $return["content"] .= fgets($fd, 1024);
    }

    $results = stream_get_meta_data($fd);
    /*
    Make sure we didn't time out while reading the responce again.
    */
    if ($results[timed_out] == 1)
    {
     $return["error"] = true;
     $return["errordesc"] = "Timed out while reading from $get_url[host]";
     $return["content"] = "";
     fclose($fd);
     return $return;
    }

    fclose($fd);
    return $return;
   }
   else
   {
    $return["error"] = true;
    $return["errordesc"] = "Server returned: $http_response";
    fclose($fd);
    return $return;
   }
  }
 }
?>

porter · « **Reply #3 on:** September 10, 2006, 05:43:49 pm »

Ahh, thank you! I should have checked the development version. My solution does not work after all, don't try it anyone

Khalem · « **Reply #4 on:** September 10, 2006, 05:59:16 pm »

The following code should work for 0.2.
Please provide feedback on it.

Code: [Select]

<?php
 /*
 Gets a URL
 Heavily inspired by Shadowgod's AOXML class and examples from php.net comments.
 */
 function get_site($url, $server_timeout = 5, $read_timeout = 10)
 {
  /*
  Parse the URL so we can use it in our raw socket request
  */
  $get_url = parse_url($url);

  /*
  Open the socket
  */
  $fd = fsockopen($get_url[host], 80, $errno, $errstr, $server_timeout);

  /*
  Make sure the socket was created successfully
  */
  if (!$fd)
  {
   return 0;
  }
  else
  {
   /*
   Set the timeout to prevent it from hanging the bot
   */
   socket_set_timeout($fd,$read_timeout);

   /*
   Send the HTTP request
   */
   fputs($fd, "GET $get_url[path] HTTP/1.0\r\n");
   fputs($fd, "Host: $get_url[host]\r\n");
   fputs($fd, "Connection: Close\r\n");
   fputs($fd, "User-Agent: BeBot/$bot_version\r\n\r\n");

   /*
   Check if the server is giving us what we wanted
   */
   $http_response = fgets($fd);

   $results = stream_get_meta_data($fd);
   /*
   Make sure we haven't timed out while waiting for a response
   */
   if ($results[timed_out] == 1)
   {
    fclose($fd);
    return 0;
   }

   if ( ereg("200 OK", $http_response) )
   {
    $content = "";
    
    /*
    Read the contents
    */
    while (!feof($fd))
    {
     $content .= fgets($fd, 1024);
    }

    $results = stream_get_meta_data($fd);
    /*
    Make sure we didn't time out while reading the response again.
    */
    if ($results[timed_out] == 1)
    {
     fclose($fd);
     return 0;
    }

    fclose($fd);
    return $content;
   }
   else
   {
    fclose($fd);
    return 0;
   }
  }
 }
?>

porter · « **Reply #5 on:** September 10, 2006, 06:34:42 pm »

Yes, that works. I just edited it in and have so far no problems.

Dabaron · « **Reply #6 on:** September 11, 2006, 09:36:02 pm »

using what Khalem posted if it was someone not in my whoiscache I got an insta "FC too slow to respond". It looked like it didn't even try and wait at all for a response. is the $server_timeout and $read_timeout in seconds or is that milliseconds?

porter · « **Reply #7 on:** September 11, 2006, 11:10:57 pm »

Quote from: Dabaron on September 11, 2006, 09:36:02 pm

using what Khalem posted if it was someone not in my whoiscache I got an insta "FC too slow to respond". It looked like it didn't even try and wait at all for a response. is the $server_timeout and $read_timeout in seconds or is that milliseconds?

My "solution" did that instant give up even after the lookup server started to behave. Using Khalem's latter code with a 0.2.8 I have no problems at all so far. I have not added a whois cache so unless it is included in 0.2.8 that might be what we have different.

Timeouts in the script have to be seconds not ms - I can quote for a fact:

Quote

stream_set_timeout -- Set timeout period on a stream
Description
bool stream_set_timeout ( resource stream, int seconds [, int microseconds] )

Socket_set_timeout aliases to stream_set_timeout. The server timeout is only defined as float in the PHP documentation but a 5ms timeout in a web server context would make no sense, so I trust it to be seconds as well.

Khalem · « **Reply #8 on:** September 12, 2006, 08:41:06 am »

It is in seconds.

Use the following code to get debug info to console.
0.3 has get_site sending the errors back to the calling functions, but this can't be done in 0.2 without breaking backwards compactibility.

Code: [Select]

<?php
 /*
 Gets a URL
 Heavily inspired by Shadowgod's AOXML class and examples from php.net comments.
 */
 function get_site($url, $server_timeout = 5, $read_timeout = 10)
 {
  /*
  Parse the URL so we can use it in our raw socket request
  */
  $get_url = parse_url($url);

  /*
  Open the socket
  */
  $fd = fsockopen($get_url[host], 80, $errno, $errstr, $server_timeout);

  /*
  Make sure the socket was created successfully
  */
  if (!$fd)
  {
   echo "get_site debug: Creating socket failed\n";
   return 0;
  }
  else
  {
   /*
   Set the timeout to prevent it from hanging the bot
   */
   socket_set_timeout($fd,$read_timeout);

   /*
   Send the HTTP request
   */
   fputs($fd, "GET $get_url[path] HTTP/1.0\r\n");
   fputs($fd, "Host: $get_url[host]\r\n");
   fputs($fd, "Connection: Close\r\n");
   fputs($fd, "User-Agent: BeBot/$bot_version\r\n\r\n");

   /*
   Check if the server is giving us what we wanted
   */
   $http_response = fgets($fd);

   $results = stream_get_meta_data($fd);
   /*
   Make sure we haven't timed out while waiting for a response
   */
   if ($results[timed_out] == 1)
   {
    fclose($fd);
    echo "get_site debug: Socket timed out while waiting for server response\n";
    return 0;
   }

   if ( ereg("200 OK", $http_response) )
   {
    $content = "";
    
    /*
    Read the contents
    */
    while (!feof($fd))
    {
     $content .= fgets($fd, 1024);
    }

    $results = stream_get_meta_data($fd);
    /*
    Make sure we didn't time out while reading the response again.
    */
    if ($results[timed_out] == 1)
    {
     fclose($fd);
     echo "get_site debug: Socket timed out while reading data\n";
     return 0;
    }

    fclose($fd);
    echo "get_site debug: Success\n";
    return $content;
   }
   else
   {
    fclose($fd);
    echo "get_site debug: Server error, http response was: $http_response\n";
    return 0;
   }
  }
 }
?>

Malosar · « **Reply #9 on:** September 12, 2006, 03:21:31 pm »

I'm guessing this would be a good idea to change the function in whois-update.php as well?

Alreadythere · « **Reply #10 on:** September 12, 2006, 03:38:37 pm »

It may be usefull to reduce the timeout in the updatescript a bit, true.

Haven't seen much need though, as I'm running that script daily in the background, and as long as it does it's work, I don't care if it needs 2h or 4h.

Khalem · « **Reply #11 on:** September 12, 2006, 03:46:18 pm »

Quote from: jjones666 on September 10, 2006, 02:14:57 am

I rigged up a "debug" thing to the whois cache so it tells me if something requested hits the lookup table or not.

Noticed that whenever someone enters the private group, whois cache is queried twice, plus an .xml download is also initiated (necessary or not). Just wondered if anyone had any idea why this is?

I did check all of my modules associated with private group have been changed to use the lookup features of the cache.

-jj-

Missed this initially.
I can't say for sure offhand, but i do know that there is a few instances of double work being done right now. It's something i'd like to work on when time allows and is probably going to happen for 0.5/0.6 when the hooking code changes go in.

jjones666 · « **Reply #12 on:** September 12, 2006, 05:18:35 pm »

The teams module was the culprit, I changed that to also use whois cache for information, will post updated file tonite.

-jj-

Khalem · « **Reply #13 on:** September 14, 2006, 02:34:58 am »

In the middle of a reinstall on this system and my main coding box is down for the count with a busted GFX card.
Will try to remember to fix this once i get everything back in order.