Author Topic: Persistent Whois cache with out-of-bot upgrading (Read 53555 times)

Alreadythere · « **on:** December 02, 2005, 12:29:02 pm »

As posted here I am using a cache for all whois entries. Read the linked thread for more information, posting just the files here.

I'm using one bot module, and an out-of-bot cronjob for this (running on linux, but there are ways to automatically run jobs at specific times under most or all OS supported by bebot).

The bebot module for bebot branch 0.2.x
The script to update the cache make sure this script is not in the modules or core directory!
Save as .php
The config file for the update script, add the infos here has to be in the same directory as the update script

Notes:

To use the the cache you need to modify all get_site() calls done to get user information. Just repace the get_site() with $this -> [bot ->] lookup($username), and modify the result handling. The return value of the lookup() function has speaking names, so it shouldn't be hard.
Don't run the script while FC updates it's database, you won't get results then. The update is usually somewhere between 09:00 GMT and 11:00 GMT, I am running the script at 13:00 GMT, once daily, as the DB isn't updated more the once daily either.

Updated version (12/14/2005):

removing stale entries after user-defined time now. Done in the update script, default is set to 36h.
checks for stale entries in the lookup function, after user-set time (default 26h) it automatically queries FC's site for an update, and writes that into the table.

Updated version (12/16/2005):

names of existing orgs get updated now

Updated version (04/17/2006):

smarter script
script divided in config file and update part for easier updating in the future

Updated version (10/03/2006):

set_time_limit(0); set, as long as save_mode isn't enabled the script won't be aborted due to too long execution time
update-script supports used defined addons after all updates are done, use the file whois-update.addon.
mysql_escape_string() and stripslashes() added to remove errors with strange characters in names. First name, last name and org names are escaped.
Added some more information to the output when the update script is done
Modified the cache to use the same field names and order of fields in the table as the version shipped with the 0.3 branch. This is a purely database change, the names in the returned array haven't been changed. So no further modification to bebot files are needed besides replacing the WhoisCache.php. To avoid wrong behaviour of the cache you need to update the table! The SQL code below will do the actual updating, it's in a comment in the WhoisCache.phps file too. Replace whois with the name of your table. Depending on the size of the cache those update commands may take a few minutes to run.

Code: [Select]

ALTER IGNORE TABLE whois ADD id int(15) NOT NULL default '0' FIRST ;
ALTER IGNORE TABLE whois CHANGE name nickname VARCHAR(20) NOT NULL;
ALTER IGNORE TABLE whois ADD pictureurl VARCHAR(100) NOT NULL AFTER org_rank_id, ADD used BIGINT(25) NOT NULL AFTER pictureurl;
ALTER IGNORE TABLE whois ADD INDEX (used); 
ALTER IGNORE TABLE whois ADD INDEX (updated);

Updated version (10/05/2006):

worked around some errors
removed all calls to custom function (tablenames, settings). Cache can now be added without any modifications to Bot.php

Updated version (01/16/2007):

update script handles timeouts on http queries now

Updated version (03/30/2007):

Added fix for the notices in the xmlparse() and get_site() function
updating pictureurl now too, no idea why I forgot that
optimized handling of org members a bit

Alreadythere · « **Reply #1 on:** December 02, 2005, 12:32:33 pm »

Here as example for the changes my current whois_player() function in Whois.php.
If you compare it to the default one in bebot, I just replaced the get_site() call and the further parsing with the lookup() call.

Code: [Select]

    /*
      Returns info about the player
    */
    function whois_player($name)
    {
      $who = $this -> bot -> whois -> lookup($name);

      if (!empty($who["nick"]))
      {
        $at = "<font color=#00ff60>(AT " . $who["at"] . " - " . $who["at_name"] . ")</font> ";
        $result = "\"<font color=#ffff00>" . $who["nick"] . "</font>\"";
        if (!empty($who["firstname"]) && ($who["firstname"] != "Unknown"))
          $result = $who["firstname"] . " " . $result;
        if (!empty($who["lastname"]) && ($who["lastname"] != "Unknown"))
          $result .= " " . $who["lastname"];

        $result .= " is a level <font color=#ffffff>" . $who["level"] . "</font> " . $at . $who["gender"] . " " . $who["breed"] . " <font col
or=#ffffff>" . $who["profession"] . "</font>, " . $who["faction"];
                                if (!empty($who["rank"]))
                                        $result .= ", " . $who["rank"] . " of " . $who["org"] . ".";
                                else
                                        $result .= ".";
      }
      else
        $result = "www.anarchy-online.com was too slow to respond.";

      return $result;
    }

Akarah · « **Reply #2 on:** December 03, 2005, 03:25:35 am »

in your update script, i had to change one line:

Code: [Select]


$sqlquery = "SELECT * FROM " . $tablename . " WHERE updated <= " . $thistime;

for some reason you'd used single-quotes before the SELECT and after the <=.

gave an error that way

worked like a charm otherwise, though. thanks a ton!

Alreadythere · « **Reply #3 on:** December 03, 2005, 01:23:34 pm »

Oh my, yes, you are right there.
I had a fixed tablename in my script (didn't plan on post it at first), and typical for me I didn't test the script I posted. Should start to do that

Another note: you will get some warnings as output if the script tries to update a character that is deleted. Haven't done anything to remove those toons.

Khalem · « **Reply #4 on:** December 04, 2005, 06:22:53 pm »

I like this very much. However i have one request.

Using an external script to update the cache is nice, but i do feel there should be a fallback option for those who do not wish to run an external script.

Add a simple check to the whois that checks the last updated time, if it has not been updated in over 30-36 hours, update the cache for that person.

I still think a file based cache could be better suited though, but this is a step up from the current system regardless.

Alreadythere · « **Reply #5 on:** December 04, 2005, 07:20:49 pm »

Quote from: "Khalem"

Add a simple check to the whois that checks the last updated time, if it has not been updated in over 30-36 hours, update the cache for that person.

I'll try to go around to it, I already got the updated field, so it won't be that hard really (just one check more).

Quote from: "Khalem"

I still think a file based cache could be better suited though, but this is a step up from the current system regardless.

Personally I prefer database systems, as it's hard for any filesystem or self-written function to reach the speed a professional database has in lookups.

Besides, even with 38k entries now in my table the total size is only 4.5mb. Easy to handle memory wise.

Xenixa · « **Reply #6 on:** December 06, 2005, 01:03:05 am »

Been playing with this one today. I like it alot.

I don't mind large db tables as I have enough machine to handle it. What I like most about it is I don't have alot of members using up my band width doing whois againist FC's database.
Doing whois would often times just time out if I had alot of members on chatting away and issuing other commands and what not.

BTW since I run on Win2K (and this works for XP also) if you want to schedule this to run once a day here's a handy document to look at.
http://windows.about.com/od/customizationsforwindows/l/aa001022b.htm

I created a batch file for the Windows Task Scheduler to run, seems to be the easiest method.
w_update.bat

Code: [Select]

@cls
@php w_update.php

Place that .bat in same Folder as php.exe and the Update php file. Which I renamed w_update.php since I have other update scripts I been playing with.

Akarah · « **Reply #7 on:** December 06, 2005, 03:28:28 am »

i have no need for anything beyond what is already done. i may cook up a script to remove deleted characters at some point, but really it's "good enough" for my purposes.

Thanks again, Already

/Aka

Xenixa · « **Reply #8 on:** December 07, 2005, 12:39:56 am »

Modified the Guest Join announcement function I had stuck in Relay_GUILD.php so players joining guest channel don't keep getting looked up on FC's site.
Just insert this at the bottom if your using Already's Whois cache:

Code: [Select]

<?
        /*
   Sends info to private group and guild chat on guest who joined
 */
 function pgjoin($name) 
 {
  $highlight = $this -> highlight;
 // Gets character info from whois cache or anarchy online website
  $who = $this -> bot -> whois -> lookup($name);

      if (!empty($who["nick"]))
      {
        $at = "<font color=#33FF33>(AL " . $who["at"] . " - " . $who["at_name"] . ")</font> ";
        $result = $highlight . "\"" . $who["nick"] . "</font>\"";
        if (!empty($who["firstname"]) && ($who["firstname"] != "Unknown"))
          $result = $who["firstname"] . " " . $result;
        if (!empty($who["lastname"]) && ($who["lastname"] != "Unknown"))
          $result .= " " . $who["lastname"];

        $result .= " a level " . $highlight . $who["level"] . "</font> " . $at . $who["gender"] . " " . $who["breed"] . " " . $highlight . $who["profession"] . "</font>, " . $who["faction"];
  if (!empty($who["rank"]))
   $result .= ", " . $who["rank"] . " of " . $who["org"] . " joined <botname>.";
  else
   $result .= ", joined <botname>.";
      }
    
  $this -> bot -> send_pgroup($result);
  $this -> bot -> send_gc($result);
 }
?>

Note: Also using the highlight color in this one so function Relay needs to look like this in the same module:

Code: [Select]

function Relay (&$bot)
    {
      $this -> bot = &$bot;
      if(!$this -> bot -> highlight_color) {
   $this -> highlight = "<font color=#FFFF00>";
   } else {
   $this -> highlight = "<font color=" . $this -> bot -> highlight_color . ">";
  }
 }

And just one other tid bit. If you want to announce people Parting the bots private channel do this to Online.php -> Look for:

Code: [Select]

/*
      This gets called if someone leaves the privgroup
    */
    function pgleave($name)
    {
      if (isset($this -> pgroup[$name]))
        unset($this -> pgroup[$name]);
    }

Change to:

Code: [Select]

/*
      This gets called if someone leaves the privgroup
    */
    function pgleave($name)
    {
      if (isset($this -> pgroup[$name]))
        unset($this -> pgroup[$name]);
        $this -> bot -> send_gc($name ." has Left the Guest Channel");
    }

Wanuarmi · « **Reply #9 on:** December 11, 2005, 07:29:50 am »

this should come standard with version 3

with an on/off switch of course or something like that

it works like a charm btw ^^
its almost spooky

Akarah · « **Reply #10 on:** December 11, 2005, 05:42:34 pm »

Code: [Select]

mysql> select COUNT(*) from whois;
+----------+
| COUNT(*) |
+----------+
|    41142 |
+----------+

yes.. very spooky

the only change i am considering is one that will let me put it in a database called 'whois' so multiple bots can access it more easily..

Alreadythere · « **Reply #11 on:** December 11, 2005, 06:40:25 pm »

49291 entries, growth definitly slowed down for me.

Quote from: Akarah on December 11, 2005, 05:42:34 pm

the only change i am considering is one that will let me put it in a database called 'whois' so multiple bots can access it more easily..

That's what I'm doing, all my 8 bots access my whois-table.
I've done it using my get_tablenames function, so I can easily share tables across bots. I posted a version of it in the 0.3.0 thread I think.

Xenixa · « **Reply #12 on:** December 14, 2005, 02:19:48 pm »

Well, ummm.. bummer, the offline Character database at FC is now behind a Squid.

Kinda saw this coming. Now if too many requests are made on the those XML files in a short period of time, you'll start to get connection refused messages from the Squid server... bleh

Ex:
While trying to retrieve the URL: http://216.74.158.92:8080/content/community/people/lookup/chars.html

The following error was encountered:

* Connection Failed

The system returned:

(111) Connection refused

The remote host or network may be down. Please try the request again.

Generated Wed, 14 Dec 2005 13:08:36 GMT by www.worldofmidgard.com (Squid/2.2.STABLE5)

Alreadythere · « **Reply #13 on:** December 14, 2005, 04:27:00 pm »

Hm, could it be that they took down their database during the downtime? Haven't checked my logs yet though, not sure what the cache did today...

Alreadythere · « **Reply #14 on:** December 14, 2005, 06:54:47 pm »

Updated version:
- removing stale entries after user-defined time now. Done in the update script, default is set to 36h.
- checks for stale entries in the lookup function, after user-set time (default 26h) it automatically queries FC's site for an update, and writes that into the table.

Updated files in the links in first post.

Author Topic: Persistent Whois cache with out-of-bot upgrading (Read 53555 times)

Alreadythere

Persistent Whois cache with out-of-bot upgrading

Alreadythere

Persistent Whois cache with out-of-bot upgrading

Akarah

Persistent Whois cache with out-of-bot upgrading

Alreadythere

Persistent Whois cache with out-of-bot upgrading

Khalem

Persistent Whois cache with out-of-bot upgrading

Alreadythere

Persistent Whois cache with out-of-bot upgrading

Xenixa

Persistent Whois cache with out-of-bot upgrading

Akarah

Persistent Whois cache with out-of-bot upgrading

Xenixa

Persistent Whois cache with out-of-bot upgrading

Wanuarmi

Re: Persistent Whois cache with out-of-bot upgrading

Akarah

Re: Persistent Whois cache with out-of-bot upgrading

Alreadythere

Re: Persistent Whois cache with out-of-bot upgrading

Xenixa

Re: Persistent Whois cache with out-of-bot upgrading

Alreadythere

Re: Persistent Whois cache with out-of-bot upgrading

Alreadythere

Re: Persistent Whois cache with out-of-bot upgrading