collapse collapse
* User Info
 
 
Welcome, Guest. Please login or register.
* Search

* Board Stats
  • stats Total Members: 989
  • stats Total Posts: 18366
  • stats Total Topics: 2501
  • stats Total Categories: 7
  • stats Total Boards: 35
  • stats Most Online: 1144

Author Topic: Persistent Whois cache with out-of-bot upgrading  (Read 85105 times)

0 Members and 2 Guests are viewing this topic.

Offline Alreadythere

  • BeBot Maintainer
  • BeBot Hero
  • ******
  • Posts: 1288
  • Karma: +0/-0
Persistent Whois cache with out-of-bot upgrading
« on: December 02, 2005, 12:29:02 pm »
As posted here I am using a cache for all whois entries. Read the linked thread for more information, posting just the files here.

I'm using one bot module, and an out-of-bot cronjob for this (running on linux, but there are ways to automatically run jobs at specific times under most or all OS supported by bebot).

The bebot module for bebot branch 0.2.x
The script to update the cache make sure this script is not in the modules or core directory!
Save as .php
The config file for the update script, add the infos here has to be in the same directory as the update script

Notes:
  • To use the the cache you need to modify all get_site() calls done to get user information. Just repace the get_site() with $this -> [bot ->] lookup($username), and modify the result handling. The return value of the lookup() function has speaking names, so it shouldn't be hard.
  • Don't run the script while FC updates it's database, you won't get results then. The update is usually somewhere between 09:00 GMT and 11:00 GMT, I am running the script at 13:00 GMT, once daily, as the DB isn't updated more the once daily either.

Updated version (12/14/2005):
  • removing stale entries after user-defined time now. Done in the update script, default is set to 36h.
  • checks for stale entries in the lookup function, after user-set time (default 26h) it automatically queries FC's site for an update, and writes that into the table.

Updated version (12/16/2005):
  • names of existing orgs get updated now

Updated version (04/17/2006):
  • smarter script
  • script divided in config file and update part for easier updating in the future

Updated version (10/03/2006):
  • set_time_limit(0); set, as long as save_mode isn't enabled the script won't be aborted due to too long execution time
  • update-script supports used defined addons after all updates are done, use the file whois-update.addon.
  • mysql_escape_string() and stripslashes() added to remove errors with strange characters in names. First name, last name and org names are escaped.
  • Added some more information to the output when the update script is done
  • Modified the cache to use the same field names and order of fields in the table as the version shipped with the 0.3 branch. This is a purely database change, the names in the returned array haven't been changed. So no further modification to bebot files are needed besides replacing the WhoisCache.php. To avoid wrong behaviour of the cache you need to update the table! The SQL code below will do the actual updating, it's in a comment in the WhoisCache.phps file too. Replace whois with the name of your table. Depending on the size of the cache those update commands may take a few minutes to run.

Code: [Select]
ALTER IGNORE TABLE whois ADD id int(15) NOT NULL default '0' FIRST ;
ALTER IGNORE TABLE whois CHANGE name nickname VARCHAR(20) NOT NULL;
ALTER IGNORE TABLE whois ADD pictureurl VARCHAR(100) NOT NULL AFTER org_rank_id, ADD used BIGINT(25) NOT NULL AFTER pictureurl;
ALTER IGNORE TABLE whois ADD INDEX (used);
ALTER IGNORE TABLE whois ADD INDEX (updated);

Updated version (10/05/2006):
  • worked around some errors
  • removed all calls to custom function (tablenames, settings). Cache can now be added without any modifications to Bot.php

Updated version (01/16/2007):
  • update script handles timeouts on http queries now

Updated version (03/30/2007):
  • Added fix for the notices in the xmlparse() and get_site() function
  • updating pictureurl now too, no idea why I forgot that
  • optimized handling of org members a bit
« Last Edit: March 30, 2007, 01:20:43 pm by Alreadythere »

Offline Alreadythere

  • BeBot Maintainer
  • BeBot Hero
  • ******
  • Posts: 1288
  • Karma: +0/-0
Persistent Whois cache with out-of-bot upgrading
« Reply #1 on: December 02, 2005, 12:32:33 pm »
Here as example for the changes my current whois_player() function in Whois.php.
If you compare it to the default one in bebot, I just replaced the get_site() call and the further parsing with the lookup() call.

Code: [Select]
   /*
      Returns info about the player
    */
    function whois_player($name)
    {
      $who = $this -> bot -> whois -> lookup($name);

      if (!empty($who["nick"]))
      {
        $at = "<font color=#00ff60>(AT " . $who["at"] . " - " . $who["at_name"] . ")</font> ";
        $result = "\"<font color=#ffff00>" . $who["nick"] . "</font>\"";
        if (!empty($who["firstname"]) && ($who["firstname"] != "Unknown"))
          $result = $who["firstname"] . " " . $result;
        if (!empty($who["lastname"]) && ($who["lastname"] != "Unknown"))
          $result .= " " . $who["lastname"];

        $result .= " is a level <font color=#ffffff>" . $who["level"] . "</font> " . $at . $who["gender"] . " " . $who["breed"] . " <font col
or=#ffffff>" . $who["profession"] . "</font>, " . $who["faction"];
                                if (!empty($who["rank"]))
                                        $result .= ", " . $who["rank"] . " of " . $who["org"] . ".";
                                else
                                        $result .= ".";
      }
      else
        $result = "www.anarchy-online.com was too slow to respond.";

      return $result;
    }

Offline Akarah

  • Contributor
  • *******
  • Posts: 72
  • Karma: +0/-0
    • http://synergyfactor.net/
Persistent Whois cache with out-of-bot upgrading
« Reply #2 on: December 03, 2005, 03:25:35 am »
in your update script, i had to change one line:

Code: [Select]

$sqlquery = "SELECT * FROM " . $tablename . " WHERE updated <= " . $thistime;


for some reason you'd used single-quotes before the SELECT and after the <=.

gave an error that way :)


worked like a charm otherwise, though. thanks a ton! :)

Offline Alreadythere

  • BeBot Maintainer
  • BeBot Hero
  • ******
  • Posts: 1288
  • Karma: +0/-0
Persistent Whois cache with out-of-bot upgrading
« Reply #3 on: December 03, 2005, 01:23:34 pm »
Oh my, yes, you are right there.
I had a fixed tablename in my script (didn't plan on post it at first), and typical for me I didn't test the script I posted. Should start to do that :)

Another note: you will get some warnings as output if the script tries to update a character that is deleted. Haven't done anything to remove those toons.

Offline Khalem

  • BeBot Founder
  • Administrator
  • ********
  • Posts: 1169
  • Karma: +0/-0
    • http://www.ancarim.com
Persistent Whois cache with out-of-bot upgrading
« Reply #4 on: December 04, 2005, 06:22:53 pm »
I like this very much. However i have one request.

Using an external script to update the cache is nice, but i do feel there should be a fallback option for those who do not wish to run an external script.

Add a simple check to the whois that checks the last updated time, if it has not been updated in over 30-36 hours, update the cache for that person.

I still think a file based cache could be better suited though, but this is a step up from the current system regardless.
BeBot Founder and Fixer Kingpin

Offline Alreadythere

  • BeBot Maintainer
  • BeBot Hero
  • ******
  • Posts: 1288
  • Karma: +0/-0
Persistent Whois cache with out-of-bot upgrading
« Reply #5 on: December 04, 2005, 07:20:49 pm »
Quote from: "Khalem"
Add a simple check to the whois that checks the last updated time, if it has not been updated in over 30-36 hours, update the cache for that person.

I'll try to go around to it, I already got the updated field, so it won't be that hard really (just one check more).

Quote from: "Khalem"
I still think a file based cache could be better suited though, but this is a step up from the current system regardless.

Personally I prefer database systems, as it's hard for any filesystem or self-written function to reach the speed a professional database has in lookups.

Besides, even with 38k entries now in my table the total size is only 4.5mb. Easy to handle memory wise.

Offline Xenixa

  • Contributor
  • *******
  • Posts: 307
  • Karma: +0/-0
Persistent Whois cache with out-of-bot upgrading
« Reply #6 on: December 06, 2005, 01:03:05 am »
Been playing with this one today. I like it alot. :)
I don't mind large db tables as I have enough machine to handle it. What I like most about it is I don't have alot of members using up my band width doing whois againist FC's database.
Doing whois would often times just time out if I had alot of members on chatting away and issuing other commands and what not.

BTW since I run on Win2K (and this works for XP also) if you want to schedule this to run once a day here's a handy document to look at.
http://windows.about.com/od/customizationsforwindows/l/aa001022b.htm

I created a batch file for the Windows Task Scheduler to run, seems to be the easiest method.
w_update.bat
Code: [Select]
@cls
@php w_update.php
Place that .bat in same Folder as php.exe and the Update php file. Which I renamed w_update.php since I have other update scripts I been playing with.
<<< Hack's in Zend Studio

All my Custom Bebot files may be Found Here <-clicky

Offline Akarah

  • Contributor
  • *******
  • Posts: 72
  • Karma: +0/-0
    • http://synergyfactor.net/
Persistent Whois cache with out-of-bot upgrading
« Reply #7 on: December 06, 2005, 03:28:28 am »
i have no need for anything beyond what is already done. i may cook up a script to remove deleted characters at some point, but really it's "good enough" for my purposes.

Thanks again, Already ;)


/Aka

Offline Xenixa

  • Contributor
  • *******
  • Posts: 307
  • Karma: +0/-0
Persistent Whois cache with out-of-bot upgrading
« Reply #8 on: December 07, 2005, 12:39:56 am »
Modified the Guest Join announcement function I had stuck in Relay_GUILD.php so players joining guest channel don't keep getting looked up on FC's site.
Just insert this at the bottom if your using Already's Whois cache:
Code: [Select]
<?
        /*
  Sends info to private group and guild chat on guest who joined
*/
function pgjoin($name)
{
$highlight = $this -> highlight;
// Gets character info from whois cache or anarchy online website
$who = $this -> bot -> whois -> lookup($name);

      if (!empty($who["nick"]))
      {
        $at = "<font color=#33FF33>(AL " . $who["at"] . " - " . $who["at_name"] . ")</font> ";
        $result = $highlight . "\"" . $who["nick"] . "</font>\"";
        if (!empty($who["firstname"]) && ($who["firstname"] != "Unknown"))
          $result = $who["firstname"] . " " . $result;
        if (!empty($who["lastname"]) && ($who["lastname"] != "Unknown"))
          $result .= " " . $who["lastname"];

        $result .= " a level " . $highlight . $who["level"] . "</font> " . $at . $who["gender"] . " " . $who["breed"] . " " . $highlight . $who["profession"] . "</font>, " . $who["faction"];
if (!empty($who["rank"]))
$result .= ", " . $who["rank"] . " of " . $who["org"] . " joined <botname>.";
else
$result .= ", joined <botname>.";
      }

$this -> bot -> send_pgroup($result);
$this -> bot -> send_gc($result);
}
?>
Note: Also using the highlight color in this one so function Relay needs to look like this in the same module:
Code: [Select]
function Relay (&$bot)
    {
      $this -> bot = &$bot;
      if(!$this -> bot -> highlight_color) {
$this -> highlight = "<font color=#FFFF00>";
} else {
$this -> highlight = "<font color=" . $this -> bot -> highlight_color . ">";
}
}
And just one other tid bit. If you want to announce people Parting the bots private channel do this to Online.php -> Look for:
Code: [Select]
/*
      This gets called if someone leaves the privgroup
    */
    function pgleave($name)
    {
      if (isset($this -> pgroup[$name]))
        unset($this -> pgroup[$name]);
    }
Change to:
Code: [Select]
/*
      This gets called if someone leaves the privgroup
    */
    function pgleave($name)
    {
      if (isset($this -> pgroup[$name]))
        unset($this -> pgroup[$name]);
        $this -> bot -> send_gc($name ." has Left the Guest Channel");
    }
« Last Edit: December 11, 2005, 09:19:02 pm by Xenixa »
<<< Hack's in Zend Studio

All my Custom Bebot files may be Found Here <-clicky

Offline Wanuarmi

  • Contributor
  • *******
  • Posts: 121
  • Karma: +0/-0
Re: Persistent Whois cache with out-of-bot upgrading
« Reply #9 on: December 11, 2005, 07:29:50 am »
this should come standard with version 3 :) with an on/off switch of course or something like that

it works like a charm btw ^^
its almost spooky
« Last Edit: December 11, 2005, 08:09:00 am by Wanuarmi »

Offline Akarah

  • Contributor
  • *******
  • Posts: 72
  • Karma: +0/-0
    • http://synergyfactor.net/
Re: Persistent Whois cache with out-of-bot upgrading
« Reply #10 on: December 11, 2005, 05:42:34 pm »
Code: [Select]
mysql> select COUNT(*) from whois;
+----------+
| COUNT(*) |
+----------+
|    41142 |
+----------+


yes.. very spooky ;)

the only change i am considering is one that will let me put it in a database called 'whois' so multiple bots can access it more easily..

Offline Alreadythere

  • BeBot Maintainer
  • BeBot Hero
  • ******
  • Posts: 1288
  • Karma: +0/-0
Re: Persistent Whois cache with out-of-bot upgrading
« Reply #11 on: December 11, 2005, 06:40:25 pm »
49291 entries, growth definitly slowed down for me.

the only change i am considering is one that will let me put it in a database called 'whois' so multiple bots can access it more easily..
That's what I'm doing, all my 8 bots access my whois-table.
I've done it using my get_tablenames function, so I can easily share tables across bots. I posted a version of it in the 0.3.0 thread I think.

Offline Xenixa

  • Contributor
  • *******
  • Posts: 307
  • Karma: +0/-0
Re: Persistent Whois cache with out-of-bot upgrading
« Reply #12 on: December 14, 2005, 02:19:48 pm »
Well, ummm.. bummer, the offline Character database at FC is now behind a Squid.

Kinda saw this coming. Now if too many requests are made on the those XML files in a short period of time, you'll start to get connection refused messages from the Squid server... bleh

Ex:
While trying to retrieve the URL: http://216.74.158.92:8080/content/community/people/lookup/chars.html

The following error was encountered:

    * Connection Failed

The system returned:

    (111) Connection refused

The remote host or network may be down. Please try the request again.

Generated Wed, 14 Dec 2005 13:08:36 GMT by www.worldofmidgard.com (Squid/2.2.STABLE5)
<<< Hack's in Zend Studio

All my Custom Bebot files may be Found Here <-clicky

Offline Alreadythere

  • BeBot Maintainer
  • BeBot Hero
  • ******
  • Posts: 1288
  • Karma: +0/-0
Re: Persistent Whois cache with out-of-bot upgrading
« Reply #13 on: December 14, 2005, 04:27:00 pm »
Hm, could it be that they took down their database during the downtime? Haven't checked my logs yet though, not sure what the cache did today...

Offline Alreadythere

  • BeBot Maintainer
  • BeBot Hero
  • ******
  • Posts: 1288
  • Karma: +0/-0
Re: Persistent Whois cache with out-of-bot upgrading
« Reply #14 on: December 14, 2005, 06:54:47 pm »
Updated version:
- removing stale entries after user-defined time now. Done in the update script, default is set to 36h.
- checks for stale entries in the lookup function, after user-set time (default 26h) it automatically queries FC's site for an update, and writes that into the table.

Updated files in the links in first post.

 

* Recent Posts
Com bot module by bitnykk
[November 25, 2024, 05:36:11 pm ]


0.8.x updates for AO by bitnykk
[June 23, 2024, 03:19:47 pm ]


0.8.x updates for AoC by bitnykk
[June 23, 2024, 03:19:44 pm ]


[AoC] special char for items module by bitnykk
[February 09, 2024, 09:41:18 pm ]


BeBot still alive & kicking ! by bitnykk
[December 17, 2023, 12:58:44 am ]

* Who's Online
  • Dot Guests: 231
  • Dot Hidden: 0
  • Dot Users: 0

There aren't any users online.
* Forum Staff
bitnykk admin bitnykk
Administrator
Khalem admin Khalem
Administrator
WeZoN gmod WeZoN
Global Moderator
SimplePortal 2.3.7 © 2008-2024, SimplePortal