Author Topic: Persistent Whois cache with out-of-bot upgrading (Read 85080 times)

Alreadythere · « **Reply #15 on:** December 14, 2005, 06:56:03 pm »

Tested my script after your report Xenica, didn't have any error messeges besides the ones about non-existing HTML sites. And only a few there either, still think those are no longer existing characters.

Xenixa · « **Reply #16 on:** December 15, 2005, 02:31:06 am »

Thinking now that error occured as they were updating their network stuff.
The AO website is behind a Squid now it seems. As long as there squid isn't to slow at caching we should be ok

Alreadythere · « **Reply #17 on:** December 16, 2005, 11:44:42 pm »

Updated the script, forgot to check for renames of orgs. Org names get updated to new ones now, only change.

Solerna · « **Reply #18 on:** December 22, 2005, 03:34:04 am »

Hmm I've suddenly started to get

"b>Warning</b>: fopen(http://www.anarchy-online.com/character/bio/d/2/name/noahsdove/bio.xml): failed to open stream: HTTP request failed! HTTP/1.0 503 Service Unavailable

When using this, anyone else has?

Xenixa · « **Reply #19 on:** December 22, 2005, 07:57:39 am »

I get that also .. repeatedly for a group of names. I think the FC servers can't keep up with the requests.

Which reminds me. Been meaning to ask. Why pull individual character Bio's if they belong to an org? Should pull the Org XML file and parse that for the character data. Would be less http requests that way. You would need to keep a routine to handle non-org'd characters sure, but for the script update I think the Org XML file should be used. Just my 2 cents.

Alreadythere · « **Reply #20 on:** December 22, 2005, 08:53:17 am »

Quote from: Solerna on December 22, 2005, 03:34:04 am

Hmm I've suddenly started to get

"b>Warning</b>: fopen(http://www.anarchy-online.com/character/bio/d/2/name/noahsdove/bio.xml): failed to open stream: HTTP request failed! HTTP/1.0 503 Service Unavailable

When using this, anyone else has?

Hm, same in my logs. Strange thing is - if I query the pages manually I get a correct reply.
Could be that the speed is too high, so some DOS protection stops the further queries. If that's the case, a small delay between each HTTP query could help. But just guessing here, don't have a real clue.

Quote from: Xenixa on December 22, 2005, 07:57:39 am

Which reminds me. Been meaning to ask. Why pull individual character Bio's if they belong to an org? Should pull the Org XML file and parse that for the character data. Would be less http requests that way. You would need to keep a routine to handle non-org'd characters sure, but for the script update I think the Org XML file should be used. Just my 2 cents.

Check my update script again please

It gets a list of all org IDs in the cache first, then updates the entries for all members of those orgs. And only after that it queries the bios for all not yet updated characters.

ZubZero · « **Reply #21 on:** January 15, 2006, 04:28:43 pm »

I would like a version that stores the cache in CVS, cause I dont have the db on the same computer as the bot.

Alreadythere · « **Reply #22 on:** January 15, 2006, 04:48:49 pm »

Quote from: ZubZero on January 15, 2006, 04:28:43 pm

I would like a version that stores the cache in CVS, cause I dont have the db on the same computer as the bot.

CVS isn't a database, so putting the cache there doesn't make any sense. Actually I think the overhead would be steep.

And the database and the update script don't need to be on the same computer, just enter the correct server information in the script. It should work over network too (never tested, but don't see any reason why it shouldn't work).
The variable to set is $dbserver - which I'm defaulting to localhost.

Xenixa · « **Reply #23 on:** January 26, 2006, 09:27:13 pm »

Current Character Count in Whois Cache: 62495

Heh ever since I converted the modified Towers module by Crazied to use the Whois functions of this module that table has been growing like mad fast.

Alreadythere · « **Reply #24 on:** January 26, 2006, 10:08:51 pm »

Hehe, yes, got a similar number in my cache.
Growth slowed down a bit though

On a side note, currently contemplating to add UIDs to the cache. Would allow an easier correlation with a shrunken members table (see that thread), and could help in some cases with reverse ID lookups. As it's extremely rare that there is a change in the ID/name combo. Still possible on renames.

Zakus · « **Reply #25 on:** March 15, 2006, 06:53:53 pm »

I must be missing something obvious, but I'm getting this error when running the update script from command line on winXP (same box that has the mysql database and the bot running on it)

C:\BeBot>php whois_update.php

Fatal error: Call to undefined function mysql_connect() in C:\BeBot\whois_update
.php on line 30

Line 30 is this block:

$link = mysql_connect("localhost", $username, $password)
or die("Could not connect : " . mysql_error());
mysql_select_db($dbname) or die("Could not select database");

What am I missing here?

Alreadythere · « **Reply #26 on:** March 15, 2006, 07:32:26 pm »

Most likely a correct php ini, which has to include the module path for the mysql extension.

Something like

Code: [Select]

extension=php_mysql.dll
extension_dir = "./"

with extension_dir adapted to your environment. (Not quite certain about the correct extension name, the windows version I got has mysql included).

Zakus · « **Reply #27 on:** March 15, 2006, 07:39:31 pm »

Thanks, that fixed it.

Pharexys · « **Reply #28 on:** April 17, 2006, 09:07:17 am »

Hm, i cant get this to work...
Well I've added all at bot class,
d/led files and set em up..
I get a big "nothing"

Code: [Select]

C:\BeBot_v0.2.3>php w_update.php

C:\BeBot_v0.2.3>

more info.. I got Whoiscache.php in /modules, I got w_update.php in bebot directory.. I got bot class updated in bot.php.. cant think of what i wrong, any help appreciated.
Also modified Whois.php to use this already there module.. but Dont want to update

Alreadythere · « **Reply #29 on:** April 17, 2006, 10:23:50 am »

Did you add the needed configuration information in the beginning of the update file?
It doesn't read the conf/Bot.conf nor conf/Mysql.conf.

Only thing I can think of.

Anyways, I have updated my script a bit, it doesn't try to add empty names anymore and gives some more output.
And it's split in a config file and a php script now, for easier updating in the future.

The script, rename to php
The config file, add the infos here

Author Topic: Persistent Whois cache with out-of-bot upgrading (Read 85080 times)

Alreadythere

Re: Persistent Whois cache with out-of-bot upgrading

Xenixa

Re: Persistent Whois cache with out-of-bot upgrading

Alreadythere

Re: Persistent Whois cache with out-of-bot upgrading

Solerna

Re: Persistent Whois cache with out-of-bot upgrading

Xenixa

Re: Persistent Whois cache with out-of-bot upgrading

Alreadythere

Re: Persistent Whois cache with out-of-bot upgrading

ZubZero

Re: Persistent Whois cache with out-of-bot upgrading

Alreadythere

Re: Persistent Whois cache with out-of-bot upgrading

Xenixa

Re: Persistent Whois cache with out-of-bot upgrading

Alreadythere

Re: Persistent Whois cache with out-of-bot upgrading

Zakus

Re: Persistent Whois cache with out-of-bot upgrading

Alreadythere

Re: Persistent Whois cache with out-of-bot upgrading

Zakus

Re: Persistent Whois cache with out-of-bot upgrading

Pharexys

Re: Persistent Whois cache with out-of-bot upgrading

Alreadythere

Re: Persistent Whois cache with out-of-bot upgrading