Coming back to our discussion in the bebot 0.3.0 thread

Just making a "mirror" of all the data isnt much good, youd use more resources than you save. It only makes sense if your making a character database like Auno.org, but not otherwise.

I went ahead and implemented my whois-cache. Here what I did and my current experiences with it.

I added a new module handling all whois queries, and modified all other modules doing get_site queries for character information to use the whois module. Then I added a script outside of the bots as cron job, updating the whois-cache once each day. First getting the org roster for all orgs in the cache, then getting the current information for all not yet updated characters. With now 25k entries the update took 9mins.
For the org member lists I did something similar - I moved the update of the rosters in a bot doing nothing but handling the members lists (updating them twice a day like now), and notifying the org bots I host if an update happened, so they can update their buddy list. Can't use a script here, as the members table needs the user ids, which are only available in-game.

Those two things together work extremely well, reducing any latency I had due to whois queries to almost zero (if you are in the cache once, you stay in it forever, so there is only once any lag for an http query per character). And it reduces the time the org bots lag due to the roster updates too.

I know that I'm not running any usual configuration, hosting 4 org bots and 3 raidbots, with at least one more raidbot to follow. And I'm running them all on an almost dedicated server (it's my file server too, but that load is quite low). But I have to say for me the work it took to implement this and the additional ressources needed (the bot handling the roster updates, and the cron job for the cache) are worth the won speed.

Sure, I'll try to go around and post the relevant files later today.