collapse collapse
* User Info
 
 
Welcome, Guest. Please login or register.
* Search

* Board Stats
  • stats Total Members: 989
  • stats Total Posts: 18363
  • stats Total Topics: 2500
  • stats Total Categories: 7
  • stats Total Boards: 35
  • stats Most Online: 1144

Author Topic: Whois tables  (Read 2344 times)

0 Members and 1 Guest are viewing this topic.

Offline screwy

  • BeBot Rookie
  • *
  • Posts: 19
  • Karma: +0/-0
Whois tables
« on: July 05, 2009, 01:35:04 am »
Hey all!

A suggestion about the whois tables. The AO page as well as auno is pretty fast, which negates the need for caching really. Plus the in memory caching should suffice. So a general whois-cache on/off switch would be great.

If you keep your bot running for even a month you end up with 55k entries in that. And if you run the cron job externaly it runs for 3 hours and 55k queries to the webserver are made. This is not efficient. Especially if you consider that no entries get deleted. This is the second suggestion: add a "last queried" column to the database. If the last queried entry is too far in the past (should be configurable) the cron job will delete the whois entry. This will keep the whois table clean.

As a workaround I am truncating the whois cache on a bi-daily basis and run the cron-job every day.

Cheers!
-Screwy.

Offline Khalem

  • BeBot Founder
  • Administrator
  • ********
  • Posts: 1169
  • Karma: +0/-0
    • http://www.ancarim.com
Re: Whois tables
« Reply #1 on: July 05, 2009, 01:47:27 am »
This is a known issue. While it does not really cause a major concern given that MySQL is perfectly capable of handling the amount of rows, it does cause some other issues elsewhere.

As for not needing the cache is open for debate. It has nothing to do with speed of FC or Auno, but rather about not hitting them up uneccecarily.

On a general basis i can say that this is one of the things i would like to greatly improve on for BeBot 0.7/0.8 allowing us to discard entries that have not been accessed, and thus not updated for x hours/days, as well as making the system smarter so that roster updates and whois updates are carried out as early as possible after FC updates information, while at the same time making sure we dont have thousands of bots hitting up the servers at the same time.
BeBot Founder and Fixer Kingpin

Offline screwy

  • BeBot Rookie
  • *
  • Posts: 19
  • Karma: +0/-0
Re: Whois tables
« Reply #2 on: July 05, 2009, 01:58:56 am »
I partly agree.

But running the updates on 55k+ whois entries daily on which not even 1% is used does not easen the load on Funcom servers. So the memory cache should really suffice.

I can in all honesty say that even if every query is pulled from funcom it would reduce their load GREATLY considering the regular updates.

Plus I want to see the person who can distinct between a local hit and a pull from funcom servers ;)

Glad to see this is in the works.


-Screwy.

Offline Alreadythere

  • BeBot Maintainer
  • BeBot Hero
  • ******
  • Posts: 1288
  • Karma: +0/-0
Re: Whois tables
« Reply #3 on: July 05, 2009, 10:41:48 am »
The script shouldn't do 55k queries when running, most chars should be covered by org membership.

The script does read org rosters first, and updates all chars in there. Before optionally going over everyone else left over.

Personally a "last queried" column would be a contradiction of the idea behind the cache. I added it after funcom had problems with their character information, which meant all bots relying on that information had insane lag while unsuccessfully trying to look up the information.

If you don't run the update script the cache shouldn't grow that fast as it will only lookup characters actually queried by the bot. And it will only update their entries if the are queried again after 24h (I think that's the default, it's actually controlled by a setting).

Besides, what's the problem with 55k entries in the cache? It can't really be a space problem, the 4-5mb shouldn't even register on any current harddisc.

Offline Glarawyn

  • BeBot Hero
  • ******
  • Posts: 521
  • Karma: +0/-0
Re: Whois tables
« Reply #4 on: July 06, 2009, 05:33:25 am »
The whois cache it's one of those things you won't miss...until you need it.

Running a raid or org bot and having everything stop working because FC and/or Auno go down isn't fun.

How we go about caching things is another matter.

 

* Recent Posts
[AoC] special char for items module by bitnykk
[February 09, 2024, 09:41:18 pm]


0.8.x updates for AoC by bitnykk
[January 30, 2024, 11:16:08 pm]


0.8.x updates for AO by bitnykk
[January 30, 2024, 11:15:37 pm]


BeBot still alive & kicking ! by bitnykk
[December 17, 2023, 12:58:44 am]


Bebot and Rasberry by bitnykk
[November 29, 2023, 11:04:14 pm]

* Who's Online
  • Dot Guests: 671
  • Dot Hidden: 0
  • Dot Users: 0

There aren't any users online.
* Forum Staff
bitnykk admin bitnykk
Administrator
Khalem admin Khalem
Administrator
WeZoN gmod WeZoN
Global Moderator
SimplePortal 2.3.7 © 2008-2024, SimplePortal