It seems like someone at FC is blocking IPs of botusers, most likely to the possible high amount of http queries the whois queries can produce each day.

Same for me, I can't access www.anarchy-online.com at all, regardless of the way I try to connect - firefox, wget, lynx, telnet, whatever. Always ends in a connection refused right away, even though those clients aren't blocked by any firewall on my side.

Hm, does php have what is called cpickle in python?

Cause crawling the db and save the data needed in gzipped cpickle would be easier on bandwith.

Can you help me make the fsockopen use proxy then? My php fu is weak :/
I've tried with the example from http://www.ziguras.com/php/using-fsockopen-to-connect-to-remote-servers but can't get it to work properly. Any help would be appriciated :-)

I tested some anonymous proxies from http://www.proxy4free.com/page1.html and wgetting www.anarchy-online.com worked with : export http_proxy=http://url:port

cPickle is just to store/read objects. Objects can be data structures, functions etc. Quick and easy way to do it in python. I just thougth that getting a dump of the data from the xml we need for whois could save bandwith if we spread it with mirrors.

Just tested the proxy example on the page you posted, and it worked with my private squid proxy.

I tried it with my proxy (just the code snippet in the linked page), but timed out with some of the random proxies of your list that I tested, never got any result. Either those expect some logon or are seriously lagged.

If we do some central updating solution a text listing or even just a diff with changed entries to the day before would be best I think, and then just zip or gzip it to reduce total bandwidth usage.

My cache was up to ~120k players at one point, and of those 25k was orgless. That made the update script take a long time to finish (and to query the xml files ~26k times during that time). One way to limit the amount would be to clear the cache of orgless people (since they take up the most of the xml queries in a large cache) now and then.

Another thing to do when you have many bots running on the same machine would be to do $noupdate=true on every lookup, so it'll never fetch a new when the timestamp is old, just on people that doesn't exist. That way if you have a large cache there wont be any problem with a slow update script (not sure how long time mine took towards the end, but I think it was around 3-4h at ~80k players) and the bots fetching a fresh lookup every time someone does a whois because the cache isn't updated (could be solved by setting a much higher expire time).

I was never banned from FC, and it was just two weeks ago or so when I cleared my cache. So either they changed something since then, or your bots do an unrealistic amount of connections towards funcom and fixing that sounds like a better idea than to try and go around it.

A central whois database exists already.. it's called www.anarchy-online.com... Any off-box/lan database will be slower which is the purpose of the local cache.

My cache was up to ~120k players at one point, and of those 25k was orgless. That made the update script take a long time to finish (and to query the xml files ~26k times during that time). One way to limit the amount would be to clear the cache of orgless people (since they take up the most of the xml queries in a large cache) now and then.

I've got 42k unorged people in my cache, which created that many queries - guess FC didn't like it anymore. They blocked my pretty sudden as of friday last week.

One other way to lower the load is doing updates just for orged people, no need to delete the unorged ones, the most current version of the script simply doesn't try to update them anymore on default.
For me that will result in like 1500 queries, with a higher delay those won't create that much load on the FC site anymore.