collapse collapse
* User Info
 
 
Welcome, Guest. Please login or register.
* Search

* Board Stats
  • stats Total Members: 989
  • stats Total Posts: 18363
  • stats Total Topics: 2500
  • stats Total Categories: 7
  • stats Total Boards: 35
  • stats Most Online: 1144

Author Topic: New Items database?  (Read 3908 times)

0 Members and 1 Guest are viewing this topic.

Offline Shelnaria

  • BeBot Rookie
  • *
  • Posts: 17
  • Karma: +0/-0
New Items database?
« on: December 03, 2005, 05:20:56 pm »
Does anyone know if the program to parse the database xml output was ever released? If so could you point me in the right directio to obtain it?

Offline Zarkingu

  • Contributor
  • *******
  • Posts: 34
  • Karma: +0/-0
New Items database?
« Reply #1 on: December 03, 2005, 05:36:33 pm »
Zarkingu 210 MP RK2

Offline Xenixa

  • Contributor
  • *******
  • Posts: 307
  • Karma: +0/-0
New Items database?
« Reply #2 on: December 03, 2005, 08:10:40 pm »
Or you could use Auno's XML output of the DB.
http://auno.org/dev/aoxml.html

Been playing with that one myself, I've written a php script that pulls just the AOID, Name, and QL from the XML and inserts those into a mysql table. Still working on a custom items.php that searches correctly using hints from vhab's old IGN items search code.

Should have it finished soonish I hope. Will post it in Custom Mods forum.
<<< Hack's in Zend Studio

All my Custom Bebot files may be Found Here <-clicky

Offline Shelnaria

  • BeBot Rookie
  • *
  • Posts: 17
  • Karma: +0/-0
New Dbase!
« Reply #3 on: December 04, 2005, 07:59:54 am »
Latest Update here: 16.1.2

Important notes:
This is an experimental update. BACKUP your database before usage.

Thanks to Angels for parsing!

Offline skeezix

  • BeBot User
  • **
  • Posts: 21
  • Karma: +0/-0
New Items database?
« Reply #4 on: December 04, 2005, 02:50:22 pm »
only thing I see offhand is that the symbiants are ones from an earlier version of the DB that don't really exist, with the unique tags and such.
I think patch 15.0.5 or so changed at least the base ability reqs, then later removed the unique tags from them

Otherwise it seems to be a nice DB :)

edit - nevermind it's showing from both patches, both the previous ones with unique and higher reqs, and the updated ones more recent which works for me ;)    thanks a ton for the work :)

next edit - good lord I think I need sleep.  it seems to show only the current symbiants dropping, and I think I must have not dropped the previous table before importing this SQL info. looks very nice

the only thing is I can't search for Kyr'Ozch alien items, I have to leave out the ' and use Kyr Ozch - not a biggie but takes a little getting used to :)

Offline skeezix

  • BeBot User
  • **
  • Posts: 21
  • Karma: +0/-0
New Items database?
« Reply #5 on: December 06, 2005, 05:27:58 am »
ok well uhm err....

was this one meant to be used in place of the 15.9.2 DB, or in addition to?

There are a lot of things missing from the 16.1.2 posted, such as a lot of nanos or inability to request certain alien weapon QLs (205 kyr sword type 112 or so)

So... I combined the 2 of them, and removed the old symbiant links from the 15.9.2 database.  and like before, use this at your own risk ;)

new 16.1.2

Offline Shelnaria

  • BeBot Rookie
  • *
  • Posts: 17
  • Karma: +0/-0
New Items database?
« Reply #6 on: December 06, 2005, 07:11:03 am »
Was meant to replace. But am seeing some errors. Trying to track them down :) I like yours alot as well!

Offline skeezix

  • BeBot User
  • **
  • Posts: 21
  • Karma: +0/-0
New Items database?
« Reply #7 on: December 06, 2005, 06:21:28 pm »
I also separated a lot of it into multiple insert statements, as mysql was giving me some large packet errors when adding too many things at once.

for the most part I haven't seen anything really missing now, and seems to run great.  I think totals around 50k entries for the items DB (not at the PC) and seems fairly comprehensive now with what I think most people look for.

What things were giving errors?

Offline Shelnaria

  • BeBot Rookie
  • *
  • Posts: 17
  • Karma: +0/-0
New Items database?
« Reply #8 on: December 06, 2005, 08:15:02 pm »
Mainly mising tradeskill items. Like the Mass Relocating Robot (Shape Hard Armor) Not sure if yours corrected it or not.

Offline Blondengy

  • BeBot Creator and Founder
  • BeBot Hero
  • ******
  • Posts: 51
  • Karma: +0/-0
New Items database?
« Reply #9 on: December 06, 2005, 11:43:24 pm »
This is the original script used to make the database. I never intended to release this but this might help you in your development. (Please note: its NOT inteded to be used by someone who doesn't really know what they're doing. ;))

It uses Auno's XML file as an input.

The main reason I never inteded to release it is that its probably one of my worst pieces of work and was thrown together quickly in a few minutes. It lets MySQL do most of the work which is definatly not a good thing. Because MySQL does most of the work you should definatly run this script on the same machine as the MySQL server, you should also not run this on an oldish machine as it would take to long and you should not run it on a machine that you need for the next 30 mins or so. This script will cause your MySQL server to swallow 99% of your CPU load for 30mins or so (depending on your CPU). Its crappy, it sucks but it does the job. I think you now get the point - I hate myself for not recoding it! ;)

I'm releasing this so that you can have a look at it in the hopes that it helps you improve your script for parsing the XML. Once its run you might wanna add a few cases so that it groups the NCU Memories, the ELLTS and other special items properly (those two are the only ones I can think of at the moment that it doesn't group due to the low and high name beeing different and not following any pattern).

Here we go at your own risk: (Added a few extra comments to make it more readable)
Code: [Select]
<?
  $time_now = time();
 
  require_once("MySQL.php");

  $db = new MySQL();

  $db->query("CREATE TABLE items
        (lowid INT NOT NULL,
        highid INT NOT NULL,
        lowql INT,
        highql INT,
        name VARCHAR(150))");

  $db->query("CREATE TABLE items_tmp
        (id INT NOT NULL,
        ql INT,
        name VARCHAR(150))");
       
  $handle = fopen("150804.xml", "r");

  // start reading XML line for line
  while($line = fgets($handle, 1000))
  {
    // Check if this is the start-tag of an item we want to extract
    if (preg_match("/item aoid=\"([0-9]+)\" patch=\"([0-9]+)\" metatype=\"(w|i)\">/", $line, $id))
    {
      // Read the wanted item line for line and extract needed info
      while($line = fgets($handle, 1000))
      {
        // Looking for the name first
        if (preg_match("/<name>(.+)<\/name>/", $line, $name))
        {
          // Ignore implants and clusters - who needs them in the items DB?!
          if (preg_match("/ Implant: /", $name[1]) || preg_match("/ Cluster - /", $name[1]))
            break;
          // Now that we have a name and its not an implant/cluster we contine searching line by line
          while($line = fgets($handle, 1000))
          {
            // We got the name so we need its QL
            if (preg_match("/<ql>([0-9]+)<\/ql>/", $line, $ql))
            {
              $id = $id[1];
              // Remove some unneeded prefixes so that it groups properly.
              $name = str_replace("'", "", $name[1]);
              $name = preg_replace("/^High-Quality /i", "", $name);
              $name = preg_replace("/^Augmented /i", "", $name);
              $name = preg_replace("/^Basic /i", "", $name);
              $name = preg_replace("/^Worn /i", "", $name);
              $name = preg_replace("/^Inferior /i", "", $name);
              $name = preg_replace("/^Flawless /i", "", $name);
              $name = preg_replace("/^Low Light /i", "Extreme Low Light ", $name);
              $ql = $ql[1];

              // Slows it down a bit, but makes sure there're no duplicates
              $result = $db -> select("SELECT id FROM items_tmp WHERE name = '$name' AND ql = $ql");
              // Insert item into temp table if its not already there.
              if (empty($result))
                $db -> query ("INSERT INTO items_tmp (id, name, ql) VALUES ($id, '$name', $ql)");
              break;
            }
          }
          // We got all info we need on this item so lets continue search for the next one.
          break;
        }
      }
    }
  }
  fclose($handle);
 
  echo "Done reading XML file, now grouping items...\n";

  $result = 1;
  while (!empty($result))
  {
    // Let MySQL do the grouping - I'm lazzy today and want to get this done. (doing this 300 groups at a time cause I feel like it)
    $result = $db -> select("SELECT name FROM items_tmp GROUP BY name ORDER BY id ASC LIMIT 0, 300");
    if (!empty($result))
    foreach ($result as $run)
    {
      $name = $run[0];
      // Select all belonging to this group, arrange highid/lowid stuff and insert them into items table
      $res = $db -> select("SELECT id, ql FROM items_tmp WHERE name = '$name' ORDER BY ql ASC");
      for ($i = 0; $i < count($res); $i += 2)
      {
        if ($i < (count($res) - 1))
        {
          $lowid = $res[$i][0];
          $highid = $res[($i + 1)][0];
          $lowql = $res[$i][1];
          $highql = $res[($i + 1)][1];
        }
        else
        {
          $lowid = $res[$i][0];
          $highid = $res[$i][0];
          $lowql = $res[$i][1];
          $highql = $res[$i][1];
        }
        $db -> query("INSERT INTO items (lowid, highid, lowql, highql, name) VALUES ($lowid, $highid, $lowql, $highql, '$name')");
      }
      // Delete the group from temp table since we're done with it
      $db -> query("DELETE FROM items_tmp WHERE name = '$name'");
    }
  }
 
  echo "Operation took " . (((int)((time() - $time_now) / 6)) / 10) . " minutes.\n";
?>

As you might see it uses the MySQL.php, but a different version, one which loads the mysql.conf itself... (as I said - only to be used by people who know what they're doing).
You need to set it up on a database with an empty or none-existant "items" table. Once it filled up the items table I used a dump of that as the items.sql file.
Hope it helps you, good luck and sorry for not beeing any more help! :(

- Blondengy

Offline skeezix

  • BeBot User
  • **
  • Posts: 21
  • Karma: +0/-0
New Items database?
« Reply #10 on: December 07, 2005, 05:17:37 am »
the version I posted has the tradeskill stuff since it's the previous 15.9.2 SQL file added with the new 16.1.2 that was posted above (except the outdated symbiant links, those are the only thing removed)

Offline kuznechik

  • Contributor
  • *******
  • Posts: 60
  • Karma: +0/-0
Re: New Items database?
« Reply #11 on: December 30, 2005, 11:13:47 am »
For some reason this dump missing nano "ranged energy mastery", have id/algorithm/pppe, but not nano. Like 15.9
Kuznechik, proud bot admin of Disciples of Omni-Tek, Rimor.

Offline Akarah

  • Contributor
  • *******
  • Posts: 72
  • Karma: +0/-0
    • http://synergyfactor.net/
Re: New Items database?
« Reply #12 on: December 30, 2005, 07:44:48 pm »
For some reason this dump missing nano "ranged energy mastery", have id/algorithm/pppe, but not nano. Like 15.9

its LR Energy Weapon Mastery. (or "Ranged Energy Weapon Mastery")
« Last Edit: December 30, 2005, 08:12:39 pm by Akarah »

Offline kuznechik

  • Contributor
  • *******
  • Posts: 60
  • Karma: +0/-0
Re: New Items database?
« Reply #13 on: January 04, 2006, 06:17:25 pm »
Examined this base more closely.
Very buggy.

Lot's of double items.
Kyr weaps with and without \'
Some weapons and armors doubled.
Weapons have wrong ranges. (search for MTI B-94, got full blob, including ql152-11)
Kuznechik, proud bot admin of Disciples of Omni-Tek, Rimor.

 

* Recent Posts
[AoC] special char for items module by bitnykk
[February 09, 2024, 09:41:18 pm]


0.8.x updates for AoC by bitnykk
[January 30, 2024, 11:16:08 pm]


0.8.x updates for AO by bitnykk
[January 30, 2024, 11:15:37 pm]


BeBot still alive & kicking ! by bitnykk
[December 17, 2023, 12:58:44 am]


Bebot and Rasberry by bitnykk
[November 29, 2023, 11:04:14 pm]

* Who's Online
  • Dot Guests: 476
  • Dot Hidden: 0
  • Dot Users: 0

There aren't any users online.
* Forum Staff
bitnykk admin bitnykk
Administrator
Khalem admin Khalem
Administrator
WeZoN gmod WeZoN
Global Moderator
SimplePortal 2.3.7 © 2008-2024, SimplePortal