Development on the all new WowStack core has been progressing nicely the past days, so here is another update.
Looking back at the history of MaNGOS and our development process, the one thing that annoyed me most was our team rushing it and missing out on doing things in a future proof way. This lead to us still having working core and tools but it also neglected documentation and made it hard for new developers from our great community to join in.
As mentioned in the previous update, we set our eyes on rebuilding the core and all tools in a way that they can be used as atomic units. As such, e.g. all the code handling files from Blizzard’s MPQ archives and the files inside previously being spread over various places from map extraction/conversion tools to a VMAP library and the movement map generator was a huge issue.
I previously mentioned creating tempest and meshard to do all that, and using StormLib to extract data. While that initially worked out nicely, we decided that improvements were required after doing a few tests and refactored.
The result is a unified tempest library, with just one dependency (ZLib). Much nicer than having tons of them.
What can libtempest do?
The basic feature of our new tempest library is that it handles reading of MPQ archives on its' own. We now can read MPQ archives off version 0 and 1 with no help from StormLib. This means, our tools can access data from vanilla WoW, The Burning Crusade, and Wrath of the Lich King.
As of today, this also includes support for all compression formats and encrypted/signed MPQs.
Bonus feature: we can also read embedded MPQs, meaning executables containing MPQs can also be a source for grabbing data. Why is that needed? Some of old update files for vanilla WoW were distributed as packaged EXE files with the MPQ embedded.
Apart from MPQ handling, it encapsulates access to all file types used in the World of Warcraft game client, even textures and shaders.
The Game Client Databases
As some may have heard, the WoW game client includes a set of database files to boostrap the client and remove the need to send basic data back and forth every time client and server communicate.
MaNGOS had support for DBC extraction for a while but it was not complete.
Upgrades we made while rewriting the code:
- file format defintions for the DBC structure can now be read from XML files, there will no longer be the need to e.g. have format definitions like
const char AreaTableEntryfmt = "niiiixxxxxissssssssxixxxi";
Now you can just write XML without modifying the source code like this
<?xml version="1.0"?> <dbc> <name>AreaTable</name> <field> <type>uint32</type> <name>id</name> <key> <type>primary</type> </key> </field> ... </dbc>
- Foreign keys can now be used! This enables validation of referential integrity between DBCs. Relations between DBC files usually were handled manually within the core which is now a thing of the past. An example would be area tables which commonly link to maps (think Azeroth/Kalimdor/random dungeon). We now can just link these up, and when using DBC data get related maps, adjacent areas with a single call. No extra code needed.
The World is no longer an unknown place!
Map definitions and parsing was always a rather unhappy place. In general, MaNGOS and all other WoW emulation projects go ahead and read map tiles from WDT files, and extract terrain and object information from ADT files.
Implementations of that process usually have been a random process with a lot of guess work, and for people running servers it meant frequent regeneration of map data.
Not any more: we basically went ahead and used the information available freely on the WoWDev wiki and implemented code that exactly maps to the game data, and handles all information they contain.
What’s cool about this is that the fact that game data is structured similarly to something you all may know. Video data on YouTube e.g. when played back using VP8 is using the same kind of chunked data format.
One of the things we can now do: extract and generate minimap for all of Vanilla WoW and generate a viewable PNG from that.
Future applications for this might be something like creating a renderer to display maps using OpenGL.
Questions and Answers!
If you want to talk about this or let me know what you want to see in a fresh start the most, ping me on twitter or feel free to ping WowStack.