Python : World Of Warcraft Profile Downloader

When it comes to my lifestyle (and many other things) I definitely prefer the minimalist approach. I like consolidation. I delight in eliminating unnecessary things we no longer need. I think of it as refactoring life... And I truly do love refactoring. If my wife and I did not have our two (soon to be three) wonderful children, I imagine we would be traveling the world with just a backpack...

But there is one area where I am definitely not a minimalist... data. I am a data hoarder. I collect data from anywhere and everywhere. I download or digitize every document or set of paperwork I come into contact with in my professional, personal, and family life. All my hobbies have one or more dedicated folders where related digital material is carefully filed. I find ways to rip or exfiltrate and backup all my media (from ebooks and music to our DVD collection). It is probably a medically certifiable condition at this point...

But hey... Bytes are tiny and they pack well,,, And it could be worse. Google "hoarders" and you will see what I mean...

I play World Of Warcraft casually and my data hoarding tendency discovered that WoW maintains a web service from which you can download a JSON snapshot of your character, along with their associated achievements, pets, mounts, and so on... It was not long before I had thrown together some crude shell scripts using some macros to wget down all the JSON data my little digital hoarding heart could want... It was slow, and tedious (I had to generate a separate API call for each character achievement which required writing a macro to mass produce the wget calls needed) but it got the data I wanted.

However, also In my spare time, I have been trying to learn Python... Mostly by tackling small projects that have some Python angle to them... I recently realized that the WoW API was a perfect opportunity to use Python to write a more robust profile downloader! It would require file I/O, HTTP calls, and JSON parsing, all of which would help improve my Python skills, and it would make updating my downloaded character profiles much easier. (Need. More. Dataz!)

Since file I/O, HTTP, and JSON are all core skills, I thought it would be useful to put this project up on a blog for anyone else who is starting to learn Python to reference. (And also have the WoW API script available in its entirety to feed the addiction of fellow data hoarders!)

So, let's begin!

NOTE: To give you some context, there is an API reference for the WoW API. In general, it is very simple REST interface and is very easy to follow.

Setup

I am running this on a Raspberry Pi... (Why? Because Raspberry Pi's are awesome. And because Python works really well on Linux, but my laptop is Windows. Comments about the superiority of the Mac can be emailed here.)

The script I wrote uses the Python library httplib2 to make HTTP requests. This is installed using Python's package manager pip which does not come natively on the Raspberry Pi. To install pip you must:

sudo apt-get install python-pip  

Once pip is installed and work, you can install httplib2:

sudo pip install httplib2  

The WoW API has several different objects it can return but there is one core object for the character that is most of what I am interested in. That interface takes several parameters that govern what information about the character is returned. (More information here.) Character achievements (awarded for performing specific tasks in game) are handled through their own API however, so the character object returns a list of achievement id values that must be queried in the API for additional information.

At a high level, the script I wrote will query for the character, parse and output that response, and then loop through all the achievement id values and query for each achievement, parsing and outputting each of those responses.

Let's take it from the top!

Code

NOTE: The complete script can be downloaded from GitHub.

We will be using httplib2 and the native json module, so first we import these:

import httplib2  
import json  

I wanted to make this script flexible, so I setup a parameter for the character name and server so it is not hard coded to just my character... (It would have been better to prompt for each value from the console... That exercise is left to the readers.):

# Parameters:
#
character_name = "Marlhammer"  
server = "Khadgar"  

Now we need to initiate an HTTP GET request to get the character json. We build a URL:

# Script:
#
base_url="http://us.battle.net/api/wow/character/"  
base_url+=server  
base_url+="/"  
base_url+=character_name  
base_url+="?fields=achievements,appearance,mounts,pets,professions,progression,pvp,quests,reputation,stats,talents,titles"  

Then use httplib2 to make the GET request:

resp, content = httplib2.Http().request(base_url)  

I want to output the JSON to a file and Python provides a simple interface for File I/O (as opposed to say, Java):

output = open(character_name + '.json', 'w')  

Using the output file handle, I then parse the character JSON and write a copy to the file:

character_json = json.loads(content)  
json.dump(character_json, output, sort_keys=True, indent=4, separators=(',', ': '))  
output.write("\n")  

NOTE: The loads() method means "load string". There is also a load() method for reading JSON from a file.

NOTE: The parameters to dump() are used for "pretty printing"... The json attributes are output as sorted, and indented to make them human readable.

As mentioned above, the character achievements are not included in the character JSON in their entirety, and are instead, referenced by an array of ids. If you examine the character JSON you will see there is an achievements map that includes information about achievement progress, including an array called achievementsCompleted.

The next step is to loop over those ids and make a query for each id:

for achievement in character_json["achievements"]["achievementsCompleted"]:  
    resp, content = httplib2.Http().request("http://us.battle.net/api/wow/achievement/"+str(achievement))
    achievement_json = json.loads(content)
    json.dump(achievement_json, output, sort_keys=True, indent=4, separators=(',', ': '))
    output.write("\n")
    print str(achievement)

This portion of the script follows the same pattern as before. A request is made, the result is parsed into JSON, and then it is written to the file. The achievement id is printed to the console as a status indicator.

And then the script ends:

print "\nCompleted!\n"  

That is all there is to it!

Questions? Comments? Email me at: [email protected]