Our weapons of choice will be Python and the debugger in the Chrome web browser. To access the Chrome Debugger, hit F12 or Ctrl+Shift+J. This tool gives you a wealth of information about the interactions between your browser and the sites you visit, but for this task we’re mainly concerned with what’s going on in the Network tab. If you’re more of a Firefox person, you can use an extension called Firebug. Incidentally, the lead developer of Firebug was hired by Google to develop the Chrome Debugger. For this post I’m going to assume you’re using Chrome.

Since our goal is to programmatically perform actions on the site, you need to make an OKC account (I recommend using a temporary email service like this one to avoid spam). The next step is to figure out how to log in using Python. So, turn on the Chrome Debugger, go to the Network tab, log out of OKC, and log in again. You’ll see a bunch of stuff appear in the debugger, but what we want is all the way at the top, so scroll up there and click the the link named “login”. That will bring up a bunch of helpful information about the request we made to OKC’s servers when we clicked “Sign in”.

Mmm, data

At the top, we can see the Request URL: https://www.okcupid.com/login. Also useful is the Request Method, POST. Under the Request Headers section, there’s a field called “Cookie” which has a bunch of random-looking stuff in it. This tells us that OKC sets a cookie on your browser, so we’ll need to make sure our code uses it. Finally, there’s another section called “Form Data” containing all of the information that our computer sent to the server when we signed in. Specifically, we can see our login credentials, “username” and “password”, along with something called “dest” that has the value “/home”. Looking at our browser’s URL bar, we can deduce that the “dest” parameter denotes the path to the resource in the okcupid domain to which our browser will be redirected after the login request is completed.

So how do we log in using Python? It’s easy using the Requests library:

import requests

sesh = requests.Session()

credentials = {'username': 'okc_account_name', 'password': 'account_pword', 'dest': '/home'}
login_resp = sesh.post('https://www.okcupid.com/login', data=credentials)

print login_resp.content

We start by instantiating a Session object which will store the cookies necessary for making future requests to OKC that require authentication. Then, we create a dictionary containing the POST data needed for the login request (fill in your own username and password). Last, we make the POST request from within the Session, passing in the login URL and the credentials. If you store the output of print login_resp.content in an HTML file and open it in your browser, you’ll see the OKC welcome page for your account. Success!

Now let’s try something a bit trickier: sending a message. Go back to OKCupid with the debugger open and click the Messages tab on the left toolbar. In the Network section of the debugger there will be a request called “messages”. If you look at the Headers of that request, you’ll see that the Request URL is http://www.okcupid.com/messages and the Request Method is GET. Assuming you’ve run the login code aove, we can replicate this in Python as follows:

get_messages = sesh.get('http://www.okcupid.com/messages')

If you haven’t deleted it, there should be a message from the user “chriscoyne” in your Received Messages. Since that account apparently doesn’t respond to messages, it’s a good test subject for cracking the OKC API. If you click on his message, you’ll get another request in your debugger that looks like this:

Hi Chris!

The text following the question mark after “messages” is called a query string, and it allows you to send additional information to an HTTP server. We can set these values in Python in a way similar to our POST request above:

thread_data = {'readmsg': 'true', 'threadid': 7054982370615456828, 'folder': 1}
get_thread = sesh.get('http://www.okcupid.com/messages', params=thread_data)

If you want to avoid manually determining the thread IDs for OKCupid messages, there’s a great HTML parsing library called Beautiful Soup. I’ll talk more about scraping web pages in a future post.

Now, send a message to chriscoyne and check out what the debugger gives you.

I swear

So, we have to make another POST request. But you know what to do!

msg_data = {'ajax': 1, 'sendmsg': 1, 'r1': 'chriscoyne', 'body': 'messager? but i hardly know her!',
'threadid': 7054982370615456828,
'authcode': '1,0,1362399999,0xdb8d1bb0ffffffff;aa5242512122943168ddffffffffffffffffffff', 'reply': 1}
send_msg = sesh.post('http://www.okcupid.com/mailbox', data=msg_data)

If you go back to the message thread in your browser, you’ll see the message you just sent from your script. To avoid having to send a test message to figure out what the value for authcode should be, you can right click on the message box in Chrome and select “Inspect element” from the drop-down menu. You should see the authcode a few HTML elements above the element for the message box:

Not so hidden eh?

OKC seems to change the authcode at certain intervals as a security measure, so in a production system you’d need to dynamically grab the value before you send a message (which you can do with Beautiful Soup).

So there you have it! Hopefully this post has helped you get started reverse engineering other website APIs. If you need help, don’t hesitate to leave a comment.

Recently, I came across Processing.js and decided to give it another shot. By porting it to JavaScript, the developers have made it easier to get started and publish to the web (though I would guess that the JavaScript version runs slower than native Processing). All I needed was a project.

Naturally, I chose to implement Conway’s Game of Life. First publicized in 1970, it has fascinated many over the years due to the complexity of patterns that it produces based on a few simple rules. YouTube is full of videos depicting this cellular automaton; one of my favorites is this one taken of an installation on the Boston Greenway that I got to see in person. For Inception-level brain melting, there’s Game of Life in Game of Life. My favorite among the common patterns that arise is the glider since it always seems like it’s on a mission.

So here it is! You can reset the game by clicking anywhere on it, or you can wait five minutes and it will reset itself. I experimented with some crazier color shifting patterns as I was developing it, but I settled on a subtly undulating rainbow that I think would make for a good screen saver. Besides increasing performance to allow for a grid with more cells, the code could also be improved by adding a method for determining when the game has reached a steady state. One way would be to cache the previous few generations and compare the current generation against them, but computationally that could have a terrible worst-case performance. I’m open to suggestions! The source code is available here.

We’ll again start with a square with an inscribed quarter circle:

Recall from last time that π = 4A_c/A_sq, where A_c is the area of the quarter circle and A_sq is the area of the square. Rather than randomly place an arbitrary number of points on the figure and then determine the ratio of points inside the circular sector to total points, we can calculate the exact discretized area of the sector (thus approximating the true area of the sector) and divide by the exact area of the square.

To do so, let r be an arbitrarily large integer. A priori, we know that A_sq = r². We could then iterate over all r² points and determine the total number of points that fulfill the inequality x² + y² ≤ r². But is there a better way? You bet!

If we zoom in on the sector, we can see that the curve of the circle is not perfectly smooth; instead, it moves in discrete steps going down and to the right.

So, we can describe the area of the sector as a collection of r rectangles, each with width 1, whose heights are given by y = √(r² – x²), where { x | x ∈ Z⁺, 0 ≤ x ≤ r }. By summing all values of y generated by this domain, we are essentially using the rectangle method of numerical integration to calculate the sector’s area. Best of all, this technique is linear in r, making it much faster than examining every single point for membership in the circle.

As far as coding goes, we could write the algorithm iteratively . . .

def outside_circle(x, y, r):
	return x**2 + y**2 > r**2

def circle_area(r): # numerically integrate using the rectangle rule with delta = 1
	x, y = 0, r # start at top of circle
	circle_area = 0
	# if x == r, y (the height) will be 0, so this point will contribute nothing to the circle area
	while x < r: # go 90 degrees around the circle clockwise
		while outside_circle(x, y, r):
			y -= 1
		circle_area += y # add the current slice to the area
		x += 1
	return circle_area

. . . but that wouldn’t be very Pythonic. Instead, we can use a list comprehension combined with the sum() function:

def calculate_height(x, r):
	return math.floor(math.sqrt(r**2 - x**2))

def circle_area(r):
	return sum(calculate_height(x, r) for x in xrange(r))

How concise! Note the use of xrange() rather than range(). Because xrange() yields values of the sequence as needed rather than constructing the entire list up front, the program requires less memory to execute. For example, with r = 3,500,000, range() caused Python to use 137,812 KB of memory. Substituting xrange() lowered memory consumption to 26,280 KB, a decrease of nearly 81%! (I’m using Python 2.7.3 here. In Python 3, range() behaves like an iterator, so you should get similar memory performance to xrange().)

The list comprehension expression of the algorithm also executes much faster than the iterative version. With r = 10,000,000, the iterative program took 83 seconds to complete, while the list comprehension script finished in only 31 seconds, a 63% improvement. This experiment demonstrates the difference between imperative (procedural) and declarative (functional) programming styles. Python provides good support for either paradigm, but declarative expressions tend to be more idiomatic. With modern programming languages, it’s generally better to describe the result that you want (using, e.g., a list comprehension) rather than prescribing the exact steps to compute it since the language can usually optimize the computation better than a human.

Finally, let’s compare this method to our Monte Carlo simulation. With N = 50,000,000, the Monte Carlo script gave π = 3.14153067417 in 65.626 seconds using 4,636 KB of memory. To normalize on running time, we can use r = 21,366,107 for the numerical integration program, yielding π = 3.14159265357 in 65.525 seconds consuming 26,264 KB. Though the Monte Carlo method needed 82% less RAM, it produced an experimental error of 6.19794e-5 (1.972865e-3%) while requiring almost 2.5 times as many operations as numerical integration. Integration was six orders of magnitude more accurate (experimental error of 1.00000e-11 or 3.183099e-12%).

Additionally, the Monte Carlo simulation will produce a different approximation for π every time depending on what points are randomly generated. Numerical integration gives the same answer every time, so one doesn’t need to employ statistical methods to reason about the convergence of that algorithm.

Here’s the complete code for the numerical integration solution:

import sys
import math
from datetime import datetime as dt

def timedelta_to_microseconds(td): # http://stackoverflow.com/a/2416049
	return td.microseconds + (td.seconds + td.days * 86400) * 1000000.0

def calculate_height(x, r):
	# we could use an integer square root algorithm, e.g.
	# http://code.activestate.com/recipes/577821-integer-square-root-function/
	# that method is slower but will work for very large values of r
	return math.floor(math.sqrt(r**2 - x**2))

def circle_area(r): # numerically integrate using the rectangle rule with delta = 1
	return sum(calculate_height(x, r) for x in xrange(r))

def calculate_pi(r):
	return (4.0 * circle_area(r)) / r**2

radius = int(sys.argv[1])

before = dt.now()
pi = calculate_pi(radius)
elapsed = dt.now() - before

print 'pi = ' + str(pi)
print 'time elapsed: ' + str(elapsed)
print 'microseconds per operation: ' + str(timedelta_to_microseconds(elapsed)/(radius))

Imagine that you’re throwing darts at this collection of shapes, and you’re guaranteed that they’ll all land in the upper right section (a stretch depending on the quality of your dart game, but bear with me). Some of the darts will land within the inscribed circle, and some of those darts will also land in the inner triangle. By adding up the numbers of darts that land in each section, we can approximate the areas of each of these sections.

For simplicity, let r = 1. Since we’re restricting ourselves to the first quadrant, the (x, y) coordinates of each dart will be positive real numbers on the interval [0, 1]. We’ve constrained our system so that every dart thrown lands in the outer square, thus A_outer ≈ N, where N is the total number of darts thrown.

For a dart to land inside the circle, its coordinates must fulfill the equation x² + y² ≤ 1 (remembering that r = 1). Darts that land inside in the triangle have coordinates such that x + y ≤ 1 (where we’ve constrained x and y to be positive). To simulate a dart throw, we can randomly generate two floating-point numbers between 0 and 1 inclusive for the x- and y-coordinates respectively. Then, if they satisfy the circle equation, we increment the number of darts that hit the circle N_circle by 1, and if they satisfy the triangle equation, we increment the number of darts that hit the triangle N_triangle by 1.

Because A_circle ≈ N_circle and A_triangle ≈ N_triangle, we can say that π ≈ 4N_circle/N and π ≈ 2N_circle/N_triangle. Now let’s code up this simulation!

import random
import sys

def point_in_circle(x, y):
	return True if x**2 + y**2 <= 1 else False

def point_in_triangle(x, y):
	return True if x + y <= 1 else False # assume x and y are positive

def get_point():
	return random.random(), random.random()

def get_point_in_shapes():
	x, y = get_point()
	in_circle = point_in_circle(x, y)
	in_triangle = False
	if in_circle: # if the point isn't in the circle, we know that it can't be in the triangle
		in_triangle = point_in_triangle(x, y)
	return in_circle, in_triangle

def get_points_in_shapes(N_darts):
	N_circle, N_triangle = 0, 0
	i = 0 # keep track of how many darts we've thrown so far
	while i < N_darts:
		in_circle, in_triangle = get_point_in_shapes()
		if in_circle:
			N_circle += 1
		if in_triangle:
			N_triangle += 1
		i += 1
	return N_circle, N_triangle

def pi_formula(N_darts, N_circle):
	return 4.0 * N_circle / N_darts

def pi_formula2(N_darts, N_circle, N_triangle):
	if 0 == N_triangle: # if no darts landed in the triangle, avoid dividing by zero by falling
			    # back on the equation that doesn't use the area of the triangle
		return pi_formula(N_darts, N_circle)
	# otherwise, average the two approximations
	return N_circle * (2.0 / N_darts + 1.0 / N_triangle)

def calculate_pi(N_darts):
	N_circle, N_triangle = get_points_in_shapes(N_darts)
	return pi_formula2(N_darts, N_circle, N_triangle)

# driver
N_darts = int(sys.argv[1]) # the total number of darts to throw (given by a command-line argument)
print calculate_pi(N_darts)

(Note how each function is responsible for doing one specific task. This will simplify unit testing, a topic I’ll discuss in a later post.)

While fun to think about, this method takes a while to produce decent results. For example, I can consistently lock in the first three digits of π by “throwing” 10,000,000 darts on my computer, but it takes about 12.3 seconds to run. Additionally, the ability of this method to give a good approximation of π is limited by the precision of Python’s floating-point numbers. The darts’ positions are effectively quantized because Python only uses 53 bits to represent floats. In my next post, I’ll present a variation on this technique that allows us to take advantage of Python’s arbitrarily large integers to improve the approximation.

One of my New Year’s resolutions was to build my own desktop computer. I’d been getting frustrated with the performance of my laptop, but more importantly I had to do it for the nerd cred.

I mainly use my PC for software development, music production, and some gaming. Unfortunately, my laptop couldn’t run Diablo III at a decent resolution (forget 1920×1080) with minimal settings at even 25 frames per second. Cranking up the GPU clock speed helped a little, but laptops have enough heating issues without throwing overclocking into the mix. Additionally, when working on a musical composition, I couldn’t have very many synths, effects, or track automations running in real-time without getting choppy playback. Freezing tracks solved that issue to some extent, but it slowed down my workflow.

Choosing the parts

To get started, a coworker recommended the guide on Hardware Revolution. My budget was around $1,200, so the Tier 5 system provided a useful template which I then customized.

I began by upping the RAM from 8 to 16 GB since RAM is cheap and I like to have 400 tabs open at all times. I initially chose 2×8 sticks, but a friend recommended that 4×4 would perform better. I’ve since done some research into the differences between the two setups, but haven’t found much to explain why 4×4 would be faster. Since my motherboard only has four memory slots, upgrading would require getting all new modules. However, by the time 16 GB becomes obsolete, I would probably need to get all new RAM anyway.

For the video card, the guide recommended the GeForce GTX 660 Ti for about $300. Since I’m not a super hardcore gamer, I didn’t want to drop that much on a GPU. The standard GTX 660 at $209 seemed like a good deal to me, but another coworker persuaded me to go with the Radeon HD 7870, citing its improved specs and purported ability to better support multiple monitor setups.

I dabble in overclocking, so I wanted to get a CPU cooler to replace the stock one. Liquid cooling a system has become a lot easier in recent years due to the advent of simple-to-install, pre-assembled products. So, after doing some research, I settled on the Corsair H60. However, a friend suggested that, since I wasn’t doing any “extreme” overclocking, an air cooler would be preferable since it couldn’t leak fluid all my shiny new hardware while still giving improved cooling performance compared to the stock fan. With that advice, I selected the Cooler Master Hyper 212 EVO cooler, which, as an added bonus, was less than half the price of the H60.

Finally, the power supply. PC Part Picker will give you an estimated wattage for your system. eXtreme Outer Vision also has a good calculator. I don’t intend to run dual graphics cards in CrossFire configuration, so my power needs were a modest 365 W. A friend recommended the Seasonic X Series 560W 80 PLUS Gold on the basis of Seasonic’s reputation for building solid PSUs. This power supply has the advantage of being modular, resulting in less cable clutter within your case, while also providing power efficiently.

Here’s the final parts list.

And the parts themselves!

Putting everything together

The parts came in over the next week from both Newegg and Amazon. My girlfriend was upstate for the weekend, so I was able to take over our entire studio for the build.

Since this was my first self-assembled computer, I followed this excellent tutorial from Newegg.

In order to make sure everything worked properly, I began with an external build (as recommended in the video). Here I ran into my first bit of trouble. After slotting the CPU into the motherboard, I tried to close the tension arm, but past a certain point, it wouldn’t budge. I tried pushing it harder, but it didn’t go much further, and when I looked at the processor afterwards, it had little marks on the side where the retention piece had rubbed against it. Panicking, I started looking all over for someone who’d had the same problem. I found a post on Tom’s Hardware suggesting I just push harder. With this blessing, I gathered my courage and pushed the tension arm even harder, finally locking the CPU in place. With a sigh of relief, I continued by installing the cooler and RAM.

As you can see in this blurry photo, there’s barely enough clearance for the RAM below the CPU fan.

Just a sliver

Originally, I ordered this memory. After it shipped, I got last-minute advice from a friend who told me to get the low-clearance version of the RAM instead. A noob mistake for sure, but the RMA should finish processing any day now. =) I could also have used 2×8 GB sticks, slotting them further away from the CPU fan, or for added fun, filed down the edges of the module closest to the fan with a Dremel.

I slotted in the video card, then broke out the power supply. Since the PSU is modular, I had to try to figure out exactly which cables I needed. Picking the ones to power the motherboard and CPU was easy enough, but when it came time to power the video card, things got confusing. A number of them looked promising, but they either had an extra 2-pin connector dangling off of the 6-pin segment or they required a Molex adapter. After calling up a few coworkers, I learned that power supply cabling is fairly forgiving, so I just did whatever required the fewest adapters.

With that, the external build was complete! I considered leaving it like this for improved airflow, but I didn’t have any room on my desk to set my coffee mug.

IT LIVES!!

The internal build

Without opening the manual for the case, I started screwing in the power supply, optical drive, and motherboard. I tried mounting the mobo a couple times, but it wouldn’t sit quite right. After reading the case manual, I realized I’d been using the wrong screws for pretty much everything up until that point. Like Icarus, my hubris got the better of me. After amending the screwed-up (ha!) situation, I began wiring everything up (carefully referencing the motherboard manual along the way).

(As a side note, next time I’ll skip the external build and install the motherboard into the case before attaching the CPU cooler. The fan made it hard to access a couple of the rear motherboard mounting points.)

The last little bit of difficulty arose when installing the video and sound cards. The Gigabyte Radeon HD 7870 is quite sizable with its three fans, so maneuvering it around the cables in the case proved to be a bit tricky. But this minor speed bump did not hold me back for long.

Cable jungle

With bated breath, I powered everything up . . . and it still booted! Total build time: five hours. Time to boot Windows: five seconds (thanks, solid-state technology!).

The laptop watches jealously from the corner

Final thoughts

Building my own PC was a really fun experience. I learned a lot about computer hardware in the process, and I take pleasure in knowing exactly what is in my case and how it’s connected. I no longer fear configuration modifications like installing a second hard drive. Best of all, there’s no annoying bloatware pre-installed on my system that I have to clean out. As far as performance goes, Diablo III feels like a completely different game at 60 FPS with maxed settings, and I can use as many soft synths as I want with nary a hiccup.

I have three upgrades in mind:

1. A second hard drive. While my 3 TB external drive does a good job of holding all my data, I’d like to be able to dual-boot Linux without sacrificing any of the remaining 30 GB on my SSD to a second partition.
2. An internal case light. I’d like to be able to look down into the grille at the top of my case and see all of my beautiful hardware.
3. A third monitor. My graphics card supports up to four monitors, but I really only have space on my desk for two at the moment. So first I need to go to Ikea. Unless I build vertically. Hmm . . .

I heartily recommend that, if you’re in the market for a new PC, you consider building your own. There are tons of free resources out there to help you if you get stuck. You’ll end up with more machine for your money as well as a sense of pride in your accomplishment. And finally, you’ll always have something to talk about when you run into a fellow computer nerd. =)

Where the magic happens

sys.exit(0)