This semester I’m TA’ing a course on the Psychology of Music taught by Phil Johnson-Laird. It’s been a great course to teach because (i) so much of the material is new to me and (ii) because the study of the psychology of music brings together so many of the intellectual tools I enjoy, including music theory, psychophysics and Fourier analysis.

One topic this semester that was completely new to me was the theory of tuning: I had known about the invention of the well-tempered system of tuning, but had never heard of Pythagorean tuning or just tuning — and certainly was not aware that the well-tempered system Bach celebrated was not identical to our current equal-tempered system of tuning.

As a way of consolidating some of the knowledge I’ve gained, I decided I’d write a blog entry after several months of neglecting this blog. (For that neglect, I’ll blame a combination of grant writing, book writing, ongoing research projects and personal life developments.) In what follows, I’ll give a brief overview of the theory of tuning at a theoretical level that should be accessible to anyone who’s familiar with the names of intervals and feels comfortable thinking quantitatively.

After surveying the field, I’ll turn to a discussion of some code I’ve written in R that implements these ideas using the ‘tuneR’ package, which is one of my favorite hidden gems from CRAN. Along the way, I’ll introduce some of the simplest tools from the ‘tuneR’ package that can be used for generating computer music.

Tuning Systems: Pythagorean, Just and 12-Tet

It’s worth noting right at the start that tuning is a misleading name for the topic we’ll be discussing: we’re not talking about how one tunes a fixed instrument so that it sounds in tune, but rather we’re interested in how one defines the very notes that the instrument should be able to produce when it’s perfectly in tune.

To make that clear, let’s assume that we’ve accepted as a given that a frequency of 440 Hz will be called A. Our problem then becomes one of deciding which of the infinitely many frequencies we could produce actually deserves the label of A#, B, C, C#, and so on.

Pythagorean Tuning

The simplest solution to this problem I know of is the Pythagorean tuning system. It’s based on constructing all of the possible notes using a series of perfect fifths. If you remember the Circle of Fifths, you’ll remember that you can reach every chromatic note by ascending fifths: if you start at A, you’ll proceed through E, B, F# and so on.

The Pythagorean system implements the Circle of Fifths directly using repeated multiplication of a base frequency. To do this, you first declare that a perfect fifth is at a frequency 3/2 above your base frequency. For example, this definition implies that the perfect fifth above the A at 440 Hz has to be at a frequency of 3/2 * 440 = 660 Hz. Once you do this, you’ve defined the frequency we’ll call E.

And following on with this logic, you produce a B at 990 Hz. Of course, this B occurs an octave above the base A at 440 Hz, so you transpose it down an octave to produce the B you’ll actually use. To do this, you need to assume that an octave is at a frequency 2 times the base frequency. Since we’ve accepted that 990 Hz is a B, we divide 990 by 2 and conclude that 495 Hz should be B.

With these three notes defined, we have the following table of frequency/note pairs:

If we continue on with this logic and calculate many more multiplications by 3/2 and divisions by 2, we will eventually produce a complete table for all of the notes in the chromatic scale that looks like the following:

One thing about this table might strike you as odd if you’re mathematically savvy: the octave, which we’ve defined by fiat as a ratio of 2:1, could never have been produced by successive multiplication by 3/2, since no power of 3 will be evenly divisible by a power of 2. This is the one flub in the Pythagorean system: you can’t really produce the entire chromatic scale using only multiples of 3/2. Here we’ve solved that problem by replacing the note we would have called A with a true octave generated using multiplication by 2. Because the exact octave produced by Pythagorean tuning is slightly out of tune with our preferred definition of an octave, you may hear people refer to this discrepancy as the the Pythagorean comma.

Just Tuning

Given that we had to cheat a bit to create a proper octave using the Pythagorean tuning system based on multiples of 3/2, it makes sense to ask why we shouldn’t just allow ourselves to use other multipliers than 3/2. Looking at the Pythagoren tuning table, we see some pretty ugly fractions like 729/512. What if we forced these fractions to be simpler by employing ratios like 4/3 and 5/4 to build up the whole system?

The result of allowing ourselves several fractions beyond just those derived from 3/2 is called the just tuning system. Here we assume that perfect fifths occur at a frequency ratio of 3/2 and that perfect fourths occur at a frequency ratio of 4/3. Continuing on with this process, we eventually end up with the following tuning table:

This is the tuning that early Classical music was written in. Looking at the table you con immediately appreciate the theoretical assertion that the relative dissonance of an interval is determined by the simplicity of the ratio of frequencies between the two notes: perfect fifths are 3/2 and major thirds are 5/4, while minor seconds are 16/15 and major sevenths are 15/8. This is one of the things I most enjoy about the theory of harmony: there’s a match between the aesthetics of fractions and the aesthetics of sounds that, for me, helps to justify my sense that certain fractions are more beautiful than others.

12 Tet / Equal-Temperament

Now, if you know the history of Bach’s Well-Tempered Clavier, you know that there is a problem with the just tuning system: it sounds great in the key you used as the base (here A), but it sounds a bit out of tune in other keys. The modern 12-tet system is the most recent approach to solving this problem: you assume the gap between two semitones (e.g. A to A# or A# to B) is always the exact same multiple. Since you’ll repeat this multiplication 12 times before reaching an octave, you can conclude that two notes that are a semitone apart must be separated by the 12th root of 2. Building a tuning system using that ratio alone gives us our modern system of tuning, which is shown in the table above using the decimal expansion of the ratios instead of their representation as powers of the 12th root of 2:

Listening to the Results

We’ve just described three ways to define the notes used in Western music. But how different do they sound? To answer that, I decided to produce a series of simple sine wave audio samples that were tuned using each of the three tuning systems. To produce those audio samples, I used the ‘tuneR’ package, which I’ll describe now. Before you read on, you should install it from CRAN using the standard install.packages('tuneR') invocation.

A tuneR Tutorial

The tuneR package is an extremely convenient tool for generating audio files from R based on a numeric description of the audio stream. For the purposes of this discussion of tuning systems, we simply need to produce basic sine waves. Thankfully, that’s very easy to do with tuneR. Here’s an example:

1
2
3
4
5

library('tuneR')
 
sound <- sine(440, bit = 16)
 
writeWave(sound, '440.wav')

Here we’ve loaded the tuneR package, created a 1s snippet of sine wave audio at 16 bits resolution using the sine function, and then written out the audio to a WAV file using writeWave. If you look at your current directory and listen to this file, you’ll hear a sine wave at 440 Hz.

If you want to explore the use of sine, you can easily play with the duration of the sound by changing the duration parameter. If you want to, you can also change the sample rate and the bit rate, but I don’t see any reason to do that while exploring ideas about tuning.

More important is knowing that you can superimpose two sine waves using the `+` operator and that you can concatenate them using the bind function. To show off producing octaves, for example, you might use the following code to hear an A at 440 Hz, then an A an octave above it, and finally the harmony they produce together:

1
2
3
4
5
6
7

library('tuneR')
 
sound <- bind(sine(440, bit = 16),
              sine(880, bit = 16),
              sine(440, bit = 16) + sine(880, bit = 16))
 
writeWave(sound, 'octaves.wav')

Unfortunately, this sample code produces an error because of the naive addition we’ve implemented using the `+` operator. Adding two sine waves directly together overfills the bit rate we’re using. To safely perform addition of two sine waves, we need to normalize the results of our summation using the normalize function. This gives us just one more line of code:

1
2
3
4
5
6
7
8
9

library('tuneR')
 
sound <- bind(sine(440, bit = 16),
              sine(880, bit = 16),
              sine(440, bit = 16) + sine(880, bit = 16))
 
sound <- normalize(sound, unit = '16')
 
writeWave(sound, 'octaves.wav')

For reasons that are not clear to me, you have to specify the bit rate to normalize using the unit parameter rather than the bit parameter.

Demoing Tuning Systems

Our little octave demo is cute, but we really want to know what more interesting harmonies like major thirds and minor seconds sound like in the various tuning systems we described. To do that, I first wrote a function called interval that spits out the multiplier you need to use to produce a given interval for any of the three tuning systems. That function is in a GitHub repository I’ve set up with code for making these demos. If you download that repository, you could load my interval function using a simple call to source like the one seen below. And using this interval function, we can generate demos of various intervals as follows:

1
2
3
4
5
6
7
8
9
10
11

library('tuneR')
source('interval.R')
 
base <- 440
 
sound <- sine(base) + sine(interval('minor-second',
                                    tuning = 'pythagorean') * base)
 
sound <- normalize(sound, unit = '16')
 
writeWave(sound, 'minor_second_pythagorean.wav')

On GitHub there’s a file called test_intervals.R that will go through and generate all of the intervals in all three tuning systems. If you run that file, you’ll generate a lot of audio files you can listen to as demos of the three tuning systems we’ve described. For me, these tuning systems all produce intervals that sound surprisingly similar, though at high volumes I find it moderately easy to hear slight differences between the tuning systems. That said, I very much doubt I would pick up on them in a normal musical context.

That’s the end of my little introduction to tuning systems and the use of the tuneR package to explore them. If you’re interested in thinking computationally about music, I highly recommend playing around with tuneR until you feel like you can produce interesting results. I’m already working on trying to build up some interesting timbres to work with.

Show that every number of the form ABC,ABC is divisible by 13.

The puzzle comes up in Baron’s book as an example of an “insight problem” in which one goes from not knowing the answer at all to knowing the complete answering in a sudden moment of insight.

Several people replied to my tweet with solutions: I especially like Will Townes’s solution. In particular, if you’re familiar with modular arithmetic, I like the logic of Will’s answer because it gives a simple generalization. First, represent ABC,ABC as ABC * 1000 + ABC * 1 rather than as ABC * 1001. Then notice that

1. 1 = 1 mod 13
2. 1000 = -1 mod 13

Thus ABC,ABC = ABC * -1 + ABC * 1 = 0 mod 13. This logic can be easily extended to show that (ABC,ABC,)*ABC,ABC = 0 mod 13 no matter how many times you repeat the ABC,ABC pattern.

Allison Chaney pointed out that many fields (for instance, environmental engineering) might want to consider using these circular displays to make periodic trends clear to the viewer. That inspired me to try plotting periodic weather data using both a standard x-y plane display and a polar coordinates display. The results are shown below in two videos that I’ve uploaded to Vimeo:

There’s a clear tradeoff that’s being made when choosing between these two approaches: the polar coordinates plot, as promised, correctly connects the two “ends” of the data set. But it also makes it much harder to see the height of the graph at each point in time, so that the sinusoidal shape that can easily be seen in the x-y plane display is basically hidden in the polar coordinates display.

Since making these videos, it occurred to me that another potential visualization technique would be to project the data onto a cylinder, rather than a plane, and then progressively rotate the cylinder to reveal the time trend. This would allow heights to be seen properly, while emphasizing the periodicity. The problem with this cylindrical projection is that the entire data set is never fully visible at one time, but can only be seen by completing a full rotation of the data.

In his paper, Tukey describes one other approach: draw the periodic data twice so that the period is clearly visible. It wasn’t clear to me how to do this without some numeric hacks in ggplot2, so I’ll leave it to reader to search for Tukey’s example in the original paper.

• The first piece of ProjectTemplate news is that I won’t be the exclusive maintainer for ProjectTemplate anymore. Allen Goodman, who works at BankSimple, is now my co-maintainer and he has full commit privileges. In the next few months, the emerging group with commit privileges is likely to grow beyond the two of us, but hopefully just having one more person in charge of ProjectTemplate’s development will help to keep things moving forward.
• There’s a new draft of ProjectTemplate available on GitHub. v0.3-1 fixes problems with the YAML configuration system not working on Windows 64 machines by switching over to the DCF format that R naturally supports. Editing your configuration scripts should be trivial, but be prepared for ProjectTemplate to break on your existing v0.2-1 projects until you’ve updated them to use DCF instead of YAML.
• In addition to switching the configuration system over to DCF, ProjectTemplate v0.3-1 now uses namespaces and separate functions to implement all of the automatic data loading functions that were previously nested inside of load.project(). Hopefully this will make it easier for end users to override ProjectTemplate’s defaults, while allowing ProjectTemplate releases to automatically rolls out bug fixes to less advanced users. On that note, the list of supported file formats for automatic data loading is growing and new patches on that front are always welcome.
• A minimal project format: Some people have asked for the option to create projects without some of the clutter that the standard project format creates, such as the diagnostics and profiling directories. There’s now a minimal project format that you can use by invoking create.project() with the option create.project(minimal = TRUE).
• Starting in two weeks, the version of ProjectTemplate available on CRAN will stay in pace with the version on GitHub. If you’re still using v0.1-3, please consider upgrading or forking.
• There is now an official ProjectTemplate website at http://projecttemplate.net/ that will hopefully be the start of a new era of better documentation for ProjectTemplate. While the material on the site is still in noticeably draft form, I expect the documentation to improve considerably in the near future. If anyone out there is a graphic designer and would like to make the new site look better, please let me know by e-mailing me at jmw@johnmyleswhite.com.

For now that’s all, but there’s more ProjectTemplate news coming soon. Stay tuned!

In the process, I’ve started to learn something about how to write code that runs quickly in R. In this post, I’ll try to share some of that knowledge by describing three ways of performing maximum likelihood estimation in R whose runtimes differ by two orders of magnitude. The differences seem to depend upon two factors: (1) how I access the entries of a data frame and (2) whether I use loops or vectorized operations to perform basic arithmetic.

To simplify things, I’ll present a model that should be familiar to people with a background in economics: the exponentially discounted utility model. To implement it in R, we define the discounted value of x dollars at time t as:

1
2
3
4

discounted.value <- function(x, t, delta)
{
  return(x * delta ^ t)
}

In addition to the discounted utility model, we assume that choices originate from a stochastic choice model with logistic noise. To invert this noise during inference, we’ll use the inverse logit transform:

1
2
3
4

invlogit <- function(z)
{
  return(1 / (1 + exp(-z)))
}

To test my inference routine, I need to generate “stochastic” data of the sort you would expect to see from an exponentially discounting agent that’s indifferent between having $1 at time t = 0 and $3 at time t = 1. I’ll refer to the first good as (X1, T1) and the second good as (X2, T2). If the agent chooses (X2, T2), I’ll write that as C == 1; if they choose (X1, T1), I’ll write that as C == 0. With those conventions, the sample data is generated as:

1
2
3
4
5
6
7

n <- 100
 
choices <- data.frame(X1 = rep(1, each = n),
                      T1 = rep(0, each = n),
                      X2 = rep(3, each = n),
                      T2 = rep(1, each = n),
                      C = rep(c(0, 1), by = n / 2))

To fit the exponential model to this data set, we’ll use the optim function to minimize the negative log likelihood of the data by setting two parameters: a, the variance of the noise in the utility function; and delta, the discount factor in the discounted utility model. The three implementations of this model that I’ll show only differ in the definition of the log likelihood function, so the final call to optim to perform maximum likelihood estimation is constant across all examples:

1
2
3
4
5
6

logit.estimator <- function(choices)
{ 
  wrapper <- function(x) {-log.likelihood(choices, x[1], x[2])}
  optimization.results <- optim(c(1, 1), wrapper, method = 'L-BFGS-B', lower = c(0, 0), upper = c(Inf, 1))
  return(optimization.results$par)
}

Here, I had to specify bounds for the parameters, a and delta, because it’s assumed that a must be positive and that delta must lie in the interval [0, 1]. To deal with these bounds, one has to use the L-BFGS-B method in optim.

The first implementation I’ll show is the one I find most natural to write, even though it turns out to be the least efficient by far:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

log.likelihood <- function(choices, a, delta)
{
  ll <- 0
 
  for (i in 1:nrow(choices))
  {
    u2 <- discounted.value(choices[i, 'X2'], choices[i, 'T2'], delta)
    u1 <- discounted.value(choices[i, 'X1'], choices[i, 'T1'], delta)
 
    p <- invlogit(a * (u2 - u1))
 
    if (choices[i, 'C'] == 1)
    {
      ll <- ll + log(p)
    }
    else
    {
      ll <- ll + log(1 - p)
    }
  }
 
  return(ll)
}

In the second implementation, I define a row level likelihood function, so that the summing and logarithmic transform are vectorized.

1
2
3
4
5
6
7
8
9
10
11
12
13

rowwise.likelihood <- function(row, a, delta)
{
  u2 <- discounted.value(row['X2'], row['T2'], delta)
  u1 <- discounted.value(row['X1'], row['T1'], delta)
  p <- invlogit(a * (u2 - u1))
  return(ifelse(row['C'] == 1, p, 1 - p))
}
 
log.likelihood <- function(choices, a, delta)
{
  likelihoods <- apply(choices, 1, function (row) {rowwise.likelihood(row, a, delta)})
  return(sum(log(likelihoods)))
}

In the third implementation, I define a fully vectorized log likelihood function that avoids any explicit iteration and therefore removes most of the data frame indexing operations:

1
2
3
4
5
6
7
8

log.likelihood <- function(choices, a, delta)
{
  u2 <- discounted.value(choices$X2, choices$T2, delta)
  u1 <- discounted.value(choices$X1, choices$T1, delta)
  p <- invlogit(a * (u2 - u1))
  likelihoods <- ifelse(choices$C == 1, p, 1 - p)
  return(sum(log(likelihoods)))
}

The code I used to call all of these implementations and compare them is up on GitHub for those interested. The results, which strike me as remarkable, are below:

1. On my laptop, implementation 1 takes ~1.0 second to run.
2. On my laptop, implementation 2 takes ~0.25 seconds to run.
3. On my laptop, implementation 3 takes ~0.01 seconds to run.

In short, the third implementation is 100x faster than the first implementation with only minor changes to the code I originally wrote. Hopefully this example will help inspire others who have R code they’d like to speed up, but aren’t sure where to start.

In this, Chomsky is in complete agreement with O’Reilly. (I recognize that the previous sentence would have an extremely low probability in a probabilistic model trained on a newspaper or TV corpus.)¹

Anyone who considers themself an intellectual should be required to read this new essay by Peter Norvig. It’s the best summary I’ve ever seen of the many types of science that now exist in our world — almost all of which are moving away from the simple algebraic, deterministic models of the world that fill high school science textbooks.

1. On Chomsky and the Two Cultures of Statistical Learning

Unfortunately, I ran into some serious problems along the way. After installing everything from scratch, ‘ggplot2′ 0.8.9 was broken. Specifically, I couldn’t get error bars to work with stat_summary(). For example, this code wouldn’t work on my system:

1
2
3
4
5
6
7
8
9
10
11

# Problem with ggplot2 Version "0.8.9"
 
library('ggplot2')
 
set.seed(1)
 
example.data <- data.frame(Measurement = rnorm(5, 0, 1), Class = rep('A', 5))
 
ggplot(example.data, aes(x = Class, y = Measurement)) +
  stat_summary(fun.data = 'mean_cl_boot', geom = 'bar') +
  stat_summary(fun.data = 'mean_cl_boot', geom = 'errorbar')

Thankfully, I managed to enlist Dirk Eddelbuettel’s help through Twitter and he ran the code on his own recently updated system. Things worked fine for him, which suggested that the problem was in my system configuration. We compared package versions and discovered that he had ‘plyr’ 1.5.1 on his Ubuntu machine, while I had ‘plyr’ 1.5 on my OS X machine. After looking at CRAN, it was clear that the Mac OS X build wasn’t available on CRAN yet.

To fix this, I grabbed the source for ‘plyr’ 1.5.1 and tried to install it myself. That led to the following error:

1
2

** preparing package for lazy loading
Error: package 'plyr' is required by 'reshape' so will not be detached

The problem was that ‘reshape’ was being loaded automatically when R was starting up. Since ‘reshape’ depends on ‘plyr’, R wasn’t willing to overwrite my old ‘plyr’ 1.5 with the new ‘plyr’ 1.5.1. The solution was to edit my .Rprofile file to prevent ‘reshape’ from being autoloaded. Once I did this, I was able to run the standard R CMD INSTALL and get the new version of ‘plyr’ on my system. And after that ‘ggplot2′ 0.8.9 started working properly.

Hopefully no one else will come up against the same issue after the binary for ‘plyr’ 1.5.1 gets pushed through all of the CRAN mirrors. But if you get errors while using ‘ggplot2′ 0.8.9, look into installing ‘plyr’ 1.5.1 from source on your system.

Many thanks to Dirk for giving me so much help today.

I’m trying to collect data sets that showcase how the classical statistical distributions appear in modern contexts. I’ve already got some data that shows how the gamma distribution appears in video game scores, and now I’m hoping to find an example where the exponential distribution shows up. I think that checkins for Foursquare might be a good place to start.

To test this intuition, I’m hoping to collect some pilot data. Below you’ll find some code that you can use to help me gather data.

First, there’s a shell script to gather your own checkin data from FourSquare. To use this script, you need to substitute your e-mail address where EMAIL appears and your password where PASSWORD appears in the code below:

1

curl -u 'EMAIL:PASSWORD' https://api.foursquare.com/v1/history?l=250 > checkin_history.xml

And second there’s an R script you can use to preprocess the data from the last step into a nice format before sending it to me. If you’re not an R user, you can easily skip this step and send the data you have in its raw XML format.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

library('plyr')
library('XML')
filename <- 'checkin_history.xml'
tree <- xmlTreeParse(filename, asTree = TRUE)
checkins <- tree$doc$children$checkins
venue.names <- c()
latitudes <- c()
longitudes <- c()
for (i in 1:length(checkins))
{
  venue.names <- c(venue.names, as.character(checkins[i]$checkin[['venue']][['name']][['text']])[6])
  latitudes <- c(latitudes, as.numeric(unclass(checkins[i]$checkin[['venue']][['geolat']][['text']])$value))
  longitudes <- c(longitudes, as.numeric(unclass(checkins[i]$checkin[['venue']][['geolong']][['text']])$value))
}
checkin.data <- data.frame(Venue = factor(venue.names), Latitude = as.numeric(latitudes), Longitude = as.numeric(longitudes))
count.data <- ddply(checkin.data, 'Venue', nrow)
names(count.data) <- c('Venue', 'TotalCheckins')
write.csv(count.data, file = 'count_data.csv', row.names = FALSE)

After running these two pieces of code, the output file, count_data.csv, should look like this:

Once you’ve got data, you can send it to me by e-mail at jmw@johnmyleswhite.com.

I asked for examples of such a function on Twitter today and didn’t get any answers, so I decided to build my own. As I see it, there are two ways to visualize a function of two variables:

1. Use a 3D surface.
2. Use a heatmap.

But 3D surfaces aren’t currently available in ggplot2, so I decided to work with heatmaps. The function below provides a very simple implementation of my 3D version of curve(), which I call curve3D():

1
2
3
4
5
6
7
8
9
10

curve3D <- function(f, from.x, to.x, from.y, to.y, n = 101)
{
	x.seq <- seq(from.x, to.x, (to.x - from.x) / n)
	y.seq <- seq(from.y, to.y, (to.y - from.y) / n)
	eval.points <- expand.grid(x.seq, y.seq)
	names(eval.points) <- c('x', 'y')
	eval.points <- transform(eval.points, z = apply(eval.points, 1, function (r) {f(r['x'], r['y'])}))
	p <- ggplot(eval.points, aes(x = x, y = y, fill = z)) + geom_tile()
	print(p)
}

Here’s an example of the use of curve3D to explore the behavior of Loewenstein and Prelec’s Generalized Hyperbolic discounting function:

g <- function(x, y) {(1 + y * 2) ^ (-x / y) * (1 + y * 1) ^ (x / y)}

curve3D(g, from.x = 0.01, to.x = 1, from.y = 0.01, to.y = 1)

I'd love suggestions for cleaning this function up. Two obvious improvements are:

1. Allow the function to accept arbitrary expressions and not just functions as inputs.
2. Allow the user to see 3D surfaces or heatmaps.

I suspect that the first problem would be a great way to learn about functional programming in R -- especially R's methods for quoting, parsing and deparsing expressions.

Neil Kodner recently got me interested again in analyzing Canabalt scores statistically by writing a great post in which he compared the average scores across iOS devices. Thankfully, Neil’s made his code and data freely available, so I’ve been revising my original analyses using his new data whenever I can find a free minute.

Returning to Canabalt has been a lot of fun, especially because my grasp of statistical theory is a lot stronger now than it was when I published my last post on Canabalt scores. For example, I actually know now what I was trying to say when I publicly described my search for a Poisson distribution in the posted scores. At the time, I had just read about Poisson regressions and was therefore eager to fit Poisson models to the scores data, even though the Poisson model gave very poor results. In retrospect, it’s clear that I was misled by superficial similarities in statistical terminology. What I was really looking for in the data was something closer to a Poisson process than to a Poisson distribution. Unfortunately, I didn’t really understand Poisson processes at the time I wrote my original posts, so I only succeeded in showing that Canabalt scores could not reasonably be claimed to be Poisson distributed.

Generating Process

But now that I have more data and more experience, it’s easy to see what I was struggling to articulate before: Canabalt scores seem to be generated by something like a truncated Poisson process. From my current perspective, the generating process for a Canabalt score is essentially the following:

1. Initialize the player’s score to zero.
2. Iterate the following steps repeatedly:
1. Draw the number of meters before the next obstacle from a Poisson distribution. Add this value to the player’s current score.
2. Draw the identity of the next obstacle from a multinomial distribution.
3. For every type of obstacle, there is a constant probability p that a player will die when they come up against an instance of that type of obstacle. Given the value of p, draw a Bernoulli variable with probability p of coming up heads. If the result is heads, the player dies and their score is the current value of score, which is therefore simply the sum of several Poisson variables. If the result is tails, go back to the top of this loop.

This intuition, unfortunately, isn’t trivial to express as a model that I can quickly fit to the data I have. I haven’t tried very systematically to build a formal version of this model because there seem to be obvious problems of identification. For example, one can imagine that the values for the obstacle-specific probabilities described above can all be lowered by a constant proportion at the same time that the mean of the underlying Poisson distribution is decreased while the distribution of outcome scores is left constant. My intuition tells me that the only way around this would be to exploit the variance of the data and the restriction of the Poisson distribution to integer values, but I haven’t pushed very hard on this. If others are interested in pursuing it, I think there’s an interesting open problem here.

Thankfully, there’s another approach to modeling the data that’s simpler. As Owe Jessen noted in a comment on my earlier post, the distribution of scores looks like one parameterization of the gamma distribution. When Owe made this suggestion originally, I tried to use fitdistr from the MASS package to fit a gamma model to the scores data, but was never able to get the code to work properly. But now that I’m reasonably fluent in BUGS, it’s quite easy to fit a gamma distribution to the empirical scores data.

Gamma Distribution

Using JAGS, fitting a gamma distribution to the scores data is basically a trivial problem that requires only a few lines of code:

1
2
3
4
5
6
7
8
9
10

model
{
	for (i in 1:N)
	{
		score[i] ~ dgamma(shape, rate)
	}
 
	shape ~ dgamma(0.0001, 0.0001)
	rate ~ dgamma(0.0001, 0.0001)
}

Invoking this BUGS model from R using code that I’ve put on GitHub allows us to estimate the scale and rate parameters for a gamma distribution in a few minutes. The resulting parameterization of the gamma distribution looks very similar to the empirical density that we can estimate using a KDE:

Beyond visual comparisons, we can formally test the fit of the gamma model using a K/S test. The K/S test tells us that we should reject the gamma model, but it’s a fairly weak rejection at p = 0.005. Given that we have over a thousand data points, I think this weak rejection suggests that the gamma model is not such a bad approximation to the true score distribution.

Where the gamma model seems to noticeably fail is in the tail of the scores distribution:

To exaggerate the differences here, I’ve used a square root transform on the y axis so that we can see the bumps in the estimated density plot that are missing from the theoretical gamma model.

Since generating these images, I’ve had a chance to read a bit about heavy tailed distributions, but haven’t yet tried fitting any of them to this data set. I’ll probably start with a Pareto distribution, though I’d really like to find a discrete heavy tailed distribution over the natural numbers rather than a continuous distribution over the non-negative reals.

Looking further past the gamma model’s misfit in the tails, there’s another reason that I like the gamma model: the gamma distribution has an origins story that has some points of connection to the generative model I outlined above. Specifically, adding a bunch of exponential variables together will give you a gamma distribution (also called an Erlang distribution in this context). While I’m skeptical that the distribution of meters between obstacles can be reasonably treated as if it were an exponential distribution, the summation origin for the gamma distribution is a nice point of connection to my intuitions about a data generating mechanism that behaves like a Poisson process.

Hierarchical Gamma Model

The gamma model has another point in its favor: it’s easy to write down a hierarchical model that fits a distinct gamma distribution to subsets of the original data. By fitting multiple gamma distributions, we can easily make comparisons between the estimated score distributions for the different devices that Neil analyzed. As Neil showed, there are enough rows in the current data to do this in a principled way across devices without resorting to gamma distributions, but a hierarchical model provides us with tools for thinking about comparisons across different types of deaths, where we don’t have enough data to use density estimates or other non-parametric methods.

Without a distributional model, I’d be skeptical about estimating differences between groups with such unequal sample sizes. (Even with a model, I don’t think we can make strong conclusions about differences between all of the groups in this data set.) That said, within a hierarchical model, I feel more comfortable estimating several conditional distributions from a small data set, because the hierarchical model provides enough shrinkage to prevent us from arriving at extreme conclusions simply because some groups were undersampled in our data set. (Of course, shrinking the model parameters for small groups towards the global mean may lead us to miss differences that are real.)

All that said, the analyses I describe below implement a hierarchical model that I’ve used to estimate the mean and standard deviation of the score distribution for each of the iOS devices and each of the death types in our data set.

First, let’s model the expected score for a player who died because of each of the various possible obstacles they might come across. I’ll refer to these different types of deaths as the death types. My JAGS code for estimating the shape and rate parameters for each death type is below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

model
{
	for (i in 1:N)
	{
		score[i] ~ dgamma(shape[death[i]], rate[death[i]])
	}
 
	for (j in 1:K)
	{
		shape[j] ~ dgamma(alpha.shape, beta.shape)
		rate[j] ~ dgamma(alpha.rate, beta.rate)
	}
 
	alpha.shape ~ dgamma(0.0001, 0.0001)
	beta.shape ~ dgamma(0.0001, 0.0001)
 
	alpha.rate ~ dgamma(0.0001, 0.0001)
	beta.rate ~ dgamma(0.0001, 0.0001)
}

As you can see, I estimate group hyperparameters that partially pool data across all of the death types: these hyperparameters are themselves given weakly informative gamma priors in the last four lines of the model. The results give us the following estimates for the mean score for each death type along with the estimated standard deviation:

I’ve chosen to use means and standard deviations rather than the shape and rate parameters that we’re actually fitting, because the shape and rate parameters are on such different scales that only one of them can be visualized effectively at a time. There are simple formulas for translating between these parameterizations of the gamma distribution: the translation scheme can be found on Wikipedia.

Having estimated these parameters across death types, I can also approach the question that Neil addressed in his post by comparing the score distributions across iOS devices. The JAGS code to do so is almost identical to the code for estimating the gamma parameters across death types:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

model
{
	for (i in 1:N)
	{
		score[i] ~ dgamma(shape[device[i]], rate[device[i]])
	}
 
	for (j in 1:J)
	{
		shape[j] ~ dgamma(alpha.shape, beta.shape)
		rate[j] ~ dgamma(alpha.rate, beta.rate)
	}
 
	alpha.shape ~ dgamma(0.0001, 0.0001)
	beta.shape ~ dgamma(0.0001, 0.0001)
 
	alpha.rate ~ dgamma(0.0001, 0.0001)
	beta.rate ~ dgamma(0.0001, 0.0001)
}

In this case, the results tell a much simpler story about the differences between the three devices in our data set:

Multinomial Model

As a closing point, there’s one more modeling project for which I’ve used JAGS on the new Canabalt data: I’ve tried to estimate the probability of suffering each type of death along with an indication of the uncertainty in our estimates. Of course, this is a simple problem to solve using maximum likelihood methods: you can just plug in the empirical frequencies. But, given my growing love for Bayesian methods and the use of credible intervals to summarize uncertainty, I decided that I would estimate the multinomial model for death types using a Dirichlet prior centered on a uniform multinomial distribution. Here’s my JAGS code for estimating the multinomial model:

1
2
3
4
5
6
7
8
9
10

model
{
	deaths[1:K] ~ dmulti(p[1:K], N)
	p[1:K] ~ ddirch(alpha[1:K])
 
	for (i in 1:K)
	{
		alpha[i] <- 1 / K
	}
}

And here are the final results:

Conclusion

It’s been a lot of fun coming back to this topic. I still want to understand the outliers in the data better, but I’ll leave that for another day. If you’re interested in exploring this data set yourself, I encourage you to go to the GitHub repository I’ve set up and explore the JAGS code that I’ve used to fit these models. There you can also find higher effective resolution PDF’s of the graphs that you see here, which are admittedly a bit hard to read at this resolution.

Finally, I’d like to thank Neil Kodner for having put so much work into collecting more Canabalt data and analyzing it. Essentially all of the work I’ve done here is just a Bayesian reformulation of the questions that Neil addressed already in his own post.

Another plus for the book is that it covers some of the graphing functionality that’s provided by the base, lattice and ggplot2 packages. That last bit is particularly important to me: I hope this book represents one of the first members of a new generation of books on R that will treat ggplot2 as a default tool for building graphs in R — if not the default tool. That said, I’d actually have been happy if the book had proselytized more strongly for ggplot2, even though I can imagine that it’s probably a good thing to expose beginner users to the traditional tools in R for visualizing data in addition to the newest favorite contender.

For intermediate users, I suspect that the most useful part of the book will be the discussion in the second chapter of some of the lower level graphic parameters that par() gives you access to. I’d never learned to use those features very effectively, so the chapter covering those details was particularly helpful for me. Also, there’s a later chapter on creating maps that’s quite nice, as well as a final chapter that’s focused on producing publication-ready graphics. Both of those chapters could be very useful for more advanced readers: for example, I knew absolutely nothing about working with fonts in R before reading those sections.

In short, the R Graphs Cookbook seems like a useful book to have around for most R hackers and a potentially very valuable resource for new programmers. If you’re interested in finding a starting point for learning to build graphs in R, I’d suggest considering this book.

The Bayesian-Frequentist debate reflects two different attitudes to the process of doing science, both quite legitimate. Bayesian statistics is well-suited to individual researchers, or a research group, trying to use all the information at its disposal to make the quickest possible progress. In pursuing progress, Bayesians tend to be aggressive and optimistic with their modeling assumptions. Frequentist statisticians are more cautious and defensive. One definition says that a frequentist is a Bayesian trying to do well, or at least not too badly, against any possible prior distribution. The frequentist aims for universally acceptable conclusions, ones that will stand up to adversarial scrutiny. The FDA for example doesn’t care about Pfizer’s prior opinion of how well it’s new drug will work, it wants objective proof. Pfizer, on the other hand may care very much about its own opinions in planning future drug development.¹

To me, it’s amazing how similar the ambiguous regions of behavioral decision theory are to the major questions of theoretical statistics: people seem largely unable to systematically decide whether they want to be minimaxing (which seems very close to Efron’s vision of frequentist thought as stated here) or whether they want to be minimizing expected risk (which is closer to my own vision of Bayesian thinking). My own sense is that we learn as a global culture, over time, which error functions are least erroneous — and we do so largely by trial and error.

Most interesting to me is to consider individual differences in the error functions people effectively use: I suspect political preferences correlate with a propensity to focus on worst case thinking rather than average case thinking. Also, I’m fascinated by the way that a single person switches between worst case and average case thinking: I suspect there’s as much to be learned here as there was in understanding what drives risk seeking behavior and what drives risk average behavior.

HT: John D. Cook

1. 
   Bradley Efron : Modern Science and the Bayesian-Frequentist Controversy

But modeling devices that make sense for an unbiased decisionmaker may not make sense for a biased one. For example, why would individuals have priors and posteriors if they are destined to apply Bayes’ law incorrectly?¹

A question I often ask myself.

1. Wolfgang Pesendorfer : Behavioral Economics Comes of Age: A Review Essay on Advances in Behavioral Economics

1. Probabilists and Statisticians: “I think that’s an interesting case, but it’s in a set with measure zero.”
2. Economists: “X group’s behavior is clearly rent-seeking.”

Do any readers have good examples from other fields?

Foremost of the changes is that most of the logic for load.project() is now built into the load.project() function directly, rather than spread out into autogenerated scripts that you can edit by hand. While this makes ProjectTemplate harder for non-experts to modify, the change will make it much easier to make revisions to ProjectTemplate in the future without having to worry about existing projects falling behind because of vestigial code that’s not being automatically updated when you install a new version of ProjectTemplate.

Because more system logic is now hardcoded into functions, each project’s configuration is handled through a YAML file in config/global.yaml. Incidentally, this introduces the new directory, config/, where configuration files will go from now on.

The data loading system is also more complex than it was before. First, there’s a new hierarchy of data sources: now the system will look for data in a cache/ directory before moving on to the data/ directory. This makes it possible for you to permanently store changes to your data set in cache/ that will allow you to skip loading the raw data set. This is helpful when the original data set is enormous and you only need a radically reduced form of it for your future analyses that you’ll store in cache/.

In addition, preprocessing is now handled through a series of ordered scripts in a munge/ directory rather than just a single preprocessing script in the lib/ directory. There’s also a log/ directory, used by the new integrated log4r support, which is off by default, but can be easily set up after installing log4r from CRAN.

Finally, there’s a src/ directory where we’re going to encourage users to place their primary analyses, so that the main directory always has the same files and directories across all projects.

In addition to all of these changes, many of which were inspired by conversations with Mike Dewar, I’ve incorporated some very helpful patches in this release. Specifically, Diego Valle-Jones fixed a bug in clean.variable.name() that lead to trouble when filenames in the data/ directory began with numbers and Patrick D. Schalk contributed code that adds support for SQLite to ProjectTemplate along with general improvements to the database access codebase.

Thanks for all of the support since the last release. Please let me know if there any changes that need to be made before I turn v0.2-1 loose on CRAN.