Archives by date

You are browsing the site archives by date.

What is Correctness for Statistical Software?

What is Correctness for Statistical Software?

Introduction A few months ago, Drew Conway and I gave a webcast that tried to teach people about the basic principles behind linear and logistic regression. To illustrate logistic regression, we worked through a series of progressively more complex spam detection problems. The simplest data set we used was the following: This data set has […]

What is Economics Studying?

Having spent all five of my years as a graduate student trying to get psychologists and economists to agree on basic ideas about decision-making, I think the following two pieces complement one another perfectly: Cosma Shalizi’s comments on rereading Blanchard and Fischer’s “Lectures on Macroeconomics”: Blanchard and Fischer is about “modern” macro, models based on […]

A Cheap Criticism of p-Values

One of these days I am going to finish my series on problems with how NHST is issued in the social sciences. Until then, I came up with a cheap criticism of p-values today. To make sense of my complaint, you’ll want to head over to Andy Gelman’s blog and read the comments on his […]

The State of Statistics in Julia

Updated 12.2.2012: Added sample output based on a suggestion from Stefan Karpinski. Introduction Over the last few weeks, the Julia core team has rolled out a demo version of Julia’s package management system. While the Julia package system is still very much in beta, it nevertheless provides the first plausible way for non-expert users to […]

The Shape of Floating Point Random Numbers

The Shape of Floating Point Random Numbers

[Updated 10/18/2012: Fixed a typo in which mantissa was replaced with exponent.] Over the weekend, Viral Shah updated Julia’s implementation of randn() to give a 20% speed boost. Because we all wanted to test that this speed-up had not come at the expense of the validity of Julia’s RNG system, I spent some time this […]

Overfitting

Overfitting

What do you think when you see a model like the one below? Does this strike you as a good model? Or as a bad model? There’s no right or wrong answer to this question, but I’d like to argue that models that are able to match white noise are typically bad things, especially when […]

EDA Before CDA

EDA Before CDA

One Paragraph Summary Always explore your data visually. Whatever specific hypothesis you have when you go out to collect data is likely to be worse than any of the hypotheses you’ll form after looking at just a few simple visualizations of that data. The most effective hypothesis testing framework in existence is the test of […]

Finder Bug in OS X

Finder Bug in OS X

Four years after I first noticed it, Finder still has a bug in it that causes it to report a negative number of items waiting for deletion:

Playing with The Circular Law in Julia

Playing with The Circular Law in Julia

Introduction Statistically-trained readers of this blog will be very familiar with the Central Limit Theorem, which describes the asymptotic sampling distribution of the mean of a random vector composed of IID variables. Some of the most interesting recent work in mathematics has been focused on the development of increasingly powerful proofs of a similar law, […]

Will Data Scientists Be Replaced by Tools?

The Quick-and-Dirty Summary I was recently asked to participate in a proposed SXSW panel that will debate the question, “Will Data Scientists Be Replaced by Tools?” This post describes my current thinking on that question as a way of (1) convincing you to go vote for the panel’s inclusion in this year’s SXSW and (2) […]