The graphs below make clear — some moreso than others — that January is a distinctive month, because the mean temperature noticeably lags behind the mean length of the day. You can also see a similar pattern for the temperatures in July, which are warmer than June’s temperatures even though the days are already getting shorter. Both of these imply that it is not the mere presence of sunlight that determines the temperature each day, but rather the accumulated warmth due to the sunlight of the previous month.

To be clear about the construction of the graphs and their interpretation, the rank of a data point is its relative position in the order of the entire data set. The warmest day in a year is ranked 365 and the coldest day is ranked 1; similarly, the longest day in a year is ranked 365 and the shortest day is ranked 1. Day 1 is January 1st; day 365 is December 31st.

It will probably take me another few weeks before I can find time to prove this mathematically.

For me, this observation highlights the absurdity of the naïve apotheosis of populism and democratic institutions that constitutes a core element of the contemporary Western zeitgeist. We tend to take for granted that democracy is something intrinsically good, an assumption that gives strength to the growing scorn we see in our society for “elitism” or any other movement that threatens to usurp the will of the people. We Americans seem to invariably forget that, while democratic institutions may sculpt our society in accord with the will of the people, this in no way implies that the people’s vision of a perfect society is something we should wish to see given form. Democracy does indeed give power to the people, but it does not and can never give the people the moral integrity to put that power to proper use.

Indeed, if the age-old adage that “power corrupts” is true, then democracy might even contribute to the moral and intellectual degradation of the populations of democratic nations. Or, as seems more likely, the age-old adage is simply wrong: corruption is a part of the human inheritance, and power, like alcohol, simply brings that latent vice to the forefront.

Before I close, I should note that this is not a peculiarly American problem, though I know many people who would like to claim so. After all, the Swiss are about to vote on a law that would permanently ban the construction of minarets.

Really, when I think of all the crimes that democratic nations commit against their own moral codes, it’s enough to make me wonder if William Henry Vanderbilt was onto something when he said, “the people be damned.”

As you can see from this graph, the two approaches seem to have indistinguishable performance for large data sets, but the clustered data set approach seems to perform slightly better for data sets of size N = 10. I was quite surprised by this, as I assumed my slightly ad hoc approach would perform worse than the standard approach. I therefore would appreciate any/all of the following: theoretical analyses of the two approaches’ merits for small data sets, the discovery of errors in my code, or an insight about the R function rnorm() that implies these results regardless of the intrinsic quality of the two approaches. One other conceivable source of error in these results — which I had hoped would wish out with the large number of iterations in my simulations — is that the data sets used to perform the analysis were simply incomparable, which is problematic because I compared classical regression on scattered point data sets to slope estimation on clustered point data sets. As a follow up, I should probably consider the performance of classical regression on clustered point data sets relative to scattered point data sets, though this approach would not itself answer the question of which data collection strategy is best.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# Compare performance of scattered point regression with slope estimation.
 
# See whether one algorithm does better with certain size data sets.
 
# Compare difference in performance on 100,000 samples of four sizes:
# 10, 100, 1000, 10000.
sample.sizes = c(10, 100, 1000, 10000);
classical = c();
dual.point = c();
 
for (i in 1:length(sample.sizes))
{
  slopes = c();
  alt.slopes = c();
 
  errors = c();
  alt.errors = c();
 
  # Classical regression.
  for (iteration in 1:100000)
  {
    x = 1:sample.sizes[i];
    y = 2 * x + rnorm(sample.sizes[i]);
    slopes[iteration] = coef(lm(y ~ x))[2];
    errors[iteration] = abs(slopes[iteration] - 2);
  }
 
  # Dual point slope estimatation.
  for (iteration in 1:100000)
  {
    x = c(rep(1, sample.sizes[i] / 2), rep(sample.sizes[i], sample.sizes[i] / 2));
    y = 2 * x + rnorm(sample.sizes[i]);
    alt.slopes[iteration] = (mean(y[(sample.sizes[i] / 2 + 1):sample.sizes[i]]) - mean(y[1:(sample.sizes[i] / 2)])) / (sample.sizes[i] - 1);
    alt.errors[iteration] = abs(alt.slopes[iteration] - 2);
  }
 
  classical[i] = mean(errors);
  dual.point[i] = mean(alt.errors);
}

Is this a reasonable approach to two variable linear regressions? Is this approach less efficient statistically than sampling at many points? Or is the reason to avoid this strategy in practice is that one is uncertain of the validity of the linearity assumption in all but exceptional cases?

1. Adium: The best chat client for the Mac that I’m aware of. I use it as a client for GMail chat, AIM and MSN. I’d use iChat if it worked with all of those services as well as Adium does, but it just doesn’t as far as I can tell. There are things that iChat does that Adium can’t do — e.g. video chat –, but I don’t have any use for those features. I could also use the AIM and MSN programs provided by AOL and Microsoft, but I much prefer a single integrated program over several separate programs. (site)

2. Caffeine: A simple little program that keeps your Mac from going to sleep, turning off the monitor or activating the screen saver. Very useful when you’re giving presentations. (site)

3. Carbon Emacs: The only build of Emacs that I find reliably renders the keys on my Western Spanish keyboard. It is also the only one that seems to respect traditional Emacs key bindings, which is very important to me. (site)

4. Cubase: My favorite music composition software for the Mac. Cubase is well-deservedly famous as a MIDI sequencer and I’ve found that it’s equally good as a multitrack audio recording system. I use it along with a Toneport UX2 to record guitar and Superior Drummer 2.0 for drum tracking, and I’ve gotten great results so far. (site)

5. Cyberduck: The FTP/SFTP client I use. I’m sure there are better tools than Cyberduck (such as Transmit 3), but Cyberduck is free and does the job more than well enough for my needs. (site)

6. Delicious Library: A program to help you keep a record of all of the books, DVD’s and CD’s you own. I find it especially useful for keeping track of the books I lend to people. (site)

7. Flickr Uploader: If I’m going to upload a lot of photos to Flickr, I really don’t want to have to use a Web interface. Flickr Uploader lets me do all of the editing on my machine and then send the labelled and tagged photos as a single group to Flickr. Most importantly, the progress I’ve made in tagging photos isn’t lost when my Internet connection flakes out. (site)

8. Gimp: I’m too cheap to buy Photoshop, but the recent builds of Gimp for the Mac work well enough for my purposes. (site)

9. Graphviz: If I have to draw any sort of graph, I always use Graphviz. It’s a great interface to compilers for the DOT language developed at Bell Labs to describe graphs. If you can program at all and ever need to write up flowcharts or diagrams of any sort, I think Graphviz is the way to go. You should also know that the Pixelglow build for Macs is much better than the default. (site)

10. Growl: Growl provides one of those clever little hacks to the basic Mac user interface that Windows users always find impressive: it creates a service for displaying notifications on your screen that quickly fade away after you’ve seen them. But the truth is that Growl’s usefulness is only obvious after you’ve used it for a while. (site)

11. Handbrake: The best video transcoder I know of for the Mac. Whenever I need to change one video format into another, Handbrake’s been able to do it for me. (site)

12. Hazel: Another great service for Macs: install Hazel and you have a simple daemon that will regularly move files according to a set of rules you define yourself. I use it to sort every file on my desktop into folders specific to filetypes — moving MP3’s to one folder and PDF’s to another. It’s been a major part of my efforts to be more organized with my files. (site)

13. KeePassX: A password manager that I find very helpful for navigating the mass of passwords I need to remember without leaving the passwords as plain text anywhere on my system. (site)

14. MacFreePOPS: A simple little program that will let you access your Hotmail account from Mail as if it were a POP server. Extremely useful. (site)

15. MacFUSE: Probably the most impressive of all of the hacks created by the Mac user community. MacFUSE allows you to install new file system drivers that run entirely in user space. The result is that you’ll get easy access to NTFS (i.e. Windows) hard drives and a slew of other formats. I think everyone should put MacFUSE on their machine the day they buy it. (site)

16. Mac The Ripper: If I need to make a copy of a DVD I’ve made, Mac The Ripper makes it much easier for me to do so. Unfortunately only the older version is still freely distributed, but it works for most DVD’s. (site)

17. MarsEdit: Just as I don’t like using a web interface to upload photos to Flickr, I don’t much like using one when writing blog posts. So I do all of my writing in MarsEdit, which then handles uploading my finished posts to my server. (site)

18. Mathematica: I use Mathematica fairly frequently when I want to get a quick sense of how functions behave or when I need to evaluate an integral I’ve forgotten how to solve by hand. (site)

19. Matlab: If I need to do a lot of basic number crunching involving matrices, I always use Matlab. Additionally, I tend to use it along with PsychToolBox and DotsX for coding experiments in neuroscience and psychology. (site)

20. MySQL: I always use MySQL as the database system for every dynamic web site I build. It works perfectly on Mac OS X these days, so I tend to demo things on my own machine before moving them off to a stand-alone server. (site)

21. NetNewsWire: My favorite RSS reader for the Mac. I can’t speak highly enough of NetNewsWire’s interface or the fact that the iPhone application is just as great as the desktop version. (site)

22. OpenOffice: Again, I’m too cheap to buy a copy of Office, so I use OpenOffice. It’s managed to serve me pretty well so far. It’s still a little lacking on the Mac, but it’s getting much better with time. (site)

23. Papers: My means for storing and organizing all of my PDF files. Think of it as iTunes for PDF’s. If you read journal articles, Papers will improve your life more than you could possibly expect. (site)

24. Perian: Perian will outfit your Quicktime player with almost all of the codecs you could want. Without it, I find Quicktime almost useless. (site)

25. Perl: The classic programming language needs no introduction, but I think it’s worth noting that you’re always better building your own version of Perl and storing it in /usr/local/, where you won’t be able to destroy the version that OS X ships with by default. I’ve also found it nearly impossible to get many modules to build without some customization. (site)

26. Python: Again, I don’t think Python needs an introduction, but building your own copy seems like a very good idea to me. (site)

27. Quicksilver: A great tool for getting easy access to programs. I don’t use nearly as many of Quicksilver’s features as a lot of people do, but I find it really helpful to be able to avoid using Finder when I don’t need to. (site)

28. R: My language of choice for statistical computing and data analysis. Great tools for producing graphs and an amazing set of facilities for any statistical computation you could ever want to perform. If you want to do statistics like a grown up statistician would, R is the way to go. (site)

29. ReadIris: My favorite OCR software for the Mac. I use this every time I want to copy a long section of text I’ve scanned. I invariably have to make corrections by hand, but that’s much faster than typing everything myself from scratch. Given how well ReadIris performs for me, I have high hope that one day in my life we’ll see a properly Bayesian piece of OCR software that gets everything right. (site)

30. Ruby: Another programming language that needs no introduction, but which I’d recommend building from source and storing in /usr/local. (site)

31. ScreenFlow: The best screencasting software for the Mac I could find. Given the number of features, the quality of the interface and its relatively low cost, I doubt one could find something better for a few years to come. (site)

32. Scrivener: An amazing application that makes writing extended works (for me those are mostly translations) much, much easier. I don’t use it as often these days, but Scrivener is a brilliant tool if you do a lot of writing that can be broken into sections and outlined carefully. (site)

33. ScummVM: When I’m not working, I like to play some old LucasArts games. ScummVM makes that possible. (site)

34. Senuti: If I need to transfer a file off of an iPod (which iTunes makes impossible), Senuti is there for me. It was free for a long time, so I’m somewhat surprised to find that you’re supposed to pay for it now. (site)

35. Sequel Pro: My favorite database client system. The heir to the great CocoaMySQL application. A perfect compliment to MySQL on the Mac. (site)

36. Skype: Who doesn’t use Skype as their VoIP program? (site)

37. TexLive: When I want documents to look clean, I always use LaTeX. TexLive is the current standard distribution of LaTeX for UNIX systems and it has some great tools specifically made for the Mac. (site)

38. TextMate: They claim it, and I agree: TextMate is Emacs for the 21st century. If you are young enough that you find GUI’s helpful and don’t think touching the mouse is a crime against nature, TextMate is the best text editor you will ever find. Every Rails person worth his salt is a TextMate user and there is an endless supply of bundles to customize TextMate for the language of your choice. (I’ve recently used it a lot with Matlab, R and Erlang.) (site)

39. The Unarchiver: If you’ve ever received a compressed file you couldn’t open, get The Unarchiver and your problems will be solved. Everything else is a waste of time and/or money. (site)

40. Twitterific: My favorite Twitter client for the Mac. (site)

41. Unison: Probably my single favorite tool for the Mac. Unison lets my keep all of the files that matter to me in perfect sync between my laptop and my desktop. In practice, that amounts to a brilliant back-up system as well as making my life incredibly easier when I do some work on my laptop and then some more work on my desktop. In the end, Unison seems to be the program destined to replace rsync one day. (site)

42. VLC: The system I always use to watch anything that doesn’t open in Quicktime with Perian installed. (site)

43. VMWare Fusion: Sometimes I need to run Windows or Linux. VMWare Fusion makes it incredibly easy to do so and runs both of those operating systems with remarkable efficiency. (site)

44. Zenmap: If I need to figure out the structure of the network I’m on, nmap is the tool for doing so. Zenmap provides nmap for the Mac and also a (sometimes) helpful GUI. (site)

Rothman computed a p-value function — a continuous function of p-values mapped against a range of effect sizes. The range of effect sizes was here again measured by the relative risk ratio and includes both beneficial and nonbeneficial effects. He shows that another hypothesis, a fantastically beneficial risk ratio, RR = 4.1, shares the same p-value, .14, as the null, RR = 1.0 (2002, 125). This is common in medicine and all the sciences. To think that p-values have a 1-to-1 correspondence with a unique risk ratio is to ignore the symmetry of the p-function.¹

I wish the symmetry of the distributions used for testing significance, especially the t distribution, would be emphasized to students during their introduction to statistics. We generally test distributions so that the t-value comparison is strongly positive to see whether we can reject the null hypothesis of zero difference between the means for some two sets of observations. But it is always possible to test another null hypothesis, in which the difference between the means for the two groups is much larger than the difference we observed, that we will also always fail to reject every time that we fail to reject the primary null hypothesis of zero difference. Yet we never test this hypothesis — despite their being no good reason for this mathematically. The only justification is an implicit Bayesian prior in defense of the null hypothesis rather than its dopplegänger hypothesis in which the difference is much larger than we have seen in practice. Is this implicit underweighting of the alternative null hypothesis really sound? That is an empirical question that is, unfortunately, not likely to be answered soon, but it suggests that conventional statistical practice may consistently underestimate the effects being examined using significance testing.

Of course, this problem is itself tied to the erroneous conflation of a failure to reject the null hypothesis with its acceptance — with the result that statistically insignificant effects are treated as non-existent, rather than inconclusively determined by the data at hand. Statistically insignificant differences tend not to be tested empirically a second time, so that it is hard to know how often they are really larger than our first experiments suggested.

1. Stephen T. Ziliak and Deirdre N. McCloskey : The Cult of Statistical Significance : On Drugs, Disability and Death

To quantify the pattern that you can see in the chart, only 4% of those who’ve tried cocaine have not also tried cigarettes at some point in their lives. In contrast, 49% of those who’ve tried cigarettes have never tried cocaine. In general, people are unlikely to try cocaine, but those who do are almost certain to have tried cigarettes as well. In other words, cocaine use tells you a lot about cigarette use, but cigarette use tells you effectively nothing about cocaine use. If you meet someone who’s tried cocaine, and you assume that they’ve also tried cigarettes, these statistics suggest that your assumption will be wrong less than 5% of the time.

The variables used in this analysis were ABUSEALC, ABUSECOC, ABUSEHAL, ABUSEHER, ABUSEINH, ABUSEMRJ, ABUSEANL, ABUSESED, ABUSESTM, ABUSETRN. The meanings of these variables are somewhat obscure, but my hope is that the definition of abuse is similar enough across substances to allow for a relative frequency analysis. Every subject classified as abusing a substance was summed over and then the resulting number was divided by the total number of subjects in the data set to find a frequency of abuse per substance.

Example 1. In cross-national comparisons, there is a striking correlation between the number of telephone lines per capita in a country and the death rate from breast cancer in that country. This is not because talking on the telephone causes cancer. Richer countries have more phones and higher cancer rates. The probable explanation for the excess cancer risk is that women in richer countries have fewer children. Pregnancy — especially early first pregnancy — is protective.¹

Is Freedman correct about the protective benefits of pregnancy? This would be remarkable if true. And, in the absence of evidence to the contrary, I am likely to believe Freedman’s claims.

1. David Freedman : Statistical Models: Theory and Practice : Chapter I

I discovered that it is much more effective to act like a nice guy and be “reasonable” if you prove willing to go beyond just verbiage. You can afford to be compassionate, lax, and courteous if, once in a while, when it is least expected of you, but completely justified, you sue someone, or savage an enemy, just to show that you can walk the walk.¹

1. Nassim Nicholas Taleb : The Black Swan: The Impact of the Highly Improbable : Preface

In our days, serious arguments have been made from data. Beautiful, delicate theorems have been proved; although the connection with data analysis often remains to be established. And an enormous amount of fiction has been produced, masquerading as rigorous science.¹

I would like to believe that, if only more statisticians wrote like David Freedman, we might succeed in ridding ourselves of so much of the fashionable nonsense that masquerades as science today.

Hat tip to Jiaying Zhao for bringing this truly amazing article to my attention.

1. David Freedman : Foundations of Science : Some Issues in the Foundation of Statistics

As the campaigns progressed, a surprising number of conservatives switched their support to Obama. Thoughtful conservatives, already disturbed by the accumulation of blunders of the current Administration (the Iraq WMD, Katrina, the Justice Department scandals), culminating in its uncertain response to the financial crisis, were appalled at the iconic status that Joe the Plumber attained in the Republican campaign, the wild rumors spread by the conservative bloggers and talk-radio hosts, and the intellectual vacuity of many Republican candidates and advocates. The Republican Party seemed to have descended to anti-intellectualism–to deriding highly educated people who speak in complete sentences as “elitists,” as compared to the down-to-the-earth ignorance of Joe and his ilk–which sorts badly with the strong intellectual tradition of conservatism. It is a self-defeating strategy of conservatives to argue that “all” intellectuals are liberal and therefore conservatives should think with their guts rather than their brains.

For myself, I would be happy to see conservatism exit from the political scene–provided it takes liberalism with it. I would like to see us enter a post-ideological era in which policies are based on pragmatic considerations rather than on conformity to a set of preconceptions rooted in a rapidly vanishing past. We have accumulated a substantial history of liberal and conservative failures. The liberal failures include underestimating the cost of egalitarianism and of social engineering by judges (the Warren Court, Roe v. Wade, the near abolition of capital punishment), and the benefits of discipline, of punishment, of enforcing principles of personal responsibility, and of military force. The conservative failures include overestimating the efficiency of unregulated markets, the efficacy of military force, and the beneficent effects of religiosity. Liberals are wrong to promote unions (described by one wag, albeit with some exaggeration, as the parasites that kill their hosts) and conservatives to promote abstinence as a substitute for condoms in preventing teenage pregnancy.¹

1. Richard Posner : The Becker-Posner Blog : The Future of Conservatism — Posner

I was fascinated by the claim, albeit rather incredulous. But, rather than pursue the question of tryptophan’s effects on suicide, I decided to look into a question I’ve often wondered about: the correlation of GDP and suicide rates.

After some data diving of my own, using GDP data from the IMF and suicide data from WHO, I found no meaningful correlation between suicide rates and GDP. Interestingly, a simple scatterplot of the relevant data sets reveals that, for each gender separately, there are several very substantial outliers that make any such correlation impossible to find, as you can see below.

So the question I’m left with is, “what variables explain the very different suicide rates seen across nations in this data set?”