For COURSE PACK and other PERMISSIONS, refer to entry on previous page. For more information, send e-mail to permissions@pupress.princeton.edu

Introduction to Part I of Lo & MacKinlay: A Non-Random Walk down Wall Street

The five chapters in this first part focus squarely on whether the Random Walk Hypothesis is a plausible description of recent US stock market prices. At the time we started our investigations--in 1985, just a year after we arrived at the Wharton School--the Random Walk Hypothesis was taken for granted as gospel truth. A number of well-known empirical studies had long since established the fact that markets were "weak-form efficient" in Roberts's (1967) terminology, implying that past prices could not be used to forecast future prices changes (see, for example, Cowles and Jones (1973), Kendall (1953), Osborne (1959, 1962), Roberts (1959, 1967), Larson (1960), Cowles (1960), Working (1960), Alexander (1961, 1964), Granger and Morgenstern (1963), Mandelbrot (1963), Fama (1965), and Fama and Blume (1966)). And although some of these studies did find evidence against the random walk, e.g., Cowles and Jones (1973), they were largely dismissed as statistical anomalies or not economically meaningful after accounting for transactions costs, e.g., Cowles (1960). For example, after conducting an extensive empirical analysis of the `runs' of US stock returns from 1956 to 1962, Fama (1965) concludes that, ". . . there is no evidence of important dependence from either an investment or a statistical point of view."

It was in this milieu that we decided to revisit the Random Walk Hypothesis. Previous studies had been unable to reject the random walk, hence we surmised that perhaps a more sensitive statistical test was needed, one capable of detecting small but significant departures from pure randomness. In the jargon of statistical inference, we hoped to develop a more "powerful" test, a test that has a higher probability of rejecting the Random Walk Hypothesis if it is indeed false. Motivated partly by an insight of Merton's (1980), that variances can be estimated more accurately than means when data is sampled at finer intervals, we proposed a test of the random walk based on a comparison of variances at different sampling intervals. And by casting the comparison as a Hausman (1978) specification test, we were able to obtain an asymptotic sampling theory for the variance ratio statistic almost immediately, which we later generalized and extended in many ways. These results and their empirical implementation are described in Chapter 2.

In retrospect, our motivation for the variance ratio test was completely unnecessary.

Although Merton's (1980) observation holds quite generally, the overwhelming rejections of the Random Walk Hypothesis that we obtained for weekly US stock returns from 1962 to 1985 implied that a more powerful test was not needed--the random walk could have been rejected on the basis of the simple first-order autocorrelation coefficient, which we estimated to be 30 percent for the equal-weighted weekly returns index! We were taken completely by surprise (and carefully re-checked our programs several times for coding errors before debuting these results in a November 1986 conference). How could such compelling evidence against the random walk be overlooked by the vast literature we were fed as graduate students?

At first, we attributed this to our using weekly returns--prior studies used either daily or monthly. We chose a weekly sampling interval to balance the desire for a large sample size against the problems associated with high-frequency financial data, e.g., nonsynchronous prices, bid/ask "bounce," etc. But we soon discovered that the case against the random walk was equally compelling with daily returns.

This puzzling state of affairs sparked the series of studies contained in Chapters 3 to 6, studies that attempted to reconcile what we, and many others, viewed as a sharp contradiction between our statistical inferences and the voluminous literature that came before us. We checked the accuracy of our statistical methods (Chapter 3), we quantified the potential biases introduced by nonsynchronous prices (Chapter 4), we investigated the sources of the rejections of the random walk and traced them to large positive cross-autocorrelations and lead/lag effects (Chapter 5), and we considered statistical fractals as an alternative to the random walk (Chapter 6). Despite our best efforts, we were unable to explain away the evidence against the Random Walk Hypothesis.

With the benefit of hindsight and a more thorough review of the literature, we have come to the conclusion that the apparent inconsistency between the broad support for the Random Walk Hypothesis and our empirical findings is largely due to the common misconception that the Random Walk Hypothesis is equivalent to the Efficient Markets Hypothesis, and the near religious devotion of economists to the latter (see Chapter 1). Once we saw that we, and our colleagues, had been trained to study the data through the filtered lenses of classical market efficiency, it became clear that the problem lay not with our empirical analysis, but with the economic implications that others incorrected attributed to our results--unbounded profit opportunities, irrational investors, and the like.

We also discovered that ours was not the first study to reject the random walk, and that the departures from the random walk uncovered by Osborne (1962), Larson (1960), Cootner (1962), Steiger (1964), Niederhoffer and Osborne (1966), and Schwartz and Whitcomb (1977), to name just a few examples, were largely ignored by the academic community and unknown to us until after our own papers were published.³ We were all in a collective fog regarding the validity of the Random Walk Hypothesis, but as we confronted the empirical evidence from every angle and began to rule out other explanations, slowly the fog lifted for us.

In Niederhoffer's (1997) entertaining and irreverent autobiography, he sheds some light on the kind of forces at work in creating this fog. In describing the Random Walk Hypothesis as it developed at the University of Chicago in the 1960's, he writes:

This theory and the attitude of its adherents found classic expression in one incident I personally observed that deserves memorialization. A team of four of the most respected graduate students in finance had joined forces with two professors, now considered venerable enough to have won or to have been considered for a Nobel prize, but at that time feisty as Hades and insecure as a kid on his first date. This elite group was studying the possible impact of volume on stock price movements, a subject I had researched. As I was coming down the steps from the library on the third floor of Haskell Hall, the main business building, I could see this Group of Six gathered together on a stairway landing, examining some computer output. Their voices wafted up to me, echoing off the stone walls of the building. One of the students was pointing to some output while querying the professors, "Well, what if we really do find something? We'll be up the creek. It won't be consistent with the random walk model." The younger professor replied, "Don't worry, we'll cross that bridge in the unlikely event we come to it."

I could hardly believe my ears--here were six scientists openly hoping to find no departures from ignorance. I couldn't hold my tongue, and blurted out, "I sure am glad you are all keeping an open mind about your research." I could hardly refrain from grinning as I walked past them. I heard muttered imprecations in response.

But beyond the interesting implications that this cognitive dissonance provides for the sociology of science, we think there is an even more important insight to be gleaned from all of this. In a recent update of our original variance ratio test for weekly US stock market indexes, we discovered that the most current data (1986-1996) conforms more closely to the random walk than our original 1962-1985 sample period. Moreover, upon further investigation, we learned that over the past decade several investment firms--most notably, Morgan Stanley and D.E. Shaw--have been engaged in high-frequency equity trading strategies specifically designed to take advantage of the kind of patterns we uncovered in 1988. Previously known as "pairs trading" and now called "statistical arbitrage," these strategies have fared reasonably well until recently, and are now regarded as a very competitive and thin-margin business because of the proliferation of hedge funds engaged in these activities. This provides a plausible explanation for the trend towards randomness in the recent data, one that harkens back to Samuelson's "Proof that Properly Anticipated Prices Fluctuate Randomly."

But if Morgan Stanley and D.E. Shaw were profiting in the 1980's from the predictability in stock returns that is now waning because of competition, can we conclude that markets were inefficient in the 1980's? Not without additional information about the cost and risk of their trading operations, and the novelty of their trading strategies relative to their competitors'.

In particular, the profits earned by the early statistical arbitrageurs may be viewed as economic rents that accrued to their innovation, creativity, perseverance, and appetite for risk. Now that others have begun to reverse engineer and mimick their technologies, profit margins are declining. Therefore, neither the evidence against the random walk, nor the more recent trend towards the random walk, are inconsistent with the practical version of the Efficient Markets Hypothesis. Market opportunities need not be market inefficiencies.

3 In fact, both Alexander (1961) and Schwartz and Whitcomb (1977) use variance ratios to test the Random Walk Hypothesis, and although they do not employ the kind of rigorous statistical inference that we derived in our study, nevertheless it was our mistake to have overlooked their contributions. Our only defense is that none of our colleagues were aware of these studies either, for no one pointed out these references to us either before or after our papers were published.