The End is Nigh-ish
I don’t really know what to make of blogging, so I would like to try my hand at a Very Big Topic: Global Warming. Next week I’ll discuss my relationship with God (no, I am not his Son; but he’s my uncle).
Actually I don’t know much about Global Warming, except that it’s real, it’s largely man-made, and has uncertain but potentially disastrous consequences. It’s not much, but I suspect it’s not substantially less than what a lot of people know, including those who feel the urge to opine on the issue (I just added myself to this unsavory group).
It is becoming increasingly clear to me is that Global Warming is almost unique in that it lies at the busy intersection of Science, Engineering and Policy. I believe that the only other area that shares this feature is Bioengineering. But whereas Climate Change has no potential upside, Bioengineering can be very bad (as many zombie movies remind us) and good. Moreover it has no inertia, unlike Climate Change.
Aside from the intrinsic challenges of Global Warming, the interdisciplinary side of Global Warming is a recipe for disaster. Absolutely no one is able to read first-hand sources of knowledge from all three areas. What is instead happening is that everyone is feeding on secondary or even tertiary sources. This applies especially to economists, who are neither real policy-makers nor scientists, and should I mention engineers? So we have every columnist and prominent blogger playing modern jackass.

This coach was designed by a physicist, to specifications provided by a policy-maker, to be driven by an engineer.
Even if we ignore the “everythingologists” [1] like Krugman or Cowen, the competence attribution problem is serious. Climatologists and most Physicists for example make poor engineers, and yet they are proposing or even outlining engineering solutions on what is the most complex engineering challenge we have ever faced, and the most expensive by at least two orders of magnitude.
What is a poor man to do? For once, I am a Cartesian, in that I’d rather
[...]conquer myself rather than fortune, and change my desires rather than the order of the world, and in general, accustom myself to the persuasion that, except our own thoughts, there is nothing absolutely in our power; so that when we have done our best in things external to us, all wherein we fail of success is to be held, as regards us, absolutely impossible[...]
So I have no hope to effect change. The rational thing to do would to have lots of sex but no children while waiting for the apocalypse. I am afraid I already screwed up on that one. For those who haven’t, don’t waste your time reading the self-righteous tirades of Krugman or the cute-clever-contrarian-positivistic proposals by Dubner and Levitt.
My only recourse is a different hope: that the impact of Global Warming is felt on a timescale longer than that of policy changes. That’s a folk lesson from Control Theory. But don’t hold your breath.
[1] “everythingologist” is an imperfect translation of the italian “tuttologo”, and my tentative contribution to the Oxford English Dictionary.
Memories of an Enron Summer
I recently watched “The Smartest Guys in the Room”. It is a documentary on the spectacular demise of Enron in 2001. I am not a neutral observer. In 2000, I was close to defending my thesis at Stanford. My department had ties to Enron’s research department. I knew the literature on bandwidth pricing and a fair amount of mathematical finance, so I was considered a good fit to be a summer intern there. I accepted, and away I went.
Enron at the time was the place to be. It had been named the “most innovative company” by Fortune four years in a row. They seemed to have invented a new business model: create markets everywhere and become the dominant market maker. First, gas. Then, electric power. Then, weather. Then, bandwidth. Wait, there’s more: water, pulp paper, TV advertising slots, freight container space, RAM. Following the bandwidth announcement, the stock went from $40 to $70. In the process, Enron had dropped “energy” from his slogan “The World’s leading energy company”. Every other energy company was playing catch-up. Every MBA was applying. Enron was the New Economy rejuvenating the Old Economy. Fortune had also named Enron one of the best 25 companies to work for. All of this, and the money you could make if you performed well. The place was so incredibly cool that even Paul Krugman was consulting for Enron at the time (for a mere $50K)! This is slightly less likely than Ratzinger consulting for Hustler magazine.
The headquarters on 1400 Smith Ave looked like a deodorant stick in scale 2000/1, wrapped in aluminum foil. On the first day I was sent me to the 50th floor, but before meeting with Ken Lay I was escorted out of the building because they had not received my drug test. Auspicious start: alleged cokehead on the first day at work. It turns out, they had received it and misplaced it. I was dispatched to the trading floor of the bandwidth division to help the traders with valuation of bandwidth option contracts, and planning models for optimal purchase of dark fiber. Exiting the elevators, a highly customized version of a Confederate Hellcat would welcome you. If you looked closer, you could read “Bandwidth Hog” on the tank. I wonder where is it now.
The traders were all in their twenties or early thirties; the originators were older (as usual, salespeople are less cool than traders). Jim Fallon (not the SNL comedian) was the head of bandwidth trading. A Columbia MBA, he was personable, receptive, fast, and very honest in his opinions. For example, in July 2000 I asked him how much time did we have to become profitable. He said “18 months, or we are done”. In December 2001 Enron went bankrupt. That’s a pretty good prediction in my book. Throughout my internship, I spoke quite often to Fallon and to the head of Research. I countered with numbers the projection on bandwidth demand that Nortel presented to us. The size of the bandwidth market that was assessed by McKinsey was also 10 times bigger than what we could come up with internally. Fallon understood everything. He didn’t buy as many dark fiber baskets as before. The impression I received was that at Enron, if you had an idea and could support it with facts, you could even have the ear of the CEO. It’s an exhilarating feeling in a company with 17,000 employees.
The mood on the 44th floor was always upbeat. The traders were nice with researchers (all two of us), but clearly viewed us as “glorified admins”, minus the breasts. They were all MBAs from good schools, with a slight prevalence of Chicago U. They cursed a lot, and I, a non-native speaker groomed in hyper-polite Bay Area, learned a few new expressions, like I’ll rip you a new asshole, which by now (2008) is taught to kindergarteners as a polite way of saying “I don’t really like you”. Occasionally the traders engaged in bad taste: one evening I witnessed a “spit fight” between two traders. A trader had naked women as a desktop background, and sent porn pictures by email. He was reprimanded. Yet, in a place like IBM he would have been fired for merely thinking about sex during office hours (IBM’s motto is indeed “Think [but not about sex]“).
Another striking feature of the place was a certain self-deprecating buffoonery. There were monitors in the elevators, and they were showing news, but also humorous skits of top executives, like Joe Sutton. On Monday mornings, we’d find a message by Jeff Skilling in our answering machine, telling us that we were good but that something had to improved. Skilling was also visiting often the trading floor to talk to Fallon. I have no doubt that he had an accurate picture of the status of the division, and of other divisions as well. When I spoke to him, he was very friendly.
Two other things were relevant to me. First, almost nobody had a beard. I noticed because I have one, albeit small. There were two other guys with a beard, and they were obvious losers. That caused some costernation, since I am attached to my beard almost as much as to my reproductive organs (you could object that a beard can grow back).
Second, everyone was almost comically republican. The tone was a mix of Kudlow and O’Reilly. Even Tucker Carlson would have sounded like an incorrigible intellectual. There were a few closet democrats, but they did not speak out, and I could extract confessions only in private. It is an uncontroversial fact that energy deregulation in the UK and parts of the US has failed: prices are both higher and more volatile than before the change. Making this offhand remark at Enron would have been a fireable offense. Although I am in favor of decentralized decision making and well-thought-out deregulation, four months of unconditional, self-righteous pro-market jingoism almost made a marxist-leninist out of me.
Despite all this optimism, all was not good on the floor. For example, a trader covering the South West was buying and selling the same lit bandwidth in Florida with the same counterpart on a daily basis. Clearly, both had an incentive to show that their volumes were increasing. Another example was the much-touted deal with Sun Microsystems. In January Enron announced a “partnership” with Sun: they’d buy $100M worth of servers, and would provide Sun with bandwidth on demand. In August, we had $100M of servers languishing in a storage in Shepherd St., because Enron was a Windows, and SUN didn’t have any use for our bandwidth, since it was essentially useless. They ended up buying one or two millions worth of infrastructure. When a Chief Risk Officer for the Bandwidth division was named in the summer of 2000, he tried to locate that contract and find out who signed it. For all his efforts, he couldn’t. Nobody knew who had signed the contract. For $100M. In a financial company, whose job is to trade contracts. Not good.
I couldn’t understand how Enron made money. Surely, not in bandwidth. Our circle of friends had people from every division. The emerging markets (e.g., pulp paper) were losing money. Enron Energy Services, headed by Lou Pai, had just announced they were profitable, but nobody believed it. Everybody knew that revenues could be manipulated with mark-to-market accounting. Electric power trading was so-so. All was left was gas trading, for a company with a $60B market cap. What was so striking is that nobody was hiding this. Everybody worked hard, was enthusiastic, and admitted that their side of the business wasn’t profitable. What Enron did for me was to prove that very good people can make very bad decisions not because they are instructed to do so, but because of short-range interactions. In this respect, I disagree with “The Smartest Guys in the Room”. They explain the failure using the Milgram experiment, which shows how a person can overcome his reservations when he is being instructed by a superior (Skilling, according to the movie). Instead, to me Enron’s failure is better explained by the Lucifer Effect. It was self-organizing bad behavior.
Sometime in August, Ken Rice, the CEO of the bandwidth division, announced a deal with Blockbuster to deliver video on demand, “worth up to one billion”. Employees were ecstatic. Wall Street was ecstatic. The stock went up 5-6 dollars, up to $86, and then, on a single trade, it reached $90 at the end of the day. Traders on the 44th floor asked half-seriously “should we short the stock?”.
Unlike most people on this planet, I love Texans, who remind me a lot of Italians, even though I am not sure it it’s a compliment. I loved Houston, its museums, its restaurants. I could tolerate humidity and cockroaches. But I am not the dumbest guy in the room. In April 2001 I received an offer. My then-girlfriend-now-wife had accepted a position in New York. That was reason enough to decline the offer.
R for Pedestrians
R is a language and programming environment primarily suited for data analysis. It has been the programming lingua franca of Statistics departments for over ten years, but in the past two years its user base and visibility have grown to the point that the language has been the subject of an article on the New York Times. More and more R is used in place of Matlab by professionals in the financial industry. I thought I would list a number of resources for people who have programming experience, or for R users who want to improve their command of the language. It is meant to be selective. These are just the books, websites, blogs, etc. that I visit regularly. If brevity if the soul of wit (or of lingerie, according to Dorothy Parker), I am a cross-dressing smartass.
Books/Course notes:
- R programming for those coming from other languages by John D.Cook
- Programming in R
- R for Programmers by Norm Matloff
- R fundamentals by Thomas Lumley
- Introduction to Data Technologies by Paul Murrell
- C-C++-R in Statistics, a Harvard course
- ggplot2, a graphic system created by Hadley Wickham
- Creating R Packages
- S Language Methods and Classes by John Chambers
- Object-Oriented Programming in S-Plus by Robert Gentleman
- S4 Objects, by Thomas Lumley
- Statistical programming with R, Part 3: Reusable and object-oriented programming by David Metz
- A brief history of S by Richard A.Becker
- Introduction to High-Performance Computing with R by Dirk Eddelbuettel
- …all the other presentations by Dirk
Blogs:
- Learning R
- Brendan O’Connors
- Revolution Computing
- Romain Francois
- Cerebral Mastication
- Dataspora
- The Endeavour
- Gelman’s blog
- i2pi
Social:
- R Help newsgroup
- R on Stack Overflow
- Journal of Statistical Software
- R Journal and R News
- Meetups: New York, San Francisco
- #rstats tag on Twitter
If you have suggestions for a great site I have omitted, please let me know.
Absorption probabilities and limit orders
Many trading systems employ an entry and exit decision. You may enter a long or short position in an instrument at a given price, and close it when the return exceeds a threshold value, or falls below a tolerable value. One natural question is: if you know the trend of the price process, can you estimate the probability of hitting the upper threshold first? This is not a purely academic question. Understanding the relationship between drift and hitting probability can provide interesting insight. Beside the qualitative aspects, this probability can be used for inferential purposes as well.
There are at least three approaches that come to mind. The first is by simulation. We set parameters for the model and estimate the hitting probabilities. For example, consider a very simple dynamic, which is a good approximation of local stock (and option) price behavior:

Diffusion with absorbing states
A simple R code to estimate absorption probabilities is the following:
absorb <- function(mu,sigma,upperBound,lowerBound,x0=0){
nSamples=1e3
x=x0+rnorm(nSamples,mean=mu,sd=sigma)
y <- c(0,cumsum(x))
ta <- min(which(y >= upperBound))
tb <- min(which(y <= lowerBound))
if (ta == Inf & tb == Inf){
return(absorb(mu,sigma,upperBound,lowerBound,x0=x[nSamples]))
} else {
return(sign(ta-tb))
}
}
prAbsorb <- function(mu,sigma,upperBound,lowerBound,nVariates=1e4){
y <- vector('numeric',nVariates)
for (n in 1:nVariates) {
y[n]=absorb(mu,sigma,upperBound,lowerBound)
}
v <- table(y)/nVariates
names(v) <- c(upperBound,lowerBound)
return(v)
}
prAbsorb(mu,sigma,upperBound,lowerBound)
Another approach is to model the process in discrete time as random walk with drift. Binomial trees are a popular technique for option pricing at least since Cox and Rubinstein introduced them in 1976. The price process takes discrete values, say $m$. It increases by one unit with probability , or decreases by one unit with probability
. The thresholds are set at
and
. This example is called the “gambler’s ruin” and is treated in most introductory books on Markov Processes.
How does one compute the probability of hitting the ceiling
starting from
? Assume the price process starts at
. Then, by conditioning on the next step move, we have
with the boundary conditions . Solving this recursively yields
where are the log-odds of the probabilities.
The continuous-time can be found as a limit to the discrete-time formula (taking the limit is rather cumbersome, for a human, but trivial for a CAS like Mathematica). The final result is identical, except that .
A more elegant approach would be to derive the formula above directly in a continuous time framework. Doing so requires a knowledge of stopping times and of the optional stopping theorem though, and I don’t know if anyone is really interested.
Finally, we can extend this result to the more realistic case where the stock price follows a geometric brownian motion.
Without loss of generality, we consider returns, i.e., we set , and we write the boundaries in exponential form:
and
. Let us consider
. The event
hits
is the same event as
hitting
. Now, we apply Ito’s lemma to find that
which is again a brownian motion with drift. We apply the formula for that process to write:
where .
The formula is not very intuitive. However, if you assume that , the formula simplifies to
which is a logistic function in .
Some thoughts about the Efficient Market Hypothesis
For over four decades, only tenured professors and the occasional trader cared about the Efficient Market Hypothesis. The financial crisis of 2008 changed all that. I had an animated discussion with my barber about the semi-strong form of the EMH the other day. I thought the subject was ripe for a blog post. Rather than dispensing with a cursory/supercilious/dismissive pronouncement of the EMH, I thought it appropriate to link to a few papers and web sites, to help the reader understand what the hypothesis is about. Everybody has heard the statement that “current prices reflect all available information”. Ok, but what does it mean, really? For an informal description, read Andrew Lo’s survey, or peruse the excellent www.e-m-h.org. For a more formal description, I am going to show a heuristic argument, which I read on on The Econometrics of Financial Markets by Lo et at. Consider a world of risk-neutral investors. The information available to the investor at time is
. In technical terms this is a filtration, and consists of finer and finer amount of information as
progresses. Let’s assume that a security yields a single dividend at some future epoch
, say
. This is modeled as a random variable, whose expectation can be computed with respect to each information set. The efficient market hypothesis states that the price at time
is the expected value of the dividend, given the information available at that time:
From this definition, it is simple to show that the price process is a martingale, i.e., the expected value of a future price is given by the current price:
We now revisit the assumptions in this model. The fact that the security pays a single dividend is not restrictive, since the equations are linear. The fact that the investors are risk-neutral is also not restrictive. If an investor is risk averse, the price of the security is lower than the expected value of the dividend. The price process is not a martingale, but a sub-martingale. The most stringent assumptions regard the ability by the investor to actually know the filtration, the associated probability measure, and the market-clearing mechanism that allows the investor to purchase the security at its fair value. Do they hold? For which investors? For which securities? How do we test these hypotheses? For simplicity, concentrate again on the single-security case. We cannot compute expectations, since we do not know the probability measure. We can record prices for all epochs and look at the ratio . If the investor is risk-neutral, we should see that the average of
over long periods is “approximately” one. If the investor is risk-averse, the sum of
has a drift term linear in $\T$ and a zero-mean term. The testing of the EMH is performed for multiple securities, but the gist is the same: if you find a security, or a synthetic security (i.e., a portfolio, even an actively managed one) for which $R_t$ is systematically and abnormally below or above one, then the EMH is violated. Incidentally, there is evidence that the hypothesis is violated; see Fama’s survey article on empirical tests. The most egregious and persistent violation is the momentum effect: investing in the top-performing stocks during the past 1-3 months systematically outperforms the market.
I can’t give here a self-contained description of EMH testing. It is not easy to formulate a meaningful testing protocol, and even if we did and the test rejected the EMH, it would not be clear what part of the EMH doesn’t hold.
I will take a shortcut and drive home three points:
- We can’t talk about testing the EMH if we don’t quantify and qualify the hypothesis very carefully;
- We cannot say that the EMH is violated simply because for one security, we observe, in one period, a very large deviation of $R_t$ from one;
- the subtleties of testing the EMH are very similar to those investors encounter when backtesting new strategies. Studying the literature on EMH tests can teach a lot about strategy development.
This makes me very skeptical of a good number of recent blog posts. For example, John Quiggin, Brad DeLong, Justin Fox (audio), and Paul Krugman (here and here) criticize the EMH without any empirical framework. For them, the CDS were junk, therefore the EMH doesn’t hold. Is that it? To be honest, Fox admits that he is a chronicler who doesn’t really know what he’s talking about, so I am willing to cut him some slack. Chris Dillow and Scott Sumner are proud defenders of the EMH, but they seem to be interested in pursuing some agenda as well. To the DeLongs and Sumners of this world I want to say, what’s the matter with you people? Can’t you just stick to the subject matter and stop analogizing about the wisdom of crowds? Of all the contributors to this debate, I would single out Eric Falkenstein, for providing the most measured contribution to the debate. Not surprisingly, he has had practical experience both in risk management and strategy development. A not-so-veiled invitation to academics to get a real job.
I should end up with a video of Gene Fama (video). Being sponsored by an investment advisory firm fro which Fama is an advisor, it’s hardly impartial. Fama, who is in his eighties, is defending a piece of work for which he’ll go down in history, so it is strangely moving.