Lachlan's Rambling

Why I'm a Climate Skeptic

2012-02-07T01:18:00.001+11:00

I’m going to attempt to explain my position on climate change, mostly for my own benefit, perhaps. The controversial and divisive nature of the climate change debate means that a staggering amount is written about it on a daily basis. What can an amateur like me hope to contribute that hasn’t been better covered elsewhere? I shall attempt to offer a useful perspective. But doing this right means doing it long, so sorry about that.

My Road to “Denial”

For most of my adult life, I have believed that human pollution of the atmosphere was causing destructive changes to the climate, and that this would lead to catastrophe if we didn’t mend our ways. I know I learnt about the greenhouse effect, and that anthropogenic CO₂ was making it stronger, in high school; no later than 1989, when I was 16. I recall playing the original Civilization PC game in circa 1991. In the late stages of the game, when industrialisation was in full swing, the planet would start heating up, resulting in some areas becoming deserts and others jungles. By adopting “low pollution” technologies, the climate change impacts could be reduced, but not entirely avoided.

I read Ben Elton’s books. I listened to what Greenpeace was saying. For a time I was a member of the Australian Democrats, who were the Greens before there were Greens. When I saw An Inconvenient Truth in 2005, I was thoroughly impressed, and – this is very embarrassing – considered buying a Prius. In the 2007 Australian federal election I voted Green and looked forward to the new Labor government finally doing something about climate change.

In other words, I was a Believer, someone who supported the environmentalist agenda. But not any more. I have actually surprised myself at how rapidly my views changed. I have seriously wondered: am I a total flake, lacking in any real conviction? I report, you decide.

My first break with environmental orthodoxy was when, in my capacity as an amateur science geek, I started reading about how nuclear power plants actually worked, in detail. It didn’t take very long to figure out that almost everything environmentalists said about nuclear power and radiation was complete garbage. Most nuclear advocates also push the consensus line on climate change, as they see it as an opportunity to resuscitate the industry in the west. But I found there were some people advocating nuclear power while also expressing skepticism of climate change claims. That seemed weird.

I was perplexed that the reality of Anthropogenic Global Warming (AGW) was not universally accepted. I started wondering what the skeptics were actually skeptical of. Did they doubt the planet was warming, or merely the influence of humans? I decided I needed to fully understand the skeptical arguments to better counter them. What I discovered was that the skeptical “side” actually consists of a somewhat maddening diversity of opinions regarding exactly which aspects of the consensus are wrong. The consensus side, by comparison, was remarkably uniform in its position.

For a layperson, getting into the contentious scientific issues of climate change is not at all easy. You’ll find arguments and counter-arguments over virtually everything, and it’s not easy to tell what is important and what can be ignored. The IPCC reports themselves are almost completely useless for gaining any understanding of the issues. I had expected that the reports were designed to inform the policy debate. But no, they’re purely technocratic – written by experts, for experts. The best approach I’ve found is to listen to what experts say the IPCC report says, then confirm that myself as well as I can.

Eventually, I was able to figure out what the skeptical case is mainly all about. The first thing to understand is that there is relatively little skepticism of AGW itself. Most skeptics accept that the evidence for AGW is strong, and that its very likely real. The skepticism is about the magnitude of the anthropogenic influence, and whether we are facing a catastrophe. A more accurate label for “Climate change skeptics” would “climate catastrophe skeptics”. No one (fringe religious nuts excluded) is denying that the climate is changing. One of the central planks of the skeptical argument is that the climate is always changing, and we should not necessarily be concerned when we observe this.

The basic argument for AGW goes like this:

The average temperature of the planet has increased since circa 1940
Anthropogenic CO₂ emissions have risen steadily since circa 1940
Adding atmospheric CO₂ makes the greenhouse effect stronger, which makes the planet warmer
Ergo, anthropogenic CO₂ emissions are warming the planet

When a consensus person is impressing you with the strength of their evidence, they are usually supporting the above points. And when they ridicule the “denial” of science by skeptics, they are usually saying that skeptics deny the above. The thing is that most skeptics accept the above points. Now, there are valid scientific debates about each of these items, believe it or not. It’s not all completely settled. But overall, there seems to be agreement across both skeptics and warmists that the above points have pretty solid evidence to support them. Skeptics have spent much effort trying to show that point 1 is not as strong as we’ve been led to believe; but while there is undoubtedly problems with the instrumental temperature record, the bottom line conclusion seems to have held up.

So, it’s case-closed? Shut down Hazelwood and cover the land in wind farms, right? No, it’s no where near that simple. We have a huge, genuinely incontrovertible, mass of evidence that the earth’s climate changes all the time, all by itself, without any human influence. Of course, whatever natural climate effects were present in the past didn’t just disappear when humans started burning coal, oil and gas. So all the climate observations we make involve some combination of natural and anthropogenic influence (“forcings” in the lingo). In order to know what we’re actually dealing with, we need to understand the strength of the natural vs the anthropogenic forcings. This is called attribution, and this is where it gets really messy.

It’s when I started looking into attribution that started doubting the consensus position. There’s a bunch of details involved that seemed very important and relevant to me, but that I’d never heard any environmentalist, greenie or consensus scientist even hint at. First thing: the greenhouse warming from CO₂ is logarithmic. This means that as CO₂ is added to the atmosphere, the impact of adding more CO₂ becomes less and less. The standard analogy is to think about painting over a window: the first coat blocks a lot of light, the second coat blocks less light, and so on. Now, I don’t want to over-emphasise the importance of this – the IPCC fully takes it into account – but it was a shock to me because I realised I didn’t really understand the fundamentals of AGW. My diet of environmentalist propaganda had led me to believe that all those megatons of CO₂ were going to cause Venus-like runaway warming. Nope.

Next surprise: the ice cores. I can only recall one slide in Al Gore’s entire presentation that dealt with attribution: the slide showing the temperature and CO₂ graphs from ice core data. He then said:

The relationship is actually very complicated but there is one relationship that is far more powerful than all the others and it is this. When there is more carbon dioxide, the temperature gets warmer, because it traps more heat from the sun inside.

And indeed the graphs on the slide show a very close correlation between CO₂ and temperature. But I learned that Mr Gore was deliberately misleading his audience with this slide. What did he mean when he said the relationship is “very complicated”? Well, what the ice cores actually show is temperature increases happen roughly 800 years before increases in the CO₂ level. This is not mysterious, it’s basic physics that as the oceans warm up they will release more of their dissolved CO₂ into the atmosphere. So it’s actually not that “complicated”, Mr Gore just didn’t feel like explaining it.

The ice cores were once one of the best pieces of evidence that CO₂ had a strong influence on temperature. Initially the resolution of samples was not so high, and it did indeed appear that CO₂ and temperature increased together. Then scientists were able to improve the resolution and see that temperature changed before CO₂. The consensus now tries to act like this was totally expected. The claim now is that the planet starts warming (due to a cause we can only speculate about), this causes more CO₂ to enter the atmosphere, which then amplifies the warming (this is somehow not speculation). I find the best way to evaluate claims like this is to imagine if the result had come out the other way; if the cores had shown temperature increasing after CO₂, do you think they would say “oh, we didn’t predict that!” Of course not, it would be (with some justification) trumpeted as evidence of CO₂ being a major driver of climate change. The ice core data presents other problems for theories of CO₂-driven climate catastrophe: despite the “amplification” going on, the warming somehow stops long before anything catastrophic happens. And when the temperature falls, CO₂ levels stay high for over a thousand years. A far simpler explanation for the ice core data is that CO₂ levels respond slowly to temperature changes, but do not significantly drive temperature changes.

Sensitivity and Feedbacks

Next shock: when it comes to the total greenhouse effect of the earth, CO₂ is only a minor contributor. The most significant player is water vapour, by quite a margin. So how exactly does CO₂ act as the climate bogey-man? (Someone familiar with climate science might say “duh, climate scientists have always acknowledged the importance of water”. That is so, but I had listened to a lot of alarmists and never heard it mentioned.)

At this point, I have to introduce the notion of climate sensitivity. This is not a thing that is part of the physics of the climate, it’s basically a rough measure of how much warming we get from increasing CO₂ levels. For laypeople who are mainly interested in policy implications, the sensitivity is usually expressed as the temperature increase caused by doubling CO₂, both relative to pre-industrial levels (the doubling will occur circa 2070 assuming current trends). The last IPCC report says the climate sensitivity is probably 3.3℃, but the range of possible values is 2.0℃ to 4.5℃. The skeptics think it is significantly lower. If you want the science of the climate change debate boiled down to its essence, it is the determination of this value.

Now, it turns out we can use laboratory experiments to work out a reasonably accurate (for climate science) value for how much warming we get from adding CO₂. A climate sensitivity of around 1.2℃ from CO₂ warming alone is widely accepted by both warmists and skeptics. Huh? That’s a lot less than 3.3℃.

The discrepancy is because the lab experiments measure the CO₂ effect in isolation, but in the real climate all forcings like this are subject to feedbacks. We know of the existence of many feedbacks, both positive and negative, in the climate system. The problem is we have very little idea of how strong any of them are compared to the others. One of the major feedback components is water vapour in the atmosphere. The IPCC says that when increased CO₂ levels make the planet slightly warmer, this causes the atmosphere to hold more water vapour. Water vapour is the strongest greenhouse gas, so the earth gets even warmer. Of course, I’m leaving out a huge amount of detail in all this, much of which I don’t completely understand. But I think it’s correct to say that water vapour feedback constitutes the bulk of the warming the IPCC says we’ll get beyond the 1.2℃ direct forcing.

I just want to take a quick aside to emphasise how completely, utterly, totally bogus it is to refer to CO₂ (or even worse, “carbon”) as “pollution”. If you accept the argument that the mere fact it’s possible to accumulate too much of something means it should be called a pollutant, then you have to be consistent. Most of the warming the IPCC predicts comes not from CO₂, but H₂O. Therefore, water is a pollutant, and a more significant one than “carbon”. Reductio ad absurdum.

The big issue with water vapour in the atmosphere is that it will sometimes form clouds. Clouds can both warm and cool the earth’s surface. This means that adding water vapour to the atmosphere produces both positive and negative feedbacks. I think it’s fair to say that there’s few aspects of the climate that we understand less well than clouds. The bottom line is we can only speculate about what the net water vapour feedback is. We can’t even say with any certainty whether the feedback is negative or positive overall.

So in order to know what the climate sensitivity is, we need to know how the feedbacks work. But we don’t know that, so we use various means to try to estimate the likely range. I think you can pick any value for the climate sensitivity between 0℃ and 6℃ and you’ll be able to find one or more peer-reviewed scientific papers that say that could be the value. The IPCC’s estimate of 2.0℃ to 4.5℃ seems to have been arrived at almost entirely via modelling. This range requires net feedbacks to be positive.

The sign and magnitude of the feedbacks is the difference between major catastrophe and an effect so small it cannot be noticed against natural variation. But the direct empirical evidence for a net strong positive feedback is virtually non-existent. Models, combined with the argument “we know of no other cause of 20th century warming” (I’m not making that up, they actually say that) are all that supports it.

On the other hand, there are increasingly good reasons to think the net feedback is slightly negative. For one, almost all natural systems that have reached a stable or meta-stable state are governed by negative feedback. The fact that the earth’s climate doesn’t constantly careen from one extreme state to the next indicates that negative feedback probably dominates (it’s also worth noting another factoid never spoken of by the consensus – over the entire history of the earth, today’s CO₂ levels are relatively low). And there’s the fact that the warming predicted is just not eventuating. For over ten years now, the various temperature records, surface, satellite, ocean, have all been flat. All while CO₂ levels have continued to rise.

The way the consensus is sold is by focusing entirely on the strong part of their theory – that human CO₂ emissions make the planet slightly warmer. But the chain of reasoning should be judged by the weakest link, which is that the warming will be catastrophic due to positive feedback.

Climate Models

Climate models are just computer programs. I don’t pretend to understand all the details of how they work, but I will attempt to give a fair summary. They are written to simulate what we know and what we think we know about the physics of the earth’s climate. They take into account all known climate forcings (solar activity for example) and all known feedback mechanisms. You end up with a big pile of parameters in the model, the values of which cannot be determined by observation with any great certainty. These parameters are set by a combination of simple assumption, combined with tuning the output of the model to match certain historical records. The IPCC (ok, not them directly, the scientists whose papers they rely on) then re-run the models with anthropogenic CO₂ left out, find that the results don’t match history, and from this conclude that their assumptions of positive feedback are correct and that CO₂ is the main driver of global warming. And this is how the value for climate sensitivity is arrived at – the models say that earth with CO₂ doubled is 3.3℃ warmer than earth with CO₂ unchanged.

You may be thinking that I’m either lying or mistaken here, because that sounds completely bloody crazy. It’s simply begging the question. Well I’m not lying, but I could be mistaken. Take a look at the critical IPCC chapter on attribution, especially pp 702-703 and decide for yourself if I’m being fair.

I haven’t yet heard a convincing argument as to why using models like this is evidence of anything. A telling sentence on page 702 referenced above is “Numerous experiments have been conducted using climate models to determine the likely causes of the 20th-century climate change”. The word “experiments” here is quite misleading, as in the scientific context it implies you’re doing something empirical. But all climate models do is answer the question “if I assume the climate works like this, what are the implications?” This is undoubtedly useful, but it can’t tell you if the assumptions you start with are correct. Model outputs are not observations.

But if the climate model produces results that match real-world historical data, doesn’t that prove the model’s assumptions are basically valid? No. A quote from John von Neumann sums it up: “With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.” Climate models have many parameters. Like the relative strengths of various feedbacks. But the models also need to account for periods where CO₂ rose and temperatures fell (the 1970s for example). That’s where some of the negative temperature forcings come into play, the principle one being “aerosols”, which are mainly genuine pollutants like sulphates. We know that aerosols make the earth cooler overall (sometimes called “global dimming”) but, again, there’s huge uncertainty as to the strength of the effect. Because various positive and negative parameters are so poorly constrained by observation, it’s just a matter of tuning them until your model roughly matches the real-world data. The IPCC uses many climate models, each with different assumptions and producing slightly different future “scenarios”. Obviously they can’t all be correct, but they all match historical data to roughly the same degree.

The Bottom Line

The heart of the scientific debate about climate change is the issue of attribution. It is a real debate, concerned with real evidence. It is simply not true that the only people not going along with the consensus are few science-denying fools or paid shills for the fossil fuel industry. I didn’t learn about these issues on my own, there’s a great many knowledgable people who have come to doubt what climate change activists and the IPCC are telling us. There are very good reasons to doubt the consensus claim that almost all 20th century warming can be attributed to anthropogenic influence.

What should we do? Well, we can’t decide on a policy based on climate science alone, we have to examine the available options and their costs. That’s a whole new rabbit hole, so I won’t get into it now.

I would, however, like to address some of the common comebacks to the type of argument I’ve laid out above. When doubts about the consensus scientific position are raised, usually only professional scientists understand the details will go into bat and defend it. But most people who are advocating “action” on climate change aren’t really able to do this. Instead, it’s amazing how quickly they will declare the truth of climate science claims as basically irrelevant because of some other overriding reason.

1. We Should Look After the Environment

“It doesn’t matter if climate change science is wrong, because we have to look after the environment anyway”. This argument is actually at the heart of the climate change hysteria. Whenever the catastrophic climate change barrow is pushed, it is invariably linked to pollution and environmental problems generally. But our very real environmental challenges are not causing climate change, nor are they caused by it. Emissions from cars and coal power stations certainly are dirty and can cause environmental problems, but it’s not the dirty or toxic stuff that makes the greenhouse effect stronger. Neither do oil spills or the myriad of other ways we manage to screw things up. Whatever the merits are of the clean-up-the-planet argument are, it is a separate issue to what’s going on with the greenhouse effect. Alarming people with tales of climatic disaster is not an honest way to pursue a clean air agenda.

Australia’s “Carbon Price” is sold as the means to a “clean energy future”. So with all the nasty things emitted by fossil fuel combustion (there are many), we put a price on an emission that is not dirty or toxic, and was abundant throughout the biosphere before the first human drew breath. Yeah, that totally makes sense as a clean air policy.

Just a quick note on something you’ve probably heard about: that increasing CO₂ levels are making the oceans “more acidic”. The scare mongering about ocean acidification is stopped in its tracks by the elementary observation that the average pH of the oceans is about 8.14. That’s right, the world’s oceans are alkaline. So while it’s technically correct that increased CO₂ has made the oceans more acidic, it’s quite misleading. It’s more honest to say that the oceans are becoming more neutral, that is, less corrosive. Claims about the dangers of ocean acidification are just another desperate attempt to make CO₂ into a dangerous pollutant.

2. Fossil Fuels are Bad

Related to the previous argument, there is the “it doesn’t matter if climate change science is wrong, because fossil fuels are bad and we should stop using them”. There are indeed plenty of downsides to fossil fuels (and some major upsides too). However, this is again a separate debate from climate change. The various carbon pricing and emission reduction schemes were not sold on the basis that fossil fuels have downsides, they were sold based on predictions of drowned coastlines, dead coral reefs and endless droughts. If you want to make the argument for the negatives of fossil fuels, then make the argument on its own merits, don’t try to force your agenda with scare tactics about the climate. Oh and having a remotely practical alternative might help, too.

3. Idiots whom I hate are climate change skeptics

I’ve been really surprised how often I’ve heard some variation of this: “I can’t be a climate skeptic, because that would mean agreeing with: Tony Abbott, Andrew Bolt, Glenn Beck, assorted conservative religious nuts, et alia”. I have to admit that I have fallen victim to this line of “reasoning” myself in the past, and now do my best to guard against it. Forming your opinion on a matter based on the opinions of people you don’t respect is manifestly stupid. Need I say more?

4. We should act as a precaution

The argument goes: “because we can’t definitively prove that anthropogenic emissions are not going to destroy the world, we should play it safe and cut emissions”. This will often be backed up with some demagogic nonsense such as “we should not gamble with our children’s future”. The flaw in this argument becomes obvious when it is applied to an issue other than climate change: “because we can’t definitively prove that an asteroid is not going to destroy the world, we should play it safe and spend trillions of dollars on an asteroid defence system”.

A another deliciously silly argument I’ve often seen attached to this is that the “only” cost of such precautionary action is money. Right. Money. The thing we use to represent our available resources. The resources we use to sustain, grow and advance our civilisation. Oh, if that’s the only cost, I don’t know what all the fuss is about. The best part is that if you ask the person making this argument “should we spend more on education/health care/social justice/foreign aid/anything except defence”, they’ll say “yes” to every one without a hint of irony.

What’s lacking here is any assessment of risk and cost. Most environmentalists seem to avoid getting into any hard numbers concerning risk and cost, likely because their ideas do not come out well. That’s a topic for another post, however.

Posted via email from lockster's posterous

Applying Applicative Functors in Scala

2011-09-02T12:32:00.001+10:00

A chap named ittayd has written a nice gentle tutorial on functional programming in Scala. My only minor complaint is that he calls the applicative functor “sequential application” function apply. While this is a logical choice, in Scala apply is used everywhere to implement plain old function application, so it could lead to confusion. I think it’s better to call it <*>, as in Haskell. Read on.

I found the tutorial interesting because it talks a bit about applicative functors, and how to give them a nice syntax in Scala. The one thing that has bugged me about applicatives is that using them in Scala always felt clunkier than Haskell. Ittayd comes up with a scheme I like, but says “we have one difficulty: we can no longer define apply inside Future”. He uses a standard object-pimping implicit conversion to get around this, and that works fine. However, it is possible to define <*> inside your thing-with-an-applicative-functor, using one of my favourite Scala features: generalized type constraints. To continue with ittayd’s example using Future:

trait Future[A] { … def <*>[X, Y](x: Future[X])(implicit ev: A <:< (X => Y)) = new Future[Y] { def get = Future.this.get(x.get) def isDone = Future.this.isDone && x.isDone } }

The compiler will only allow <*> to be called on a Future that provides an appropriate function as its value.

Ittayd bootstraps things simply by lifting the function to be applied into Future, but personally I prefer the Haskell style of using <$>, aka “map”. In Scala, <$> is unsuitable for a few reasons, so I picked **::

trait Future[A] { … def **:[B](f: A => B): Future[B] = map(f) }

Obviously, this is just the standard map method with a different name, what’s the point? Well, because the operator ends with a :, it’s right associative; and because it starts with an *, it has higher precedence than <*>; so you end up with:

(marry _).curried **: joe <*> jane

Or, if you subscribe to the slippery slope theory of gay marriage:

(gayMarry _).curried **: joe <*> bob <*> george <*> fido <*> aDonkey

This is reasonably close to the Haskell idiom:

gayMarry <$> joe <*> bob <*> george <*> fido <*> aDonkey

You may wonder why obsess over the best way of dealing with applicative functors? After all, most of the applicative functors you see have an associated monad, so we can use Scala’s nice for syntax. Yes this is true for many useful applicative functors, but not all of them.

Just a quick asside on termonology. In casual conversation, you may hear it said that “Option is a monad”. But it’s more accurate to say “Option has a monad”. Because many types only have one useful monad associated with them, we tend to think that the type itself is the monad, but that’s not quite right. This doesn’t just apply to monads, of course, the same goes for monoids, applicative functors, etc. For example, numeric types have two useful monoids: one for addition and one for multiplication.

Now consider a Validation type from this Tony Morris tutorial. You can write a useful monad for this type, which implies an applicative functor as well. However, the tutorial is concerned with a second useful applicative functor for Validation. This second applicative is only relevant for the case where the Validation is using a semigroup as its error type (the tutorial explains what a semigroup is), and it allows using the semigroup to accumulate a list of all the validation failures. The classic example of this is validating data entered by a user on a form with several fields. The “plain” applicative/monad will only record the first validation failure it finds, whereas the semigroup applicative will record all the failures.

Now, finally, to my point (I think). This applicative functor that uses semigroup does not have a corresponding monad. Which means that we can’t use it with Scala’s for syntax. Hence the desire for a nice syntax for applicative functors. It is most definitely useful.

Obligatory Scalaz reference: those guys have pretty much worked out the best way to do all this stuff in Scala. But I find I learn a lot by experimenting myself.

Posted via email from lockster's posterous

A variation on Scala's Either type

2011-05-03T23:23:00.001+10:00

I’ve been using Scala’s Either type a bit recently, and found a couple of minor annoyances with it. So as a learning exercise I tried to make a variation of Either, and this is what I came up with. I also wanted to experiment with syntax a little. See what you think.

Either is a disjunctive type with two possible values: a left and a right. It places no meaning at all on left or right, but 99% of its uses in practice involve putting normal, expected values in right, and errors or other “exceptional” values in left. This means that almost all the time you end up getting the RightProjection, e.g.:

for { v1 <- foo(…).right v2 <- bar(…).right v3 <- baz(…).right } yield …

Whereas I’ve never used .left even once.

The second minor issue is that Either has some limitations with patterns in for expressions, because of how it is forced to implement the filter method. See this Stack Overflow question for the details.

So I created a type that is explicitly designed to handle the possibility of “exception” conditions. Because the word “exception” already has a well known meaning on the JVM, I called this type Anomaly instead. Actually, I ended up calling the type |: because I thought it worked nicely syntactically. Here’s a comparison with Either:

val e1: Either[String, Int] = Right(30) val e2: Either[String, (Int, Int)] = Right((10, 2)) val e3 = for { i <- e1.right j <- e2.right } yield i + j._1 + j._2 val e4: Either[String, Int] = Left("error")

vs “Anomaly”:

val a1: String |: Int = Expected(30) val a2: String |: (Int, Int) = Expected((10, 2)) val a3 = for { i <- a1 (j, k) <- a2 } yield i + j + k val a4: String |: Int = Anomaly("error")

The |: operator is right-associative, which helps for the occasional cases where you have an Anomaly within an Anomaly, eg. String |: String |: Int is equivalent to Either[String, Either[String, Int]]. Of course, depending on your taste you could reverse the order of type arguments, using an operator like :|, or use a name instead of an operator.

There’s a cost to making the for expression pattern matching work like this, however. In addition to the Anomaly and Expected cases, I had to introduce a NoValue case object, which is what is returned when a filter does not match. This is annoying, because now folding or pattern matching must account for three cases instead of two. And unless you have explicit filters or patterns matching particular values, the NoValue outcome won’t come up, so it’s a pain to have to check for it. What I ended up doing was providing an alternate two-parameter fold method that calls error in case of a NoValue. You just have to be careful about only calling it when you know it’s safe.

All-in-all I’m still wondering whether this experiment is worthwhile, but here’s the code in case anyone is interested. There are other utility methods that could be added:

sealed abstract class |:[+A, +E] { def anomaly: A def expected: E def isExpected: Boolean = false def isAnomaly: Boolean = false def isNoValue: Boolean = false  def map[X](f: E => X): A |: X  def flatMap[X, AA >: A](f: E => AA |: X): AA |: X  def withFilter(f: E => Boolean): A |: E = this final def filter(f: E => Boolean): A |: E = withFilter(f)  def foreach(f: E => Unit): Unit = {}  final def fold[X](fExpected: E => X, fAnomaly: A => X, fNoValue: => X): X = this match { case Anomaly(a) => fAnomaly(a) case Expected(e) => fExpected(e) case NoValue => fNoValue }  final def fold[X](fExpected: E => X, fAnomaly: A => X): X = this match { case Anomaly(a) => fAnomaly(a) case Expected(e) => fExpected(e) case NoValue => error("Attempt to fold NoValue case") }  final def join[X, AA >: A](implicit ev: E <:< (AA |: X)): AA |: X = this match { case Anomaly(a) => Anomaly(a) case Expected(e) => e case NoValue => NoValue } }  final case class Expected[+A, +E](expected: E) extends (A |: E) { override def isExpected = true def anomaly: A = Predef.error("Not an anomaly")  override def withFilter(f: E => Boolean) = if (f(expected)) this else NoValue  def flatMap[X, AA >: A](f: E => AA |: X) = f(expected)  def map[X](f: E => X) = Expected[A, X](f(expected))  override def foreach(f: E => Unit) { f(expected) } }  object Expected { def cond[A, E](test: Boolean, expected: => E, anomaly: => A): A |: E = if (test) Expected(expected) else Anomaly(anomaly) }  final case class Anomaly[+A, +E](anomaly: A) extends (A |: E) { override def isAnomaly = true def expected: E = Predef.error("Not an expected value")  def flatMap[X, AA >: A](f: E => AA |: X) = Anomaly[AA, X](anomaly)  def map[X](f: E => X) = Anomaly[A, X](anomaly) }  object Anomaly { def cond[A, E](test: Boolean, anomaly: => A, expected: => E): A |: E = if (test) Anomaly(anomaly) else Expected(expected) }  case object NoValue extends (Nothing |: Nothing) { override def isNoValue = true def anomaly = Predef.error("Not an anomaly") def expected = Predef.error("No value")  def flatMap[X, AA >: Nothing](f: Nothing => AA |: X) = this  def map[X](f: Nothing => X) = this }

Posted via email from lockster's posterous

Ceylon: Interesting for the Wrong Reasons

2011-04-15T12:27:00.001+10:00

A couple of days ago, Gavin King announced that he is leading a team at RedHat that’s creating a new JVM language called Ceylon. See here for most of what is known so far. The stated motivation, in a nutshell is that they like Java very much, but it has deficiencies that are preventing progress in key areas, and they are generally feeling “frustrated”. They don’t have a complete compiler yet, but they’re working on it, and I get the impression RedHat is pretty serious about it.

Gavin King’s slides provide a fascinating insight into the kind of thinking that remains so dominant in the Java community. Gavin King is well known in the Java world as the inventor/leader of Hibernate and various other JBoss projects. He certainly knows Java inside out, and his list of “frustrations” is astute, if incomplete. While I haven’t dug into all the details so far revealed about Ceylon, but it seems likely that they’ll succeed in their goal of creating “a better Java”. If given the choice between using Java and Ceylon, I could certainly see myself choosing Ceylon. But Ceylon is a terrible idea. The problem is that they are treating the symptoms, not the disease; and they’re repeating exactly the same mistakes that resulted in them finally hitting a dead-end with Java.

Possibly the best outcome of Ceylon is that it emphatically refutes the argument that “Java is all we need”. Few have put more hard labour into trying to make Java feasible than Gavin King and the JBoss folks. They and a few others have spent the last decade or so producing a staggering number of bloated tools, libraries and frameworks in order to keep Java going. Don’t get me wrong, there’s a lot of extremely useful Java libraries out there, but there’s also a huge amount of guff that exists solely to compensate for the impracticality of the Java language. Ceylon is the “main-stream” admitting, finally, that not all problems can be solved the Java way: more frameworks, more libraries, more tools, more code.

So Mr King and friends have finally hit the wall with Java. They’re a bit late to the party, but we should welcome them none the less. So why do I say Ceylon is a terrible idea? The short answer is: Scala. The long answer is, well, longer. My biased summary of the situation is this: after years of having to build more and more layers of increasingly baroque frameworks in order to make progress, they realise that they can no longer keep up with their competitors (cough_C#cough_) because Java the language is an unavoidable “bottleneck”. They identify certain essential abstractions that are needed to solve their immediate problems, and proceed to design a new language that provides those essential abstractions, while remaining as “familiar” as possible.

In other words, Ceylon is all about making “a better Java”. There is certainly support for a evolutionary approach like this. Some argue that Java itself was (in language terms) “a better C++”, so why not follow that successful approach now? Well, if that worked so well for Java, why are even its greatest fans now trying to replace it? I’m reminded of Linus Torvalds' reaction to “Subversion is a better CVS”. The problem with Subversion isn’t that is poorly designed or built; it isn’t. The problem with Subversion is that the goal itself just isn’t very useful.

This is how it’s going to go: Ceylon will be built, they’ll make a SDK library, then there will be Hibernate-Ceylon, Spring-Ceylon, WebFrameworks-Ceylon 1 through 63,854, Logging4Ceylon, Ceylon.util.logging, commons-logging-ceylon and Simple Logging Facade 4 Ceylon. All these will be an improvement over their Java equivalents. Then, in another five or so years they’ll hit some new pain points and realise that some of those other “academic” abstractions are actually useful in the real world after all. For example, Ceylon supports higher-order functions (but, inexplicably, not lambda expressions). If we could go back in time 5 years and ask Gavin King “how important do you think higher-order functions are?”, what would he say? My guess is “they’re not essential for Real Work™”.

But it’s still an improvement, so why not embrace it? Because we have an already have a much better option that’s already doing the job in the real world: Scala. Or look at the interesting stuff going on in .NET land. Ceylon’s goals look positively narrow and uninspiring by comparison. Of course, Gavin King was immediately asked “why not Scala?”. It is my sincere hope that this question will continue to be asked of him. His answer consists of the same misconceptions that Scala advocates have been trying to dispel since forever. Take this statement from Mr King, from an interview in InfoQ:

One important force that helped make Java so successful was the things Java left out. But you need to be very smart here: Java’s decision to leave out higher-order function support was enormously harmful. Now, obviously the Scala team have gone down a very different path here and have created a very feature-rich type system. As a general rule I would prefer to solve my problems with fewer, more general constructs, than with more, less powerful constructs.

This is the exact inverse of the truth. No one who has made a serious effort to understand Scala could make such a statement. “Fewer, more general constructs” is exactly what Scala is all about! My summary of Scala’s mission statement would be as follows:

Scala aims to maximise the potential for useful abstractions, whilst:
- giving assurance that your abstractions are correct (static typing)
- allowing you to do what is necessary to get the performance you need
- maintaining seamless interoperability with Java

Scala aims to be compatible with Java, but it’s about much more than just “a better Java”. Ironically, it seems as though Ceylon will not interoperate with Java as seamlessly as Scala does (e.g. Scala keeps null and type erasure). It’ll be interesting to see how that works out. To get an idea of how serious the Scala team is about powerful but practical abstractions, look at how they handle the tensions that crop up between abstraction and performance. Great examples of this are how Scala handles the unpleasantness of arrays on the JVM, and also the @specialized annotation.

Scala is by no means perfect. The trade-offs it makes will not suit everyone fully. But I cannot help but scratch my head over statements like this (Gavin King, again):

But I guess I should mention that the number one technical problem that we simply can’t solve to our satisfaction in Java – or in any other existing JVM language – is the problem of defining user interfaces and structured data using a typesafe, hierarchical syntax.

I could very well be missing something, but this would seem like a problem that is right in Scala’s wheelhouse. Perhaps it may not be solved satisfactorily today, but it could done with far less effort than creating a whole new language (and new standard library, and new tool chain…). It is depressing to think of what these guys could achieve working with Scala instead of pouring resources into Ceylon.

You may wonder, “if you think Scala is so good, then use it and be happy, why worry about Ceylon?” Well, I worry because I think Ceylon is worse than Scala, but it could win anyway. That actually seems to be the more common outcome in these situations. I would much prefer a world with Scala jobs in demand than one with Ceylon jobs in demand. So, yes, it’s all about me being selfish. I would say to all Scala fans: don’t be afraid to be a little selfish and evangelise for the better outcome.

Posted via email from lockster's posterous

Creationism and climate change denial

2010-12-21T17:26:00.001+11:00

I’ve noticed this more and more: people lumping “climate change denial” and “evolution denial” (creationism) into the same category. The argument is that both of these are the result of the same religious conservative “anti-science” movement. In a very narrow sense, this is true; creationists generally do not seem to believe in anthropogenic global warming (AGW), for the same bogus reasons they don’t believe in evolution. But the reverse is not true: many (most I would say) AGW skeptics are not creationists. Lumping these two things together allows the environmentalists to dismiss any criticism of the established climate change narrative as the ravings of anti-scientific religious nuts. But there are some very significant scientific and economic arguments around climate change that are most definitely not in the same class as creationist arguments. Time for some de-lumping.

One major difference between climate change and evolution is how the evidence is understood and presented. It is now a familiar experience to see someone on TV telling us we need to immediately reduce emissions to avoid climatic catastrophe. In the rare event that such a person is asked to provide evidence of a human cause, the most common answer I’ve observed is along the lines of “ice caps and glaciers are melting, and the scientific consensus is that human CO₂ emissions are to blame.” Now, when was the last time you heard someone asked for evidence supporting evolution to respond with “life forms adapt to their environment and the scientific consensus is that Darwinian evolution is the cause.” This is not a small point. The fact that “consensus” is so regularly rolled out to support AGW, but is never used to support evolution shows that the levels of scientific understanding and debate are completely different.

No one talks about the “scientific consensus” for evolution because there’s no need to: evolution has a mountain of cold hard facts on its side, all confirmed by independent observation countless times. Climate research is very tough by comparison: our observational capabilities generally aren’t good enough to falsify competing theories. Instead, we have the IPCC saying that since we can’t establish the facts by direct observation, we’ll establish them by consensus. You can’t take the IPCC publications, repeat the experiments and confirm their results, because at the end of the day it’s just opinion.

My point here is not to show that the IPCC is wrong, but simply to show that the scientific process that it follows is fundamentally and necessarily different to that of evolutionary biology, or any other hard science for that matter. Questioning the consensus-derived “facts” of climate research is not the same thing as questioning the empirically verified facts of evolution.

Another difference I think is worth pointing out is how politics is so commonly mixed with climate science, in contrast to evolution science. Tell me if this scenario sounds familiar: well credentialed scientist goes on TV and says “my research shows that climate change is likely to significantly damage coral reefs…” and barely pausing for breath continues “so it’s clear we must act to cut CO₂ emissions.” This casual connection of minor scientific research to sweeping reorganization of the world’s energy production is now so common place that people don’t find it remarkable. And if you accept the climate change problem but doubt the solution you’ll likely still receive the “denier” pejorative. Evolutionary biology simply isn’t political in this way.

Skepticism regarding climate change is not comparable to the denial of the established facts of evolution. Climate research has a long, long way to go before it achieves the level of confidence we have in evolution. It is very wrong to think that all opposition to the climate change narrative is driven by Biblical literalism.

Posted via email from lockster's posterous

WikiLeaks

2010-12-09T13:55:00.001+11:00

Attack and Counter-Attack

I must confess, no story in recent months has captured my attention like the still-developing WikiLeaks saga. The ongoing dump of diplomatic cables combined with Julian Assange being charged with some sort of sex crime in Sweden means this is going to be in our face for a while yet.

The response to WikiLeaks has been loud, angry, hyperbolic attacks from “tough on terror” types in the U.S., and a corresponding thoughtless, reflexive defence from many “progressive” types. Amongst the noise are a few precious thoughtful critiques.

Clearly, WikiLeaks is not engaged in terrorism, and Julian Assange is not remotely comparable to the most junior jihadist. Sarah Palin’s pronouncements on the subject are pure demagoguery (by now I think we can fairly declare that an axiom with regard to any statement she makes). And Senator Lieberman needs to seriously deflate his sense of his own importance, and recognise that maintaining the government’s operational security isn’t actually in his job description.

I believe all these clumsy attacks against WikiLeaks will fail in the end, and they deserve to fail. In fact, I think they are completely counter-productive: sympathy for WikiLeaks is growing as people become concerned about the government trying to restrict speech.

The defenders of WikiLeaks are themselves hugely overreacting, however. The “attacks” against WikiLeaks and Mr Assange so far consist of: a few random nuts-with-a-mike threatening violence, some companies cutting off service, and a number of people expressing the opinion that U.S. law has been broken. Taken together, all this doesn’t come close to being a serious attempt to censor WikiLeaks. In the event that Mr Assange is charged by the DOJ, which is unlikely in my opinion, he’ll have his day in court. In the even more unlikely event that he’s actually convicted of espionage, then it should be the law you’re complaining about rather than the enforcement of it. And as Christopher Hitchens points out, those who choose to engage in civil disobedience must be prepared to face the consequences.

(The reason I say charges are unlikely is that I can’t see how the DOJ could credibly prosecute Mr Assange without also prosecuting the editor of the New York Times, which would be sure to end in disaster. But, who knows?)

At a time when publishing a cartoon depicting the Prophet Mohammed means you run the very real risk of being murdered, I can read people earnestly telling me that Amazon and Paypal cutting off WikiLeaks represents the greatest free-speech fight of our generation. Perspective is sorely lacking here.

Transparency

Overall, I’m far more concerned about the way WikiLeaks is being lauded and defended than the way it’s being attacked. I’ve seen a call for Julian Assange to be awarded the Nobel Peace Prize and another for him to be named Australian of the Year. This is madness.

The word most often used by WikiLeaks fans is transparency. More transparency is good, and WikiLeaks provides more transparency, therefore WikiLeaks is good. Mr Assange certainly understands his fan base very well, as this is the line he takes in his deceitful, self-serving piece in The Australian. Well, I need to make a few points about transparency.

First, transparency is a means, not an end in itself. The ends are, essentially, ensuring that government office holders are acting within the law and in accordance with the will of the people. Of course, this is both a good and necessary thing. In principle, I’m all in favour of leaking government secrets when there’s a clear public interest in doing so. This has served us well in the past, and will do so again in the future.

But it must be recognised that transparency may also lead to negative ends. This is why responsible journalists weigh, as best they can, the public interest against the potential downsides before publishing leaked information. This article provides a good description of the negative ends that may result from the “transparency” that WikiLeaks has forced on us.

Now we must delve into the theories and ideology of Julian Assange. He has explained these in detail, but I shall do my best to summarise them fairly here. What the defenders of WikiLeaks don’t seem to have realised is that Mr Assange actually has no interest in improving transparency, or holding governments to account. His goal is essentially to conduct information warfare against what he views as “conspiratorial authoritarian regimes”. This is why, in sharp contrast to all the talk of transparency, WikiLeaks does everything it can to conceal its own activities. It’s an organisation built for conducting information warfare.

So, the objective of indiscriminately releasing all the diplomatic cables was not to inform the public, the objective was to impede the ability of the “regimes” to conduct diplomacy in the future. This is a goal that may very well have been achieved. Julian Assange is not trying to improve the system, he’s trying to smash it. In a nutshell, he’s an anarchist using information as his weapon. And after he has smashed every institution of liberal democracy, what then? Like any good anarchist, he has nothing more to say.

The fact that so many of the leaks have been “against” the U.S. government leads to the unavoidable conclusion that Mr Assange considers that government to be a “conspiratorial authoritarian regime”. Judging by his rhetoric, I strongly suspect he views all governments that way. Now we come to the really critical point: if Julian Assange is right about this, then his actions make perfect sense and should be supported. It should be no surprise by now that I think he is completely wrong.

It is necessary to follow the implications of accepting Mr Assange’s thesis. He is claiming that our governments that are ostensibly democratic systems working for the people, are in fact authoritarian regimes that are actively conspiring against the people. Your nation’s constitution is a lie. Our traditions of liberal democracy are actually myths. Debate is pointless. Your vote, irrelevant. Breaking the machine with information warfare is the only way to truly affect change.

Now if I try to muster every last reserve of cynicism and despair at my disposal, I cannot bring myself close to believing such a thing. Am I a naive, dewy-eyed, babe in the woods?

Returning the subject of transparency, I would argue that in historical terms we’re actually doing reasonably well on the transparency front these days. Let us consider some of the government activities that the WikiLeaks cheerleaders are generally most concerned about: detention without trial, “enhanced interrogation”, handing prisoners over to foreign authorities known to practice torture, extraordinary rendition, support for the Karzai regime/racket, I could go on. Many of these issues concern me too. But we knew about these policies before the valiant Mr Assange rode onto the scene. The leaks have, at most, underscored a few things. These policies are no more likely to be changed after the leaks than they were before.

Opponents of these harsh policies need to face up to a conclusion that appears most unwelcome to them: conspiracy and lack of transparency are not what keeps the policies alive. The real reason they persist is much more prosaic: the electorate, broadly speaking, is ok with them. What’s called for is less vandalism and more old-fashioned criticism. Make your arguments and try to convince your fellow citizens. Don’t be afraid to be boring about it (I certainly have no such fear – ha!). Or forget all that and go with cynical ranting about a public stupefied by mass media and consumerism. Either way.

In conclusion, I do not want to see WikiLeaks silenced or Julian Assange in jail for espionage (I’m not going to comment on the Swedish charges, thus far they seem completely tangential). However, those of us who believe that liberal democracy has some life in it yet need to criticise the undemocratic and destructive ideology that drives WikiLeaks. This is a secretive, unelected, unaccountable group that releases whatever information it can get its hands on. It indiscriminately and purposefully harms the ability of our government office holders to do the work we elected them to do. Our oft-derided citizenry at least have the pragmatic sense to recognise that those to whom we have delegated power can better achieve our common good when some secrecy is maintained. Julian Assange should respect that democratically determined outcome, and give up this reckless test of his pet conspiracy theories.

Posted via email from lockster's posterous

Some more thoughts on Apple

2010-10-27T10:24:00.007+11:00

A few more points that occurred to me, following up on my last post.

Whither the mouse?

Since its inception in 1984, the Mac user interface has obviously been designed for desktop computers with a mouse and a keyboard. When notebook computers came along and we wanted to be able to use them without carrying a mouse everywhere, or without even a flat, stable surface to put the mouse on, they had to emulate the mouse as best they could. This may now be reversing.

In the announcement of Mac OS X Lion, there was great emphasis placed on the use of multitouch gestures as a means of control. I think Lion will be the first OS designed for notebooks with multitouch trackpads, with the desktop Macs left to emulate the MacBooks as best they can. If you consider that a sizeable majority of the hardware Apple sells today is mobile and primarily operated via some form of multitouch instead of a mouse, this move seems sensible. And all new desktop Macs ship with the multitouch-capable Magic Mouse as standard now.

The Costs of Supporting Java

With regard to Java, there's been a few comments along the lines of "all the hardware Apple sells to Java developers must more than cover the cost of the handful of engineers needed to keep maintaining Java". I think this underestimates the costs involved. Supporting Java means, in addition to the engineers, that:

Java must be accounted for in development plans
Java must be accounted for in QA plans
Customer support needs to deal with it
Security vulnerabilities need to be investigated and fixed

The true cost isn't so much the money, it's that it unavoidably diverts some of the company's attention from things it would rather focus on. While continuing with Java support wouldn't be hugely onerous for Apple, eliminating distractions that aren't bringing much benefit can really help competitiveness.

Also, as the vast majority of Java work (since the switch to Intel) has been on the GUI, I'd expect the Java team knows the guts of the Mac GUI pretty well by now. Engineering talent isn't exactly a fungible resource. There's probably a development manager somewhere in Apple salivating at the thought of having the Java people on her team.

Apple and the Server Side

Folks have pointed out that Apple is a heavy user of server-side Java in their web services. I can think of many ways they can deal with that, and have no idea which option they'll take. But I think we can be sure they've got a plan and aren't leaving it to chance. And despite everyone (including me) getting very animated over "Apple dropping Java", it's worth remembering that their Java team isn't going to be shutting down tomorrow.

More generally, Apple's plans for the cloud are completely opaque to me right now. We're all assuming they have big plans for cloud services: they built that huge new data centre, and their hardware strategy seems to imply more reliance on cloud services in the future. The iPad's greatest weakness is that so much of its functionality, including just initialising it, requires USB synchronisation with a PC or Mac. MobileMe syncing is great, but it only works for a few things.

The new MacBook Air models were just announced, including one with an 11" screen, which I think is a first for Apple notebooks. When making the announcement, Steve Jobs said something like "we think this is the future of MacBooks". With their emphasis on mobile, wireless use and limited storage, these too are crying out for more cloud services.

And I think this could be Apple's greatest weakness. If I were Steve Jobs, the thing that would worry me most about Android isn't its "openness" (however you care to define it), or the temporary situation of shelf-space in U.S. phone outlets, but the fact it is backed up by the world's best cloud services. And they're only going to get better. Obstacles remain, but the movement towards doing more and more in the cloud is inexorable. Where Google has no rival, Apple's track record is not that great. The iTunes store is a huge success, obviously, but it's not really all that "cloudy", as you need a Mac with enough hard drive to hold all your content. MobileMe is expensive, and often unreliable and slow; it really only survives due to the first class client support on Mac and iOS. The iWork service is not bad, but lacks Google's killer feature of collaborative editing. Apple has its work cut out for it when it comes to the cloud.

Flash, Core Animation and Multitouch

In my last post I claimed Flash didn't support Core Animation or Multitouch. Well, @johnyanarella set me right on that. Flash does support both of these. I haven't used that many Flash apps, so now I'm wondering to what extent the multitouch support is actually used in apps.

Java and the Mac

2010-10-22T22:52:00.003+11:00

So Apple has quietly announced that it is deprecating its support for Java on the Mac. The wording was (not uncharacteristically for situations like this) a bit vague on the details of what this means. I think two things are definite: 1) Apple will continue to support Java 6 on Snow Leopard until it reaches end of life; which I think happens when the OS after Lion is publicly released. 2) Apple will not port Java 7+ to the Mac. The main unresolved question is whether Java 6 will be supported on Lion. My guess is that it will be offered as an optional download, similar to Rosetta in Snow Leopard. Apple won't make any promises though, as they will want to have the option to drop it if it's too much effort.

As a Mac user who has been working on the Java platform for many years, this is a fairly big deal for me. I think it's an interesting event that provides a chance to examine both Java's position and Apple's plans for the Mac. People in the Mac Java community are speculating about the deeper meaning behind this move, especially when combined with Apple effectively banning Java platform applications from the new Mac App Store. So time for me to add my speculation to the noise.

Is Apple trying to force the use of their platform and tools?

No. True, Apple has made it crystal clear on numerous occasions (the Mac App store requirements are just the latest case) that they want developers targeting their OS platforms to use their development platform (Objective-C, Cocoa, Xcode). But I don't think they are deprecating Java in an attempt to force the issue. After all, Lion will come with Perl, Python and Ruby all pre-installed.

Does Apple want to kill cross-platform desktop apps?

No. I think it's true that, strategically speaking, Apple has no interest at all in cross-platform apps (outside of the browser). But along with announcing the deprecation, the latest Apple Java release also includes brand-new support for installing multiple versions of the Java platform, explicitly mentioning support for third-party versions. It's clear that Apple has no problem at all with Java running on Macs. I think they want Oracle to take it over, but I doubt they bothered to make any sort of deal with Oracle before making this move. Update: as I was drafting this, news came through that Steve Jobs has replied to a question about this, which basically confirms that he thinks Oracle should be responsible for Java on the Mac.

Then why did Apple do this?

The reason isn't that complicated: Apple no longer needs Java. If you make a list of what Steve Jobs sees as the critical objectives for Apple, it becomes immediately obvious that maintaining a Mac port of Java is not helping to advance any of them. Of course, neither does maintaining, say, Apple's port of Python. But Python takes very little effort to port and maintain. The Java port requires a team of engineers permanently dedicated to it. Also, the huge success of iOS has given Apple the confidence that their approach to working with third-party developers is working out great for everyone. The prospect of Java developers and applications abandoning the Mac is no longer remotely scary for them. Apple have decided they'd rather pay the costs of dropping Java than keep maintaining it.

But what are Apple's critical objectives?

The main theme of the "Back to the Mac" event on October 20 was the idea of moving iPad software and hardware features into the Mac, in a way that made sense for the Mac. My theory is that Apple has now oriented its entire strategy around the iPad. I think Steve Jobs probably regards the iPad has the best thing Apple has ever made. This is because the iPad provides the "purest" user experience: you hold it in your hands, rotate it, move it, directly manipulate the information you're working with via touch, and it dedicates itself to the task at hand. Of course, the iPhone has these characteristics too, but if will forever be limited by the (obviously worthwhile) trade-off of fitting in your pocket. The larger screen size of the iPad allows the touch interface to reach its full potential.

Slight diversion: in the recent financial call, Jobs said that 7" was a lousy form-factor for a tablet. You had better believe that he meant it. I will bet that Apple will never release an iPad with a different form factor. Going smaller puts to many constraints on the options for touch UI, so you end up with a bulky iPhone. Going larger would be nice, but it rapidly becomes unwieldy (the fact the iPad can be easily held and moved around is a critical part of the "experience"). The dimensions were chosen to balance the needs of video with those of e-books. Believe me, Apple chose the form-factor it did very carefully.

And yes, something else Jobs likes about the iPad is that Apple has more control over it. While many ascribe this to cynical motives, I believe Jobs honestly sees this as being good for the customer, and part of the reason for the success of iOS devices. Whether this view is correct will, in the end, be decided in the marketplace.

What is the future for the Mac?

So Apple sees the iPad as the future. But while it might be a "pure" experience, there's obviously a large number of important things that the iPad simply cannot do¹. So Macs aren't going anywhere, but Jobs wants to make them more iPad-like. In the Mac event, Jobs called out what this meant: curated App Store, full screen apps, app "Launchpad", autosave/resume. What is equally interesting is what this implies they are leaving behind. During Craig Federighi's demo of Lion on the 20th, he showed the use of swipe to switch between a) full screen apps b) dashboard c) "the desktop". Perhaps I'm reading too much into this, but what today constitutes "the Mac Desktop experience" seems, in Lion, to be just another full-screen app you can switch to. A Rosetta-like box to run "the legacy stuff". I think Lion could be Apple's first step in really redesigning the Mac from first principles.

This iPad-like approach also reinforces other trends already underway. The traditional document-centric Mac app is basically dead. Most apps will be one-window, "content-centric", and abstract the filesystem completely. And they'll be made with Apple's tools. In other words, iPhoto '11 is the model. It's going to be fascinating to see how far Apple can take this. There have been people saying, literally for decades, that the fundamental desktop concept of movable, resizable, overlapping windows was the worst idea in the history of user interfaces. Will they be proven right? I have no idea. The best part is, as others have pointed out, Windows cannot easily copy what Mac OS X is doing. Greater distinction between the keyboard/pointer OS platforms will be a great thing, in my opinion.

Where does Java fit in all this?

It doesn't. Regardless of whether you think Apple's plans for the Mac are great or terrible, it's clear that heavy-weight "cross-platform platforms" like Java and Flash, which are controlled by others, could hardly be more useless to Apple. They don't interfere with Apple's plans, but they don't help them either. Java and Flash apps don't use compelling OS X technologies like Core Animation. They're completely clueless when it comes to multitouch. Apple only cares about one "cross-platform platform": the light-weight and mobile-friendly HTML5, which they can also influence.

Will Oracle Provide Java on Mac?

I'm not sure, but I think yes. The answer to this question will also answer many questions about what Oracle really has planned for Java and JavaFX. In my opinion, JavaFX has already lost the cross-platform war to HTML5 (I like JavaFX a lot, and would love Oracle to prove me wrong here). Oracle claims to be committed to developing and promoting JavaFX, so it apparently believes it can turn things around. But there's one thing I'm certain about: JavaFX cannot succeed as a credible platform without first-class support on the Mac. If Oracle is actually serious about JavaFX (which is client-side Java now) then it needs to very soon announce that it will be adding Mac as a fully supported platform, by Java 8 at the latest. Bringing Mac support online in a Java 7 update would be better. If Oracle lets Java on the Mac rot, then the death of client-side Java will be certain and final.

Where did it all go wrong for Java?

A few people have pointed out that just 10 short years ago Steve Jobs was proudly proclaiming that the Mac would be the premier platform for Java, and now after years of disinterest Apple is finally giving up on Java altogether. How times have changed. In 2000, Apple desperately needed apps on the Mac - any apps, and of course developers. Now, it's Sun/Oracle that really needs Java on the Mac, and Apple can afford to be indifferent.

Even if Oracle produces a brilliant implementation of JavaFX for the Mac, things do not look great for Java on the client. I won't make this post even longer by discussing the reasons for client Java's failure, but it's not because Apple didn't give Java a fair chance. Speaking as someone who really believed in and supported the "write once, run anywhere" ideal of Java, the fact has to be faced: the dream is well and truly finished. Where Java has failed, HTML5 has succeeded, primarily by keeping its goals modest and being ruthlessly pragmatic.

¹ But I think that we haven't even scratched the surface of what the iPad is capable of. Many of its limitations are "a simple matter of software". I think iOS 4.2 is going to give a big boost to the iPad's utility, as will iOS 5 and beyond. Having said that, there'll always be some jobs that just need a mouse pointer and a keyboard.

Scala's Missing Monad

2010-09-30T20:04:00.000+10:00

I've been writing more and more pure functional code in both Scala and Java recently. An issue I found myself running into quite often is this: say you have a function that sometimes returns no results (e.g. looking up a key in a map). A common way to deal with that in Scala is to return an Option:

Simple enough. But then a typical use of such a function is to map it over all members of a collection:

What I find I often want to do in a situation like this is to only bother with the results if all the applications of foo returned a result. In other words, I want to turn the strings list above into an Option[List[String]], which is Some(List(foo(1).get, foo(2).get, foo(3).get)) if all those applications of foo give values, and None otherwise. I couldn't see any method in the Scala standard library that does what I want.

As a bit of a functional programming n00b, one approach I've found to be useful is to work out the Haskell type signature of the function I want, feed that into Hoogle and see what I get, if anything. In this case, the Haskell type is:

The top Hoogle result for that is the sequence function:

And look, the sequence function works on any monad, not just Maybe/Option. For example, if my foo function had been returning Either[SomeErrorType, String], then the sequence function would give the first error in the list if there was an error, or the list of results if there was no error. So this is a useful function, but as I said, I can't find it in the standard Scala library.

When I thought about how to implement sequence in Scala, I immediately ran into trouble. It is often said that "Scala has monads", and we know that the flatMap method is doing a monadic bind. But this support isn't much more than a convention for method names and type signatures, combined with nice syntax in the form of for expressions. There is no monad type declared anywhere in the standard library. Without a monad type, the sequence function would have to be re-implemented over and over: Option.sequence, Either.sequence, List.sequence, etc.

The good news is that Scala's type system can express the monad type. And the better news is that some really smart people have already done all the work for us in a library called Scalaz. So I had a look, and lo and behold there is the sequence function:

That may look nothing like the Haskell function, but that is mostly a result of how typeclasses are expressed in Scala. Trust me, this is the same function. Also, this Scalaz function is more general than the Haskell function above. It turns out that sequence works for any "traversable" thing (not just lists) containing anything that is an applicative functor (not just monads). Haskell also has the equivalent fully generalised sequence function in the Data.Traversable library.

So what did I learn from this exercise? First that the lack of explicit types for monad, functor and friends in Scala's standard library is a greater problem than I'd expected. Second, if you're writing pure functional Scala code, you should be using Scalaz. Since all Scala programmers should be writing pure functional code, it follows that all Scala programmers should be using Scalaz!

Cocoa vs Haskell

2010-01-08T10:57:00.005+11:00

So here's the deal - if you want to write a Mac GUI app, you want to use the Cocoa framework. As a practical system for writing desktop GUI apps, the Cocoa stack simply has no equal anywhere. Part of what makes Cocoa so great is the Objective-C language, which adds a very flexible and dynamic OOP system on top of standard C.

Objective-C turned out to be a great match for creating desktop GUIs. It's obviously a general-purpose language, but for general programming it would be no where near my first choice. My first choice right now would be Haskell, even though I'm still a beginner and I pretty much suck at it. But I get why Haskell is so good (and so much fun).

So if I want to write a Mac GUI app, can I use Haskell? I've had a play with HOC which is a very neat bit of software that bridges Objective-C to Haskell. You can call Haskell from Objective-C and implement Objective-C classes in Haskell. There are some challenges trying to use HOC—it's not exactly straightforward to get it up and running and integrated into your XCode project; and it's under active, but slow, development and doesn't have a stable release. But at a more fundamental level I've come to the conclusion that, while it's very cool, HOC probably isn't worth it.

If you can get HOC working, you'll have a full-blown bridge that lets you use Cocoa APIs from Haskell. But is that actually worthwhile? The things that Haskell excels at are basically useless when using Cocoa, which is completely based around stateful objects being mutated by events and messages. While you could have all your code—including the Cocoa GUI stuff—in Haskell, the GUI code will all be imperative using the IO monad. All the effort just gets you Objective-C with nicer syntax.

So for Mac apps, I think sticking with Objective-C for the GUI code is the best choice. As a language, it sucks compared to Haskell, but it just works very well with the Cocoa APIs. But what about using Haskell to implement the Model and do other computational heavy-lifting under the hood? This turns out to be much simpler than a full Objective-C bridge. GHC of course offers a standardised foreign function interface to C out of the box, and Objective-C is a superset of C. So you can call Haskell from Objective-C code just as you would from plain C, and it works quite naturally.

But it's a bit trickier to build a complete Cocoa application that does this. There is a HaskellWiki page that describes one approach to getting a complete XCode project set up. I tried it out and it works, although things don't integrate very well. You have to compile the Haskell code using ghc outside of the XCode build process, and you have to manually track down all the libWhatever.a files that your Haskell code needs to link against. I've made a simple demonstration project for XCode 3.2 that works for me on OS 10.6: http://github.com/quelgar/HaskellCocoaDemo. This is basically the classic Currency Converter app from Apple's Cocoa tutorial, but it does the currency calculation in Haskell. Here's the rough list of steps I performed, starting from the standard XCode template for a Cocoa Application:

In the project build settings:

Set the architecture to just "i386". GHC is today only built for 32-bit Intel on Mac
Add "-liconv" to "Other Linker Flags". GHC libs need it.
Add your GHC include directory to "Header Search Paths". For GHC 6.12.1 on my system, this was "/Library/Frameworks/GHC.framework/Versions/Current/usr/lib/ghc-6.12.1/include/".
Add "/Library/Frameworks/GHC.framework/Versions/Current/" to the "Library Search Paths" and mark it recursive. Not sure if this is actually necessary since we end up adding all the libraries to the project manually anyway.

The Haskell function that will be called from C is in HaskellConvert.hs, which uses the typical FFI stuff. You compile this with GHC like so: "ghc -c HaskellConvert.hs". This outputs various files, of which the following have to be added to your XCode project:

HaskellConvert.o
HaskellConvert_stub.o

You now have to add all the lib*.a files from GHC that your Haskell code needs to link successfully. The wiki page I linked to above describes one way of doing this. It's not pretty, but it works. Once you know you need a particular .a file, just drag it from the finder to the project, and accept all the defaults in the add file dialog. It will be added to the link stage automatically.
Modify main.m to perform the Haskell runtime setup and teardown.
In this demo, Haskell is called from Converter.m, so this file needs #import "HaskellConvert_stub", and then just make the call to Haskell.

Probably the biggest downside of this plain C approach vs HOC is that you can't pass Objective-C objects to Haskell (well, you can, but they are not easy to use from Haskell, to say the least). So you have to convert whatever Objective-C data structures you're using in the GUI to plain C types for interfacing with your Haskell APIs. Haskell wrappers for the OS X C APIs for CoreFoundation and the Objective-C runtime might make using Objective-C objects from Haskell feasible without a full language bridge, but I'm not sure about this.

The GHC installer for Mac installs the bulk of GHC under /Library/Frameworks/GHC.framework, but I don't think it actually works as a Mac framework. I could be doing something wrong, but adding it to XCode as a framework didn't work (XCode just said it wasn't found when I tried to build, from memory). It would be great if GHC could be made a fully-functional Mac framework, as we would no longer need to manually set include and lib search paths, or explicitly add every Haskell .a file to the project. But to make this useful, I think we'd need some automated way of building any Cabal package as a framework. I'm not sure if that's feasible.

Some simple Scala null tricks

2009-12-15T23:12:00.006+11:00

Scala code often needs to deal with null references, unfortunately. Such is the price of Java interoperability. One of my favourite utility methods is:

Scala 2.8 offers this using Option(ref) instead of ?(ref). However, I can't figure out why the Scala 2.8 method accepts non-reference values as well.

Another simple thing that may come in handy is an extractor for non-null values:

What's the advantage of using an extractor? Say you have a collection of references, some of which may be null, and you want to ignore the null references:

The FSF doesn't care if they're lonely

2009-07-06T12:45:00.004+10:00

Daniel Jalkut, a leading light of the Mac developer community recently blogged about the GPL and some of consequences of it for collaborative software development. I agree with what he says, and I dislike how the GPL limits collaboration in a number of ways. But I thought it might be useful to go over the background and objectives of the GPL.

There's an important thing to remember when talking about the GPL: the impact, good or bad, on the practicalities of software development have nothing to do with its objectives. The authors of the GPL, the Free Software Foundation just aren't concerned by the kinds of issues that Daniel raises. The fundamental principle of the FSF is that anyone should have complete freedom to study, modify and share all computer software. The end-game of the FSF is that all restrictions that apply to software, such as copyrights and patents, be abolished altogether (in so far as they currently apply to software). In other words, they want any software written by anyone to have no legal protection from examination, modification and redistribution by anyone else. The FSF really wants all software to be in the public domain, without the option of copyright protection.

The GPL is their attempt to bring this about given the reality that copyright of software exists today and shows no signs of going away. The GPL is designed to use the copyright system against itself; the FSF calls this "copyleft". In typical commercial software, the copyright is used to prevent the licensee (user) of the software from doing things with it that the author doesn't like (such as modification or redistribution). The GPL instead prevents the licensee from adding any new restrictions. The end result of putting software under the GPL is basically the same as if copyright protection for that software didn't exist at all.

Now, you can make many arguments pointing out the practical downsides of removing copyright restrictions on software (whether by abolishing software copyright, or via the GPL). Daniel's post is a good example of this. But to the FSF, these considerations are secondary to the principle that any restrictions to what users can do with their software is morally Wrong. You can see an example of this in the GNU Manifesto where they acknowledge that adopting their policies could result in programmers earning less. Their view seems to be that doing the "right thing" can have effects that aren't necessarily desirable, but we just have to live with as best we can. Perhaps a good analogy is free speech, which necessarily increases the risk of children accessing pornographic or violent material.

I try to take a pragmatic "what best serves the common good?" approach to matters, which leads me to agree with the FSF with regard to software patents, but disagree with them on copyright. I think patents, when applied to software, are all harm and no benefits. For copyright, I can see greater upside than downside to the current system. It's true that without software copyrights we would all have greater freedom regarding how we can use our computers, but it's also likely that our computers would be of much less practical use to us. Do professional photographers and artists prefer GIMP or Photoshop?

Just to be clear, arguing over the implications of the GPL on software development practice is a very useful thing to do. Many people who don't care about the political position of the FSF still chose to use the GPL simply because it happens to suit their needs, and there's nothing wrong with that. But having a clear understanding of the costs and benefits is essential.

Project Euler Problem 8

2009-06-01T23:11:00.006+10:00

I guess I should warn that these Project Euler posts have spoilers, in case you want to try the problems yourself ☺. My Scala solution to problem 8:

val input = "73167176531330624919225119674426574742355349194934" +
"96983520312774506326239578318016984801869478851843" +
"85861560789112949495459501737958331952853208805511" +
"12540698747158523863050715693290963295227443043557" +
"66896648950445244523161731856403098711121722383113" +
"62229893423380308135336276614282806444486645238749" +
"30358907296290491560440772390713810515859307960866" +
"70172427121883998797908792274921901699720888093776" +
"65727333001053367881220235421809751254540594752243" +
"52584907711670556013604839586446706324415722155397" +
"53697817977846174064955149290862569321978468622482" +
"83972241375657056057490261407972968652414535100474" +
"82166370484403199890008895243450658541227588666881" +
"16427171479924442928230863465674813919123162824586" +
"17866458359124566529476545682848912883142607690042" +
"24219022671055626321111109370544217506941658960408" +
"07198403850962455444362981230987879927244284909188" +
"84580156166097919133875499200524063689912560717606" +
"05886116467109405077541002256983155200055935729725" +
"71636269561882670428252483600823257530420752963450"

val digits = input.map(_.asDigit).toArray

def multiply(index: Int) = digits.slice(index, index + 5)
    .foldLeft(1)(_*_)

val multiples: Stream[Int] = {
    def rec(n: Int): Stream[Int] = Stream.cons(multiply(n),
        if (n > digits.length - 5) Stream.empty else rec(n+1))
    Stream.cons(mult(0), rec(1))
}
println(Iterable.max(multiples))

For convenience, this specifies the input as a string, then uses RichChar.asDigit to create a corresponding array of integers. The multiply function uses left fold to multiply together sequences of five digits. The multiples value is a stream (lazy list) of the multiples of all groups of five consecutive digits from the input. And finally, the Iterable object provides a convenient method to find the maximum value in any Iterable containing Orderable things.

Project Euler in Scala

2009-06-01T17:19:00.008+10:00

Project Euler is a bunch of mathematical/programming problems. I've been trying to improve my Scala skills by using it on some of the Euler problems. I'm trying to use functional programming styles as much as possible. My solutions definitely haven't been very optimized, but I am learning, which is the aim.

Euler Problem 1

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23. Find the sum of all the multiples of 3 or 5 below 1000.

My solution:

def sum(nums: Iterable[Int]): Int = (0 /: nums)(_ + _)

println(sum((3 until 1000).filter(i => i%3 == 0 || i%5 == 0)))

The 3 until 1000 creates a "range", which is a lazy sequence of numbers from 3 to 999 inclusive. Because it's lazy, you don't get 997 integers sitting around in memory, you only get those integers which pass the filter. The filter itself shows the Scala syntax for a "lambda expression" (anonymous function). The full syntax is (arg1, arg2, ..) => function body, but when you only have one argument (a single Int, in this case) you can leave out the brackets.

The sum function is the operator syntax for a left fold over nums. The _+_ is an even more terse form of a lambda expression that you can use for really simple expressions. The first underscore is replaced by the first argument to the function, and the second is replaced by the second argument.

Logging vs String Construction

2006-10-17T14:30:00.001+10:00

A classic logging problem in Java is this:

logger.finest(expensiveStringConstruction());

The problem with this is that even when the logger's level is above FINEST and the message is not logged, the message string must still be constructed. In cases where constructing the string involves a lot of work, your code can be slowed down significantly even when logging is off. The standard solution is like so:

if (logger.isLoggable(Level.FINEST))
{
  logger.finest(expensiveStringConstruction());
}

By adding the if statement around the logging, the expensive string construction is only done if the message is actually going to be used. The problem with this is that if you have a lot of logging statements, your code soon ends up littered with these ifs that just obscure what is really happening. Not to mention that you have to type it all in.

This problem could have been tackled in a different way. It was a mistake for the logging methods to take String objects as parameters. This mandates that the messages must be constructed before passing them to the logger. A better approach is to delay creating the message string until the last possible moment, when we know it will be needed. One option would be to have an interface called, say, LogMessageFactory with a method String createLogMessage() and have the log methods accept the interface instead of String. But a new interface isn't necessarily needed, as good old Object already has a toString method. If loggers took Object's instead of String's, then they could just call toString when logging actually takes place.

Well, the logging API actually does let you do this. LogRecord objects can store a message string and an arbitrary set of parameter objects. The standard formatters will use MessageFormat to construct the string that is finally logged. Here's an example:

public final class TestLog
{

    public static final Logger logger =
        Logger.getLogger("test");

    public static void main(final String[] args)
    {
        final TestLog testLog = new TestLog();
        logger.finest(testLog.toString());     // 1
        logger.log(Level.FINEST, "{0}", testLog);   // 2
    }

    public String toString()
    {
        try
        {
            // expensive string construction here
            Thread.sleep(5000L);
        }
        catch (InterruptedException e)
        {
            assert false;
        }
        return "Hello, world";
    }

}

If the logger is configured to log at FINEST or below, the lines marked 1 and 2 behave identically and the message is logged. However, if the logger's level is above FINEST, then line 1 will take 5 seconds to do nothing and line 2 will just check the level and exit, without calling toString at all.

The advantage of logging this way is that you don't have to put all those ifs statements in. But there are disadvantages. You can't use the convenience logging methods like severe and info, etc. You have to use one of the log methods that accept Object parameters. Unfortunate, but not too egregious.

The bigger disadvantage is that the string generation has to be inside a toString method. While this can be done inside an anonymous inner class, the syntax for that isn't exactly convenient. If you have a bunch of log messages that are generated completely differently, then using anonymous inner classes everywhere will be far worse than just using the if statements. Where it would make sense is if you have a bunch of log statements where the expensive portion of the message creation is the same.

For example, you could replace this:

if (logger.isLoggable(Level.FINEST))
{
   logger.finest("Stuff happened " + expensiveStringCreation());
}
if (logger.isLoggable(Level.FINEST))
{
   logger.finest("Foo happened " + expensiveStringCreation());
}
if (logger.isLoggable(Level.FINEST))
{
   logger.finest("Bar happened " + expensiveStringCreation());
}

with:

final Object msgFactory = new Object()
{
    public String toString()
    {
        return expensiveStringCreation();
    }
};
logger.log(Level.FINEST, "Stuff happened {0}", msgFactory);
logger.log(Level.FINEST, "Foo happened {0}", msgFactory);
logger.log(Level.FINEST, "Bar happened {0}", msgFactory);

I'm assuming here that there's code in between these log calls that changes the state and hence the string returned by expensiveStringCreation.

There's a lot of talk at the moment about adding closures to Java. Whatever ends up being adopted, this is definitely one of the scenarios it should assist with (although what I'm describing only requires the most basic capabilities of closures). I'm a fan of the CICE proposal, although some of the objections raised seem valid. The CICE proposal requires an interface with a single method, so if we did have the LogMessageFactory interface I proposed above, we could just do:

logger.finest(LogMessageFactory()
    { return "Stuff happened " + expensiveStringCreation(); });

XML vs Streams

2006-09-25T22:21:00.000+10:00

Ever get the feeling that XML sometimes makes things harder than they need to be? One of the problems that I and others have run into is that when parsing XML using SAX (or, in my case, using JAXB to unmarshall XML, which uses SAX under the hood) from an InputStream, the stream is always closed after the parsing is complete. This is surprising behaviour in Java, because you're not supposed to close streams you don't own. This can be a problem when using network sockets or when you want to write two or more unrelated XML documents to the same stream.

The root cause of this isn't anything in Java or SAX, it's the XML specification itself. There's simply no way to tell when an XML document ends. Every well-formed XML document will have an end tag, so you'd think the parser could simply stop when it reached the end tag. However, the XML specification allows for an arbitrary amount of comments and processing instructions to follow the main element. So when reading from a stream, the parser has no choice but to keep reading until it gets to the end of the stream. After reading to the end of the stream, SAX calls close on it. I believe the reason for this is that the stream has been completely read anyway, so there's no point keeping it open.

I've had a couple of cases where I've wanted to write two or more independent XML documents to the same stream. In one case, I had a thread producing XML documents and writing them to a PipedOutputStream. Another thread would read from the corresponding PipedInputStream. The second case was a save file for an application that happened to include two unrelated XML documents. Because of the "uncertain ending" feature of the XML spec, neither of these things are possible.

Not unless you use customised input and output streams, that is. After some searching, I found some classes from Xerces called WrappedOutputStream and WrappedInputStream that allowed me to do what I wanted. They actually solve the more general problem of reading data of unknown length without reading too much. I ended up writing my own implementations, but I used the same "protocol" as these classes. I can now read and write XML over streams without having to use a new stream for each document.

Java's clone mechanism is teh suq

2006-03-26T22:06:00.000+11:00

Doing cloning properly in Java is way harder than it should be. I thought I knew all the traps, but recently found yet another. In a nutshell, making inner classes cloneable is almost always a bad idea. The reason is that inner classes have an internal reference to an instance of their enclosing class, which is implicitly final and can't be changed as part of a clone. For example:

class Outer implements Cloneable
{
    int i = 0;
    Inner inner = new Inner();

    public Object clone()
    {
        try
        {
            Outer clone = (Outer)super.clone();
            clone.inner = (Inner)inner.clone(); // A.
                 Cloned inner still refers to this Outer instance
            return clone;
        }
        catch (CloneNotSupportedException e)
        {
        }
    }

    class Inner implements Cloneable
    {
        int j = 0;
        
        public Object clone()
        {
            try
            {
                return super.clone();  // B. WARNING!
                      - this does the Wrong Thing
            }
            catch (CloneNotSupportedException e)
            {
            }
        }

        void inc()
        {
            i++;
            j++;
        }
    }
}

This code is attempting to do a deep clone of an Outer instance, which requires a deep clone of the Inner it references. The problem occurs at B, because the object returned by super.clone refers to the same Outer instance as the original. So if you do this:

Outer outer1 = new Outer();
Outer outer2 = (Outer)outer1.clone();
outer2.inner.inc();  // will change i in outer1 instead of outer2
System.out.println(outer1.i);
System.out.println(outer2.i);

You get:

1
0

Because the cloned outer2.inner has an internal reference to the original outer1 instance. There may be cases where you want it to work that way, but it seems very unlikely. So what to do? Calling super.clone() from an inner class is basically broken, and there's no way to fix it. If you're able to make the inner class final, this can be avoided relatively easily by using a copy constructor to clone the inner class instead of Object.clone:

class Outer implements Cloneable
{
    int i = 0;
    Inner inner = new Inner();

    public Object clone()
    {
        try
        {
            Outer clone = (Outer)super.clone();
            clone.inner = clone.new Inner(inner);  // C.
                  Use "qualified" copy constructor
            return clone;
        }
        catch (CloneNotSupportedException e)
        {
        }
    }

    final class Inner  // no longer Cloneable
    {
        int j = 0;
        
        Inner(Inner inner)
        {
            j = inner.j;
        }

        void inc()
        {
            i++;
            j++;
        }
    }
}

This gives the desired output of:

0
1

Check out the change at line C, it's actually valid syntax. It's called a "qualified" new, where you explicitly specify the enclosing instance of the new inner class instance.

Using copy constructors for cloning is only OK for final classes, because they aren't polymorphic. If we had sublcasses of Inner involved here, then it gets harder - Object.clone is the only easy way to get the right class instantiated. Idea will actually give a warning on line C for this reason.

Another, possibly better, solution is to make Inner a standard (non-inner) class with an explicit, non-final reference to its Outer instance. Basically, do the inner class stuff by hand. Because the reference to the outer class isn't final, it can be fixed up before returning from clone(). Of course, this lacks to convenience that was the motivation for inner classes in the first place.

More info on Java's cloning woes: http://www.adtmag.com/java/articleold.asp?id=364.

Handling InvocationTargetException

2005-12-23T00:07:00.001+11:00

A quick hint on handling java.lang.reflect.InvocationTargetException.

This sucker is thrown whenever you invoke a method via reflection (typically via Method.invoke), and that method throws any sort of exception. So it doesn't indicate a problem with the reflection mechanism, it is simply how the reflection mechanism notifies you of exceptions thrown by the method you called.

Whenever you catch this exception (and you will have to catch it at some level), the first thing you should do is call getCause(). This returns the exception thrown from the method that was invoked via reflection, and that's what you care about. So whenever you need to log or report an occurrence of InvocationTargetException, always use the cause object rather than the InvocationTargetException itself.

How not to catch Exceptions

2005-12-22T23:37:00.001+11:00

When it comes to bad Java practices, top of my list would have to be catching exceptions like this:

try
{
   // stuff - anything really
}
catch (Exception e)
{
   // maybe log it if you're lucky
}

This is usually done because the "stuff" throws more than one kind of checked exception, and the code to handle each case is the same. So what's wrong with it? The biggest problem is that this not only catches the checked exceptions you're interested in, but also all RuntimeExceptions. The vast majority of RuntimeExceptions are only thrown from buggy code, and catching them is rarely a good idea. The above catch handles all kinds of exceptions the same way, regardless of whether it was caused by a bug or an ordinary failure that can happen from time to time when a program is running (e.g. a network failure). This only obscures bugs.

Consider the case where a bug causes a NullPointerException to be thrown - perhaps the most common type of bug in Java. If the catch clause simply logs the exception message (not uncommon), then you have to check the log to have any idea why your app is misbehaving. And if you don't log the stack trace (again, not uncommon), you still won't have a clue what caused it. A much better option is to simply let RuntimeExceptions go, which produces a screeching halt in your app and, in most cases, an immediate stack trace dump telling you exactly what happened. The details of what happens when you don't catch a RuntimeException at all will vary depending on the environment you're running in, but it's always safe to do so. For example, servlet containers usually output the stack trace in the HTTP response when a servlet throws a RuntimeException.

A good principle of catching exceptions is to only catch the those specific exceptions that you can actually deal with. E.g.

try
{
   // stuff that throws AException
   // stuff that throws BException
}
catch (AException e)
{
    // deal
}
catch (BException e)
{
    // deal
}

This way just deals with the exceptions that you can sensibly recover from and lets any RuntimeExceptions that indicate bugs to be thrown to the caller. If the code to handle both AException and BException is the same, you could always move it to a method.

"But what if the code throws, like, 10 different kinds of checked exceptions?". I've found it's actually pretty rare to have a block of code that throws more than about 4 unrelated checked exceptions. Some of the reflection APIs are pretty bad with this, however. It can be a pain to enter basically identical catch clauses for every one. A good alternative is:

try
{
   // stuff that throws AException
   // stuff that throws BException
   // stuff that throws CException
   // stuff that throws DException
}
catch (RuntimeException e)
{
   throw e;
}
catch (Exception e)
{
   // deal with all checked exceptions in the same way
}

This is actually not a bad way to get the best of both worlds (although I usually find it clearer to just handle each checked exception individually).

I think in hindsight, the inheritance hierarchy of exceptions is backwards. Instead of the RuntimeException extends Exception extends Throwable hierarchy, it should have been CheckedException extends UncheckedException extends Throwable. That would allow catching all checked exceptions in a single catch clause, while not obscuring unchecked exceptions thrown by buggy code.

In a similar vein, you almost never want to declare a method with throws Exception. This is basically throwing away the whole checked exception system and saying "this method can throw any kind of exception". This tends to make exception handling in the caller painful, and may give the caller no good option but to declare throws Exception itself. Before long, you end up with all your methods declared as throwing Exception and you have little indication of what exceptions you might actually have to deal with at runtime. (There is an argument to made that checked exceptions are a bad idea altogether, but that subject will have to wait for another day.)

There are a few cases where you might have a legitimate reason to catch Exception, or Error, or Throwable. Here are some of them:

When implementing Runnable. Reporting RuntimeExceptions thrown from Runnable.run is a bit of a pain in Java versions 1.4 and earlier. It's often easier just to catch them yourself and log the entire stack trace somewhere it can't be missed. Java 5 has a nicer mechanism for doing this, however.
When calling some sort of plug-in module. In this case, a bug in the plug-in doesn't mean the whole application needs to abort. You can catch the exception, notify the user, unload the plug-in and move on.
When dealing with poorly designed APIs that throw Exception.