Turn the question on it's head and make up a story where the math matches the observation in every circumstance. If you can, and I'm not sure I could, work backwards from there to find the breaking point. Or just remember that the map is not the territory, the finger that points to the moon is not the moon, and get on with things.

The description of the case doesn't make sense to me, either. But I'm not having an easy time imagining what the philology example would be like, which makes me worry that there might be something specific about the philology example that would affect things. I presume there's a reason you're avoiding giving too many details on purpose, but if you only reconstruct the features of the case that you consider relevant, and the case doesn't make sense to you, it may not be very revealing that the case doesn't make sense to others; the problem may be that something you are treating as irrelevant and not mentioning actually matters.

I mean, if I imagine that we're looking at words in a body of literature, and horses are word A and zebras are word B, and the argument is intended to show that word B was actually in use in that body of literature (as opposed to only appearing as a result of slips of the pen, copyist errors, etc.), then I can't see that this statistical argument proves anything; what we'd really want is some data about how common such errors are and what forms they usually take, in order to determine how many errors would be likely to arise by chance. Comparing to a hypothetical no-error case seems, as you say, a red herring, entirely and bizarrely beside the point. But this is also so obvious that I find it hard to imagine that anything like this was the original argument. Perhaps I am being too charitable.

The main reason I posted this is that I am sometimes wrong about things. Maybe the zebra example turns out to make sense in some way I hadn't thought of. Maybe Yagami is using some sort of standard method. Maybe there's some failure mode I haven't thought of. It would be really good to know this before I make an ass of myself with the review. And talking about asses - there are some wild asses in Mongolia which got left out of my parable - but they're kind of cute so here is a link.

You seem to say that the difficult part of driving is staying in the lane. That's by far the easiest part of driving, both for humans and computers.

Humans can hundred of thousand years of evolution to get walking right, I don't consider it to be that exiting as something like operating machinery like cars, bikes and airplaines.

Here is my understanding of the issue.

Traffic works well because driving is one of the activities that is relatively easy to internalize and perform as a matter of habit, like walking or riding a bicycle. System 1 thinking is usually fast, predictable and reliable. You don't need to "focus intently". It is true that you cannot take your eyes off the road (or, more accurately, shift your passive attention (see http://www.cdl.org/resource-library/articles/attention2.php) away) for longer than a brief instant, otherwise the subconscious feedback loop which includes visual and auditory inputs, your brain, your extremities and the vehicle breaks down, not because you have to consciously "focus intently". The "magnets in the road" are built into the area of your brain that controls your reflexes. That's why learning safe driving habits is so important.

Not everyone is equally capable of internalizing the driving process, or put in enough time to do so. These people white-knuckle it, constantly engaging their full slow and unsuitable System 2 in the loop, and consequently they find the normal driving activity exhausting, rather than relaxing. It is often easy to notice these drivers by their slow reaction to traffic lights, overly careful driving style and by generally overcompensating for road hazards. After all, System 2 is taxing, slow and unreliable. Ironically, these people may be safer drivers overall, since they never let their attention wander.

This passive attention escalates to active when something unusual happens, like when you hear emergency vehicles and have to make decisions which are not internalized. Or when some other car wanders into your lane, or if the traffic stops unexpectedly. Anything that either breaks the passive attention loop or causes this escalation from passive to active attention to slip is dangerous: texting, arguing, listening to a child fussing in the back, concentrating on a phone call.

In contrast, there are activities which do not naturally lend themselves to internalizing. Programming is one of them. Even after doing it for decades, people are still as consciously engaged in it as they did in the beginning. Thus your hope for a safe AGI "by analogy with safe driving", seems misplaced to me.

Indeed. The problem of driving has been set up by car designers so that it should be easy for us to solve it. By contrast, the problem of creating a safe AGI has not been set up so that it should be easy for us to solve (if you don't believe that there is a God who has made this the best of all possible worlds that is...). So I don't think the analogy works. If you want to make an argument making use of human past performance, it would be better to use examples of problems (that we've solved) that weren't set up by us (e.g. moon-landing, relativity theory, computers, science in general, etc).

I'm occasionally still amazed that traffic works as well as it does. I must say I'm hesitant at using this example to claim that people are more capable than you might think. Driving is just something humans happen to be competent at. There are plenty of things roughly as complicated as driving a car that people aren't surprisingly good at.

This also reminded my of something people said at the latest meetup. At least two people told me they had deliberately tried to get more scared of driving, because they had noticed they had less fear in a car than on a plane despite planes being safer.

Good post - this has struck me too. By the way, this is a good example showing that social life and human behaviour in general is much more "law-like" and indeed predictable than many "anti-positivists" in the social sciences would have it. Car drivers' behaviour is remarkably regular and predictable, even though it is, as you say, in no way trivial to drive a car.

Still many mistakes which end up causing accidents are made, and thus I'm sure automatized or semi-automatized cars could decrease the number of accidents hugely.

What you effectively mean is that most people don't reach the edge described very nicely visually here:

http://www.businessinsider.com/the-illustrated-guide-to-a-phd-2012-3