Strongly agree with "programming", but strongly disagree with "learning an instrument". I wasted lots of time on that. The costs are high, and all the benefits are either dubious or can be more cheaply obtained through other means.

If you want to play with a (rather tame, since it doesn't always use its regeneration powers) Bucholz Hydra, here's a link for you: http://www.madore.org/~david/math/hydra.xhtml

For my part, I knew about hydra games and had forgotten the name, but the context made it fairly obvious that this was a joke about the hydra being so hard to kill that you can't prove you do it with only Peano arithmetic.

... Huh. I didn't even consider the possibility that it might be an Inferius before now - I just assumed it was Imperius.

I took the survey (answered nearly everything).

A program this simple? Yes.

[EDITED to add: And I did say to test it on the primes up to 100.]

[EDITED again to add ...] Here, just for reference, is what I would write, in less than a minute, to do the job. It is only intended to work for integers >= 2, and need not be bearably efficient for integers that aren't very small.

def is_prime(n):
for i in range(2,n):
if n%i == 0: return False
return True

Four short, simple lines corresponding quite exactly to the definition of primality.

So, what could be wrong in this program that could make it give a wrong answer for 1159? I can think of the following things. (1) I could have messed up the range of values of i to test against. (2) I could have got my variables muddled up, testing i%n instead of n%i or something of the kind. (3) I could have got my return conditions backwards, making this a not-prime test. (4) I could have messed up the control flow in some way that, e.g., does something crazy when we fall off the end of that loop. (5) I could have mixed up the % (modulus) operator with something else like division or exclusive-or. (6) I could have got myself confused about what the condition for primality actually is. (7) I could have done something else that I haven't been able to think of.

Well, most of those are very low-probability, especially given that I've listed them explicitly and checked the program. Further, it's easy to see that all of them would almost certainly fail on testing against the numbers from 2 to 100, and it seems likely that most errors in category 7 would do so too. (E.g., one can imagine messing up the test in such a way that numbers of the form prime^2 get mistakenly identified as prime, or so that it only catches primes of the form 4k+1, or something.)

And, lo and behold, I then did

print [p for p in range(2,100) if is_prime(p)]

and got

[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]

which has 25 entries (which I know, or think I know, is the right answer), consists entirely of numbers I believe I know to be prime, and includes every number < 100 that I believe I know to be prime.

I am very comfortable indeed reckoning this as at least 1000:1 evidence that any specific positive integer on the order of 1000 is prime iff my is_prime function says it is and gives the same result on repeated runs.

Other failure modes I haven't listed above but want to mention that I have thought of: There could be integer overflow problems. (But not for numbers this small, and not in Python unless there's a weird Python bug I haven't heard of.) There could be ridiculous errors in Python itself that make it get simple arithmetic on small numbers wrong. (But I'm running Python 2.7.3, which was released some time ago and was for some time the most commonly used Python version; I would surely have heard if it had that sort of error; in any case that class of error is very, very rare.) My computer could be messed up by cosmic rays or something. (Which is why I'd run the program multiple times on 1159.)

Or, of course, my brain could be messed up in some way that specifically interferes with my understanding of prime numbers or of what it takes to check a program's correctness. The first is already dealt with in the discussion above. The second is one of the reasons why I would also check against someone else's list of smallish prime numbers and do a number-theoretic test by hand. (And also, I think, rather improbable.)

I don't like your use of the word "probability". Sometimes, you use it to describe subjective probabilities, but sometimes you use it to describe the frequency properties of putting a coin in a given box.

When you say, "The brown box has 45 holes open, so it has probability p=0.45 of returning two coins." you are really saying that knowing that I have the brown box in front of me, and I put a coin in it, I would assign a 0.45 probability of that coin yielding 2 coins. And, as far as I know, the coin tosses are all independent: no amount of coin toss would ever tell me anything about the next coin toss. Simply put, a box, along with the way we toss coins in it has rather definite frequency properties.

Then you talk about "assigning probabilities to each possible probability between 0 and 1". What you really wanted to say is assigning a probability distribution over the possible frequency properties.

I know it sounds pedantic, but I cringe every time someone talks about "probabilities" being some properties of a real object out there in the territory (like amplitudes in QM). Probability is in the mind. Using the word any other way is confusing.

It just occurred to me that we may be able to avoid the word "intelligence" entirely in the title. I was thinking of Cory Doctorrow on the coming war on general computation, where he explain unwanted behaviour on general purpose computers is basically impossible to stop. So:

Current computers are fully general hardware. An AI would be fully general software. We could also talk about general purpose computers vs general purpose programs.

The Idea is, many people already understand some risks associated with general purpose computers (if only for the various malware). Maybe we could use that to draw attention to the risks of general purpose programs.

That may avoid drawing unwanted associations with the word "intelligence". Many people believe that machines cannot be intelligent "by definition". Many believe there is something "magic" between the laws of physics and the high-level functioning of a human nervous system. They would be hard-pressed to admit it outright, but it is at the root of a fundamental disbelief of the possibility of AI.

As for actual titles…

• The Risks of General Purpose Software.
• General Purpose Computers can do anything. General Purpose Programs, will. (Sounds better as a subtitle, that one.)

(Small inconvenience: phrasing the title this way may require to touch the content of the book itself.)

Possibly emphasize 'risk' as opposed to 'danger'? "The Risks of Artificial Intelligence Development"? "Risks from the Development of Superhuman AI"?

That only tells you that if you just rely on the scientific method, it won't result in only benevolent knowledge. You could use another method to filter for benevolence.

If contraception is significantly less widespread among wizards than among muggles, then considering their quality of medical care, their population seems anomalously low.