I think I may have misread your previous statement then:

I do not claim that the existence of an orderly universe is undeniable proof of the existence of God; I simply claim that it is evidence which suggests that the universe is planned, and therefore that there is (or was) a Planner.

But since you agreed that the Planner is basically God, I read that sentence as saying,

I do not claim that the existence of an orderly universe is undeniable proof of the existence of God; ... it is evidence which suggests that the was planned by a God.

Is the only difference between the two statements the "undeniable" part ? If so, then I get it.

Surely it depends more directly on the value of P(Agent is intelligent|Agent prefers sequential numbers)?

My point was that it's possible that any intelligent agent who developed via some form of evolution would be more likely to prefer sequential numbers, merely as an artifact of its development. I'm not sure how likely this is, however.

I think I see what you're saying -- discovering that something as unlikely as a GLUT actually exists would shake your beliefs in pretty much everything, including the Turing Test. This position makes sense, but I think it's somewhat orthogonal to the current topic. Presumably, you'd feel the same way if you became convinced that gods exist, or that Pi has a finite number of digits after all, or something.

I agree that this is a reasonable conclusion to make once you assume the existence of a certain kind of deity.

Yes. I do not claim that the existence of an orderly universe is undeniable proof of the existence of God; I simply claim that it is evidence which suggests that the universe is planned, and therefore that there is (or was) a Planner.

Wait, isn't the Planner basically God, or at least some kind of a god ?

However, in a biased, unfair lottery, in which the result is predetermined by an intelligent agent, the sort of patterns that might appeal to an intelligent agent (e.g. 1, 2, 3, 4, 5, 6) are more likely to turn up.

That would be an interesting test to run, actually, regardless of theism or lack thereof: are sequential numbers more likely (or perhaps less likely) than chance in our current American lottery ? If so, it would be pretty decent evidence that the lottery is rigged (not surprising, since it was in fact designed by intelligent agents, namely us humans).

So P(bias|(1, 2, 3, 4, 5, 6)) > P(bias|(35, 3, 19, 45, 15, 8)).

That depends on the value of P(Agent prefers sequential numbers|Agent is intelligent).

In any case, are sequential numbers more likely to turn up in sequences that are not directly controlled by humans, f.ex. rolls of reasonably fair dice ?

Religious people do this all the time; they call it "fulfilling prophecies". Atheists usually discover that such prophecies are hopelessly vague, but theists disagree; they believe the prophecies to be quite specific; or, at least, specific enough for their purposes, given the fact that their God obviously exists.

...and that vast surprise colors all of my intuitions about interactions with nominally intelligent systems. Given that, it's not clear to me why I should keep believing that I was having an intelligent conversation a moment earlier.

What do you mean by "intelligent conversation" ? Do you mean, "a conversation with an intelligent agent", or "a conversation whose contents satisfy certain criteria", and if so, which ones ? I'll assume you mean the former for now.

Let's say that you had a text-only chat with the agent, and found it intellectually stimulating. You thought that the agent was responding quite cleverly to your comments, had a distinct "writer's voice", etc.

Now, let's imagine two separate worlds. In world A, you learned that the agent was in fact a GLUT. Surprised and confused, you confronted it in conversation, and it responded to your comments as it did before, with apparent intelligence and wit. But, of course, now you knew better than to fall for such obvious attempts to fool you.

In world B, the exact same thing happened, and the rest of your conversation proceeded as before, with one minor difference: unbeknownst to you, the person who told you that the agent was a GLUT was himself a troll. He totally gaslighted you. The agent isn't a GLUT or even an AI; it's just a regular human (and the troll's accomplice), a la the good old Mechanical Turk.

It sounds to me like if you were in world B, you'd still disbelieve that you were having a conversation with an intelligent agent. But in world B, you'd be wrong. In world A, you would of course be right.

Is there any way for you to tell which world you're in (I mean, without waterboarding that pesky troll or taking apart the Chinese Room to see who's inside, etc.) ? If there is no way for you to tell the difference, then what's the difference ?

By the way, I do agree with you that, in our real world, the probability of such a GLUT existing is pretty much zero. I am merely questioning the direction of your (hypothetical) belief update.

But leaving that aside, if faced with something I somehow am convinced is a GLUT Chinese Room ... I would probably conclude that the entire conversation up to that point has been devoid entirely of any true understanding.

I wouldn't; or, at least, not necessarily.

Just for simplicity, let's say we the agent and the experimenter are conversing via text-based chat. The agent and the experimenter take turns outputting a single line of text (it could be a very long line); the experimenter always goes first, saying "hello".

In this case, the agent's side of the conversation can be modeled as the function F(H(t), t) where t is the current line number, and H(t) is the sequence of inputs that the agent received up to point t. Thus, H(0) is always {"hello"}, as per above. H(2) might be something like {"hello", "your name sounds funny"}, etc.

We know that, since F is a function, it is a relation that maps each possible input to a single output -- so it's basically a lookup table. In the ideal case, the number of possible inputs is infinite (trivially, we could say "hello", "helloo", "hellooo", and so on, and infinitum), and thus the lookup table would need to be infinitely large. However, we humans are finite creatures, and thus in practice the lookup table would only need to be finitely large.

Of course, practically speaking, you'd probably still need a storage device larger than the Universe to encode even a finite lookup table of sufficient size; but this is a practical objection, which does not a priori prohibit us from implementing a Turing-grade agent as a GLUT.

Anyway, this is getting into GAZP vs. GLUT territory, and that's being covered elsewhere in the thread.

There are already quite a few comments on this post -- do you have a link to the thread in question ?

That kind of makes sense. Of course, the standard objection to your answer is something like the following: "This seems like a rather inefficient way to design the ideal society. If I was building intelligent agents from scratch, and I wanted them to conform to some ideal; then I'd just build them to do that from the start, instead of messing around with tsunamis and common colds".

To test for general intelligence, you can't test on the specific skill the bot's trained in.

I think we might have different definitions of what "general intelligence" is. I thought it meant something like, "being able to solve novel problems in some domain"; in this case, our domain is "human conversation". I may be willing to extend the definition to say, "...and also possessing the capacity to learn how to solve problems in some number of other domains".

Your definition, though, seems to involve solving problems in any domain. I think this definition is too broad. No human is capable of doing everything; and most humans are only good at a small number of things. An average mathematician can't compose music. An average musician can't do calculus. Some musicians can learn calculus (given enough time and motivation), but others cannot. Some mathematicians can learn to paint; others cannot.

Perhaps you mean to say that humans are not generally intelligent, and neither are AIs who pass the Turing Test ? In this case, I might agree with you.