It's been tried, and our computational capabilities fell woefully short of succeeding.

Is that because we don't have enough brute force, or because we don't know what calculation to apply it to?

I would be unsurprised to learn that calculating the folding state having global minimum energy was NP-complete; but for that reason I would be surprised to learn that nature solves that problem, rather than finding a local minimum.

I don't have a background in biology, but my impression from Wikipedia is that the tension between Anfinsen's dogma and Levinthal's paradox is yet unresolved.

When I was studying under Amotz Zahavi (originator of the handicap principle theory, which is what you're actually discussing), he used to make the exact same points. In fact, he used to say that "no communication is reliable unless it has a cost".

Having this outlook on life in the past 5 years made a lot of things seem very different - small questions like why some people don't use seatbelts and brag about it, or why men on dates leave big tips; but also bigger questions like advertizing, how hierarchical relationships really work, etc.

Also explained a lot about possible origins of (some) altruistic behaviors; Zahavi's favorite examples were from the research he and his wife conducted, wherein they observed small groups of social birds (forgot the species, sorry) where altrusitic behavior is common. And it turns out, it's the dominant birds who behave altrusitically, rather than exploit their weaker brethren - but doing so as a show of strength. My own favorite example is when a lower-status male caught a worm and tried feeding it to the alpha male. The latter proceeded to beat him up, take the morsel, and force-feed it back to the weaker male.

Best course I ever took :)

These questions seem decidedly UNfair to me.

No, they don't depend on the agent's decision-making algorithm; just on another agent's specific decision-making algorithm skewing results against an agent with an identical algorithm and letting all others reap the benefits of an otherwise non-advantageous situation.

So, a couple of things:

1. While I have not mathematically formulated this, I suspect that absolutely any decision theory can have a similar scenario constructed for it, using another agent / simulation with that specific decision theory as the basis for payoff. Go ahead and prove me wrong by supplying one where that's not the case...

2. It would be far more interesting to see a TDT-defeating question that doesn't have "TDT" (or taboo versions) as part of its phrasing. In general, questions of how a decision theory fares when agents can scan your algorithm and decide to discriminate against that algorithm specifically, are not interesting - because they are losing propositions in any case. When another agent has such profound understanding of how you tick and malice towards that algorithm, you have already lost.

I think you are doing this wrong, usually by the time I have convincing human-like androids I'm pretty near turning the Moon into computronium. You should focus on buying more processing power at this stage, perhaps set up a few hard to detect deep sea domes.

(nods) Likely.

And I agree that humans might be able to detect attempts at deception in a system at that stage of its development. I'm not vastly confident of it, though.

It is quite conceivable that all the necessary information is already out there, but we haven't been able to connect all the dots just yet.

Wouldn't this mean that creating fully functional self-replicating nanotechnology is just a matter of performing some thorough interdisciplinary studies (or meta-studies or whatever they are called) ? My impression was that there are currently several well-understood -- yet unresolved -- problems that prevent nanofactories from becoming a reality, though I could be wrong.

depending on how seriously you take it, such analysis might still take place, hard and prolonged though it may be).

Yup, agreed that it might.
And agreed that it might succeed, if it does take place.

One can have "detectors" in place set to find specific behaviors, but these would have assumptions that could easily fail. Detectors that would still be useful would be macro ones - where it tries to access and how - but these would provide only limited insight into the AI's thought process.

Agreed on all counts.

Re: what the AI knows... I'm not sure how to move forward here. Perhaps what's necessary is a step backwards.

If I've understood you correctly, you consider "having a conversation" to encompass exchanges such as:
A: "What day is it?"
B: "Na ni noo na"

If that's true, then sure, I agree that the minimal set of information about humans required to do that is zero; hell, I can do that with the rain.
And I agree that a system that's capable of doing that (e.g., the rain) is sufficiently unlikely to be capable of effective deception that the hypothesis isn't even worthy of consideration.
I also suggest that we stop using the phrase "having a conversation" at all, because it does not convey anything meaningful.

Having said that... for my own part, I initially understood you to be talking about a system capable of exchanges like: A: "What day is it?"
B: "Day seventeen."
A: "Why do you say that?"
B: "Because I've learned that 'a day' refers to a particular cycle of activity in the lab, and I have observed seventeen such cycles."

A system capable of doing that, I maintain, already knows enough about humans that I expect it to be capable of deception. (The specific questions and answers don't matter to my point, I can choose others if you prefer.)

It is not accusation or insult. It is the case though that the people in question (Luke, Eliezer) need to assume the possibility that people they are talking to are more intelligent than they are - something that is clearly more probable than not given available evidence - and they seem not to.

Yes, I agree that there's no guarantee that humans would behave as you describe.
Indeed, I don't find it likely.
But, sure, they might.

=== I agree that a stack trace can exist outside the AI's zone of control. What I was expressing skepticism about was that a system with even approximately human-level intelligence necessarily supports a stack trace that supports the kind of analysis you envision performing in the first place, without reference to intentional countermeasures.

By way of analogy: I can perform a structural integrity analysis on a bar of metal to determine whether it can support a given weight, but performing an equivalent analysis on a complicated structure comprising millions of bars of metal connected in a variety of arrangements via a variety of connectors using the same techniques is not necessarily possible.

But, sure, it might be.

======

I'd love to see a plausible line of thought by which the AI would try and succeed in deceiving humans at the get-go point without exposing its intentions.

Well, one place to start is with an understanding of the difference between "the minimal set of information about humans required to have a conversation with one at all" (my phrase) and "the most basic knowledge about humans" (your phrase). What do you imagine the latter to encompass, and how do you imagine the AI obtained this knowledge?

Yes, I understand what "exponential complexity" means :-)

It sounds, then, like you're on the side of kalla724 and myself (and against my Devil's Advocate persona): the AI would not be able to develop nanotechnology (or any other world-shattering technology) without performing physical experiments out in meatspace. It could do so in theory, but in practice, the computational requirements are too high.

But this puts severe constraints on the speed with which the AI's intelligence explosion could occur. Once it hits the limits of existing technology, it will have to take a long slog through empirical science, at human-grade speeds.