Okay, thanks for confirming that Newcomb's problem is a relevant motivating example here.

"I don't know how to solve this problem, nor do I understand the exact structure of the calculation my computer program will perform in the course of solving this problem, nor can I state a mathematically precise meta-question, but I'm going to rely on the AI solving it for me 'cause it's supposed to be super-smart,"

I'm not saying that. I'm saying that self-modification solves the problem, assuming the CDT agent moves first, and that it seems simple enough that we can check that a not-very-smart AI solves it correctly on toy examples. If I get around to attempting that, I'll post to LessWrong.

Assuming the CDT agent moves first seems reasonable. I have no clue whether or when Omega is going to show up, so I feel no need to second-guess the AI about that schedule.

(Quoting out of order)

This is not only ugly...

As you know, we can define a causal decision theory agent in one line of math. I don't know a way to do that for TDT. Do you? If TDT could be concisely described, I'd agree that it's the less ugly alternative.

but also has worse implications for e.g. meeting an alien AI who wants to cooperate with you, or worse, an alien AI that is trying to blackmail you.

I'm failing to suspend disbelief here. Do you have motivating examples for TDT that seem likely to happen before Kurzweil's schedule for the Singularity causes us to either win or lose the game?

Is that really so bad, if it takes the state of the world at the point before it self-modifies as an unchangeable given, and self-modifies to a decision theory that only considers states from that point on as changeable by its decision theory? For one thing, doesn't that avoid Roko's basilisk?

Doesn't have a name as far as I know. But I'm not sure it deserves one; would CDT really be a probable output anywhere besides a verbal theory advocated by human philosophers in our own Everett branch? Maybe, now that I think about it, but even so, does it matter?

It is useful to separate in one's mind the difference between on one hand being able to One Box and cooperate in PD with agents that you know well (shared source code) and on the other hand not firing on Baby Eaters after they have already chosen not to fire on you. This is especially the case when first grappling the subject. (Could you confirm, by the way, that Akon's decision in that particular paragraph or two is approximately what TDT would suggest?)

The above is particularly relevant because the "have access to each other's source code" is such a useful intuition pump when grappling with or explaining the solutions to many of the relevant decision problems. It is useful to be able to draw a line on just how far the source code metaphor can take you.

There is also something distasteful about making comparisons to a decision theory that isn't even implicitly stable under self modification. A CDT agent will change to CDT++ unless there is an additional flaw in the agent beyond the poor decision making strategy. If I create a CDT agent, give it time to think and then give it Newcomb's problem it will One Box (and also no longer be a CDT agent). It is the errors in the agent that still remain after that time that need TDT or UDT to fix.

But it will calculate that expected value using CDT!expectation, meaning that it won't see how self-modifying to be a timeless decision theorist could possibly affect what's already in the box, etcetera.

*nod* This is just the 'new rules starting now' option. What the CDT agent does when it wakes up in an empty, boring room and does some introspection.

But it will calculate that expected value using CDT!expectation, meaning that it won't see how self-modifying to be a timeless decision theorist could possibly affect what's already in the box, etcetera.

The CDT is making a decision about whether to self-modify even before it meets the alien, based on its expectation of meeting the alien. How does CDT!expectation differ from Eliezer!expectation before we meet the alien?

Doesn't have a name as far as I know. But I'm not sure it deserves one; would CDT really be a probable output anywhere besides a verbal theory advocated by human philosophers in our own Everett branch?

Yes, because there are lemmas you can prove about (some) decision theory problems which imply that CDT and UDT give the same output. For example, CDT works if there is exists a total ordering over inputs given to the strategy, common to all execution histories, such that the world program invokes the strategy only with increasing, non-repeating inputs on that ordering. There are (relatively) easy algorithms for these cases. CDT in general is then a matter of applying a theorem when one of its preconditions doesn't hold, which is one of the most common math mistakes ever.

But I'm not sure it deserves one; would CDT really be a probable output anywhere besides a verbal theory advocated by human philosophers in our own Everett branch? Maybe, now that I think about it, but even so, does it matter?

Yes, for reasons of game theory and of practical singularity strategy.

Game theory, because things in Everett branches that are 'closest' to us might be the ones it's most important to be able to interact with, since they're easier to simulate and their preferences are more likely to have interesting overlap with ours. Knowing very roughly what to expect from our neighbors is useful.

And singularity strategy, because if you can show that architectures like AIXI-tl have some non-negligible chance of converging to whatever an FAI would have converged to, as far as actual policies go, then that is a very important thing to know; especially if a non-uFAI existential risk starts to look imminent (but the game theory in that case is crazy). It is not probable but there's a hell of a lot of structural uncertainty and Omohundro's AI drives are still pretty informal. I am still not absolutely sure I know how a self-modifying superintelligence would interpret or reflect on its utility function or terms therein (or how it would reflect on its implicit policy for interpreting or reflecting on utility functions or terms therein). The apparent rigidity of Goedel machines might constitute a disproof in theory (though I'm not sure about that), but when some of the terms are sequences of letters like "makeHumansHappy" or formally manipulable correlated markers of human happiness, then I don't know how the syntax gets turned into semantics (or fails entirely to get turned into semantics, as they case may well be).

But it will calculate that expected value using CDT!expectation, meaning that it won't see how self-modifying to be a timeless decision theorist could possibly affect what's already in the box, etcetera.

This implies that the actually-implemented-CDT agent has a single level of abstraction/granularity at like the naive realist physical level at which it's proving things about causal relationships. Like, it can't/shouldn't prove causal relationships at the level of string theory, and yet it's still confident that its actions are causing things despite that structural uncertainty, and yet despite the symmetry it for some reason cannot possibly see how switching a few transistors or changing its decision policy might affect things via relationships that are ultimately causal but currently unknown for reasons of boundedness and not speculative metaphysics. It's plausible, but I think letting a universal hypothesis space or maybe even just Goedelian limitations enter the decision calculus at any point is going to make such rigidity unlikely. (This is related to how a non-hypercomputation-driven decision theory in general might reason about the possibility of hypercomputation, or the risk of self-diagonalization, I think.)