I don't really understand your approach yet. Let's call your decision theory CLDT. You say counterfactuals in CLDT should correspond to consistent universes. For example, the counterfactual "what if a CLDT agent two-boxed in Newcomb's problem" should correspond to a consistent universe where a CLDT agent two-boxes on Newcomb's problem. Can you describe that universe in more detail?
Done! I didn't do it at first because I thought it'd have to be in person only, but then clicked around in the form and found that remote is also possible.
Besides math and programming, what are your other skills and interests?
Playing and composing music is the main one.
I have an idea of a puzzle game, not sure if it would be good or bad, I haven’t done even a prototype. So if anyone is interested, feel free to try
Yeah, you're missing out on all the fun in game-making :-) You must build the prototype yourself, play with it yourself, tweak the mechanics, and at some moment the stars will align and something will just work and you'll know it. There's no way anyone else can do it but you.
Yeah. My point was, we can't even be sure which behavior-preserving optimizations (of the kind done by optimizing compilers, say) will preserve consciousness. It's worrying because these can happen innocuously, e.g. when your upload gets migrated to a newer CPU with fancier heuristics. And yeah, when self-modification comes into the picture, it gets even worse.
I think there's a pretty strong argument to be more wary about uploading. It's been stated a few times on LW, originally by Wei Dai if I remember right, but maybe worth restating here.
Imagine the uploading goes according to plan, the map of your neurons and connections has been copied into a computer, and simulating it leads to a person who talks, walks in a simulated world, and answers questions about their consciousness. But imagine also that the upload is being run on a computer that can apply optimizations on the fly. For example, it could watch the input-output behavior of some NN fragment, learn a smaller and faster NN fragment with the same input-output behavior, and substitute it for the original. Or it could skip executing branches that don't make a difference to behavior at a given time.
Where do we draw the line which optimizations to allow? It seems we cannot allow all behavior-preserving optimizations, because that might lead to a kind of LLM that dutifully says "I'm conscious" without actually being so. (The p-zombie argument doesn't apply here, because there is indeed a causal chain from human consciousness to an LLM saying "I'm conscious" - which goes through the LLM's training data.) But we must allow some optimizations, because today's computers already apply many optimizations, and compilers even more so. For example, skipping unused branches is pretty standard. The company doing your uploading might not even tell you about the optimizations they use, given that the result will behave just like you anyway, and the 10x speedup is profitable. The result could be a kind of apocalypse by optimization, with nobody noticing. A bit unsettling, no?
The key point of this argument isn't just that some optimizations are dangerous, but that we have no principled way of telling which ones are. We thought we had philosophical clarity with "just upload all my neurons and connections and then run them on a computer", but that doesn't seem enough to answer questions like this. I think it needs new ideas.
Yeah, that seems to agree with my pessimistic view - that we are selfish animals, except we have culture, and some cultures accidentally contain altruism. So the answer to your question "are humans fundamentally good or evil?" is "humans are fundamentally evil, and only accidentally sometimes good".
I don't think altruism is evolutionarily connected to power as you describe. Caesar didn't come to power by being better at altruism, but by being better at coordinating violence. For a more general example, the Greek and other myths don't give many examples of compassion (though they give many other human values), it seems the modern form of compassion only appeared with Jesus, which is too recent for any evolutionary explanation.
So it's possible that the little we got of altruism and other nice things are merely lucky memes. Not even a necessary adaptation, but more like a cultural peacock's tail, which appeared randomly and might fix itself or not. While our fundamental nature remains that of other living creatures, who eat each other without caring much.
Guilty as charged - I did read your post as arguing in favor of geometric averaging, when it really wasn't. Sorry.
The main point still seems strange to me, though. Suppose you were programming a robot to act on my behalf, and you asked me to write out some goodness values for outcomes, to program them into the robot. Then before writing out the goodnesses I'd be sure to ask you: which method would the robot use for evaluating lotteries over outcomes? Depending on that, the goodness values I'd write for you (to achieve the desired behavior from the robot) would be very different.
To me it suggests that the goodness values and the averaging method are not truly independent degrees of freedom. So it's simpler to nail down the averaging method, to use ordinary arithmetic averaging, and then assign the goodness values. We don't lose any ability to describe behavior (as long as it's consistent), and we remain with only the degree of freedom that actually matters.
That makes me even more confused. Are you arguing that we ought to (1) assign some "goodness" values to outcomes, and then (2) maximize the geometric expectation of "goodness" resulting from our actions? But then wouldn't any argument for (2) depend on the details of how (1) is done? For example, if "goodnesses" were logarithmic in the first place, then wouldn't you want to use arithmetic averaging? Is there some description of how we should assign goodnesses in (1) without a kind of firm ground that VNM gives?
Yeah, that might be a good idea in case any rich employers stumble on this :-)
In terms of goals, I like making something, having many people use it, and getting paid for it. I'm not as motivated by meaning, probably different from most EAs in that sense.
In terms of skillset, I'd say I'm a frontend-focused generalist. The most fun programming experience in my life was when I built an online map just by myself - the rendering of map data to png tiles, the serving backend, the javascript for dragging and zooming, there weren't many libraries back then - and then it got released and got hundreds of thousands of users. The second most fun was when I made the game - coming up with the idea, iterating on the mechanics, graphic design, audio programming, writing text, packaging for web and mobile, the whole thing - and it got quite popular too. So that's the prototypical good job for me.