Would be interested in parts of this. Start with whatever you think you're best at writing that doesn't require earlier material and let's have comments. Some things I want to discuss and hear discussed:

• Why shouldn't I live on interest plus part of the principal during retirement? If I should, what does this do to all the numbers guides suggest?
• Why shouldn't I invest 100% in Berkshire Hathaway while young? Don't just hold up a sign saying diversification, because then I have to ask why BRK isn't as diversified as you're suggesting I be.
• What trade-off am I actually making if I donate e.g. 20% of my annual after-tax income to charity? Years until retirement, dollars per month during retirement, etc.?
• Naively each time tick I want to maximize the expectation of the log of the ratio of my investments' value at the end of the tick to the value at the beginning. What's missing from that other than non-independence of behavior during different time ticks?

I would feel right about paying $0-$15 depending on my mood to read a book on this sort of material written by someone who wrote the above post if I had that option soon after it was available without reactions from other readers. With bad reactions, $0. With no reactions after enough time that there should have been good reactions, $0-5. With good reactions, $5-40-200+ depending on content and reactions. I think more good would come of discussion on well-written initial posts than of a long monologue.

Linearity is required... what's preserved is the ranking of lotteries over outcomes. Preserving the order of "a cookie" and "two cookies but no dollar" and "three cookies but a dollar in debt" isn't enough, you also have to preserve "40% chance of a cookie and 60% chance of two cookies but no dollar".

This is in the context of reinvesting dividends of cognitive work, assuming it takes exponentially greater investments to produce linearly greater returns. For example, maybe we get a return of log(X) cognitive work per time with what we have now, and to get returns of log(X+k) per time we need to have invested X+k cognitive work. What does it look like to reinvest all of our dividends? After dt, we have invested X+log(X) and our new return is log(X+log(X)). After 2dt, we have invested X+log(X)+log(X+log(X)), etc.

The corrected paragraph would then look like:

Therefore, an AI trying to invest an amount of cognitive work w to improve its own performance will get returns that go as log(w), or if further reinvested, an additional log(1+log(w)/w), and the sequence log(w)+log(1+log(w)/w)+log(1+log(w+log(w))/(w+log(w))) will converge very quickly.

Except then it's not at all clear that the series converges quickly. Let's check... we could say the capital over time is f(t), with f(0)=w, and the derivative at t is f'(t)=log(f(t)). Then our capital over time is f(t)=li^(-1)(t+li(w)). This makes our capital / log-capital approximately linear, so our capital is superlinear, but not exponential.

Can I have a representative example of a problem where this is appropriate?

I have not yet come to terms with how constructs of personal identity fit in with having or not having a utility function. What if it makes most sense to model my agency as a continuous limit of a series of ever more divided discrete agents who bring subsequent, very similar, future agents into existence? Maybe each of those tiny-in-time-extent agents have a utility function, and maybe that's significant?

Good post. Asking "okay, how sensitive is Karnofsky's counterargument to the size of the priors?" and actually answering that question was very worthwhile IMO.

Your post was funded by MIRI. Can you tell us what they asked? Was it "evaluate Karnofsky's argument", "rebut this post", "check the sensitivity of the argument to the priors' size and expand on it", "see how much BA affects our estimates", or what?

I tried kasina meditation for a while. It was frustrating because my lack of visual mental imagery didn't jive with descriptions of how to practice. :)

I may or may not try the video game thing. Spending time is easy, spending lots of consecutive time is more costly. :) I have taken walks as you describe, except in unfamiliar areas but where I don't expect to run into things all the time. I don't see my surroundings, I just know where they are approximately (and they update when I move). My guess is that visualizing something from nothing is also part of the spectrum...? I definitely had more success with visualizing fully-featured scenes (they end up mostly as not-quite amorphous blobs but totally in the right place and kinda painted over with Imagination) than geometry.

I was given an excellent geometry problem by Dr. Nigel Thomas.

It might be worth attempting to see how you perform on certain types of spatial thinking problems that most people claim to use imagery to solve (although no correlation seems to exist between spatial thinking ability and the vividness that people report their imagery to have). Try to solve the problem below, in your head, without drawing diagrams or making calculations on paper or anything like that. The four narrow sides of a 1 cm by 4 cm by 4 cm block are painted red. The top and bottom are painted blue. The block is then cut into sixteen 1 cm cubes. 1. How many cubes have both red and blue faces? 2. How many cubes have one red and two blue faces? 3. How many cubes have no painted faces? Most people say they use imagery to do this, and count the relevant cubes in their image. Were you able to solve all or any part of the problem at all? Did it seem very difficult? How, in fact, did you solve it (if you did)? Did you have to consciously employ any formal knowledge of geometry or other mathematics (beyond counting)?

When I solved this, I had the interesting experience of Imagining the 4x4 block of 16 blocks, noting that the outside ones (all but 4) had red paint on them, and all of them had blue paint... but I only "put" blue paint on the top. My diagram was flat, oriented like a pancake. None of this was Mental Imagery. Then when I was asked how many cubes had red and blue faces, I felt around the edges of the block. Motor/haptic mental imagery. Then when I was asked how many cubes had 1 red and 2 blue faces, I immediately thought the question was 1 red and 1 blue since I didn't have blue paint on the bottom in my model (I'm not sure if I had a bottom in my model). I thought "when would they have more than 1 red? ah the corners", and then had the distinct vivid motor mental imagery of moving my hand and touching two non-corner side blocks on the left of my model, then two at the far side, then two on the right, then two on the near side, counting "2, 4, 6, 8". This was a different experience than my usual Imagining... but I'm not sure if it was qualitatively different or just more "vivid" motor mental imagery.

I approve of this post being in Main and I approve of your accommodation to the appearance of that demand and I approve of your testing what might be but I hope are not the boundaries of what people want to see in Main.