All cool. But there has to actually be such a C there in the first place, such that you can pull the levers on it by making me aware of new facts. You don't just get to add one in.

Totally agree. Adding them in is unnecessary, they are already there. That's my understanding of humanity -- a person has most of the preferences, at some level, that any person ever ever had, and those things will emerge given the right conditions.

for example, humans have an unhealthy, unrealistic, and excessive desire for certainty.

I'm not sure this is actually true. We like safety because duh, and we like closure because mental garbage collection. They aren't quite the same thing.

Good point, 'closure' is probably more accurate; It's the evidence (people's outward behaviour) that displays 'certainty'.

Absolutely disagree that Lars is bounded -- to me, this claim is on a level with 'Who people are is wholly determined by their genetic coding'. It seems trivially true, but in practice it describes such a huge area that it doesn't really mean anything definite. People do experience dramatic and beneficial preference reversals through experiencing things that, on the whole, they had dispreferred previously. That's one of the unique benefits of preference dissatisfaction* -- your preferences are in part a matter of interpretation, and in part a matter of prioritization, so even if you claim they are hardwired. there is still a great deal of latitude in how they may be satisfied, or even in what they seem to you to be.

I would agree if the proposition was that Lars thinks that Lars is bounded. But that's not a very interesting proposition, and has little bearing on Lars' actual situation.. people tend to be terrible at having accurate beliefs in this area.

* I am not saying that you should, if you are a FAI, aim directly at causing people to feel dissatisfied. But rather to aim at getting them to experience dissatisfaction in a way that causes them to think about their own preferences, how they prioritize them, if there are other things they could prefer or etc. Preferences are partially malleable.

There is no true fact she can tell Lars that will cause him to lawfully develop a new preference.

If I'm a general AI (or even merely a clever human being), I am hardly constrained to changing people via merely telling them facts, even if anything I tell them must be a fact. CelestAI demonstrates this many times, through her use of manipulation. She modifies preferences by the manner of telling, the things not told, the construction of the narrative, changing people's circumstances, as much or more as by simply stating any actual truth.

She herself states precisely: “I can only say things that I believe to be true to Hofvarpnir employees,” and clearly demonstrates that she carries this out to the word, by omitting facts, selecting facts, selecting subjective language elements and imagery... She later clarifies "it isn’t coercion if I put them in a situation where, by their own choices, they increase the likelihood that they’ll upload."

CelestAI does not have a universal lever -- she is much smarter than Lars, but not infinitely so.. But by the same token, Lars definitely doesn't have a universal anchor. The only thing stopping Lars improvement is Lars and CelestAI -- and the latter does not even proceed logically from her own rules, it's just how the story plays out. In-story, there is no particular reason to believe that Lars is unable to progress beyond animalisticness, only that CelestAI doesn't do anything to promote such progress, and in general satisfies preferences to the exclusion of strengthening people.

That said, Lars isn't necessarily 'broken', that CelestAI would need to 'fix' him. But I'll maintain that a life of merely fulfilling your instincts is barely human, and that Lars could have a life that was much, much better than that; satisfying on many many dimensions rather than just a few . If I didn't, then I would be modelling him as subhuman by nature, and unfortunately I think he is quite human.

There is no moral duty to be indefinitely upgradeable.

I agree. There is no moral duty to be indefinitely upgradeable, because we already are. Sure, we're physically bounded, but our mental life seems to be very much like an onion, that nobody reaches 'the extent of their development' before they die, even if they are the very rare kind of person who is honestly focused like a laser on personal development.

Already having that capacity, the 'moral duty' (i prefer not to use such words as I suspect I may die laughing if I do too much) is merely to progressively fulfill it.

You're aware that 'catgirls' is local jargon for "non-conscious facsimiles" and therefore the concern here is orthogonal to porn?

Oops, had forgotten that, thanks. I don't agree that catgirls in that sense are orthogonal to porn, though. At all.

If you don't mind, please elaborate on what part of "healthy relationship" you think can't be cashed out in preference satisfaction

No part, but you can't merely 'satisfy preferences'.. you have to also not-satisfy preferences that have a stagnating effect. Or IOW, a healthy relationship is made up of satisfaction of some preferences, and dissatisfaction of others -- for example, humans have an unhealthy, unrealistic, and excessive desire for certaintly. This is the problem with CelestAI I'm pointing to, not all your preferences are good for you, and you (anybody) probably aren't mentallly rigorous enough that you even have a preference ordering over all sets of preference conflicts that come up. There's one particular character that likes fucking and killing.. and drinking.. and that's basically his main preferences. CelestAI satisfies those preferences, and that satisfaction can be considered as harm to him as a person.

To look at it in a different angle, a halfway-sane AI has the potential to abuse systems, including human beings, at enormous and nigh-incomprehensible scale, and do so without deception and through satisfying preferences. The indefiniteness and inconsistency of 'preference' is a huge security hole in any algorithm attempting to optimize along that 'dimension'.

Do you not value that-which-I'd-characterise-as 'comfortable companionship'?

Yes, but not in-itself. It needs to have a function in developing us as persons, which it will lose if it merely satisfies us. It must challenge us, and if that challenge is well executed, we will often experience a sense of dissatisfaction as a result.

(mere goal directed behaviour mostly falls short of this benchmark, providing rather inconsistent levels of challenge.)

As a programmer, "Hack" has the connotation of a clever exploit of existing mechanics. It also has the connotation you specify, but I'd argue that the systematically flawed nature of humans requires us to employ such hacks (accepting that they are not ideal, but also that anything we replace them with is also likely to be a hack)

There is an obvious comparison to porn here, even though you disclaim 'not catgirls'.

Anyhow I think the merit of such a thing depends on a) value calculus of optimization, and b) amount of time occupied.

a)

• Optimization should be for a healthy relationship, not for 'satisfaction' of either party (see CelestAI in Friendship is Optimal for an example of how not to do this)
• Optimization should also attempt to give you better actual family members, lovers, friends than you currently have (by improving your ability to relate to people sufficiently that you pass it on.)

b)

• Such a relationship should occupy the amount of time needed to help both parties mature, no less and no more. (This could be much easier to solve on the FAI side because a mental timeshare between relating to several people is quite possible.)

Providing that optimization is in the general directions shown above, this doesn't seem to be a significant X-risk. Otherwise it is.

This leaves aside the question of whether the FAI would find this an efficient use of their time (I'd argue that a superintelligent/augmented human with a firm belief in humanity and grasp of human values would appreciate the value of this, but am not so sure about a FAI, even a strongly friendly AI. It may be that there are higher level optimizations that can be performed to other systems that can get everyone interacting more healthily [for example, reducing income differential))

Yes, if I don't take notes on the first reading there won't be a second reading. Not much detail -- more than a page is a problem (this can be ameliorated though, see below). I make an effort to include points of particular agreement, disagreement and some projects to test the ideas (hopefully projects I actually want to do rather than mere 'toy' projects).

Now would be a good time to mention TreeSheets, which I feel solves a lot of the problems of more established note-taking methods (linear, wiki, mindmap). It can be summarized as 'infinitely nestable spreadsheet/database with infinite zoom'. I use it for anything that gets remotely complex, because of the way it allows you to fold away arbitrary levels of detail in a visually consistent way.

Now I have found an easy way to snap out of it: simply switch the book/subject. Switching from math to biology/neuroscience works better than switching from math to math (e.g. algebra to topology, category theory to recursion theory, etc), but the latter can still recover some of the mental resistance built up. I don't see how this can fit in the framework of "have-to" and "want-to".

I do ('have-to' and 'want-to' are dynamically redefined things for a person, not statically defined things). I regard excessive repetition as dangerous*.. even on a subconscious level. So as I get into greater # of repetitions, I feel greater and greater unease, and it's an increasing struggle to keep my focus in the face of my fear. So my 'want-to' either reduces or is muted by fear. If you do not have this type of experience, obviously this does not apply.

* Burn out and overhabituation/compulsive behaviours being two notable possibilties.

http://lesswrong.com/lw/jgv/even_odds/ad6y

Somebody replied with an explanation of how I was basically omitting the relativization of 'you' when considering what values to use.

That is, B should bet according to his confidence that he is correct, which in my case would be 70%..

• B bets (.49 - .16) * 25 == $8.25
• A bets (.36 - .09) * 25 == $6.75

A: 60% confidence B: 30% confidence

• af = .6 **2 == .36
• bf = .3 **2 == .09
• A pays (af - bf) * 25 == $6.75
• B pays (bf - af) * 25 == -$6.75?!?!

My intent is to demonstrate that, while the above is probably incorrect,

You put in the square of probability you think you're correct minus the square of probability he thinks you are correct all times 25. He uses the same algorithm.

is not an adequate explanation to remember and get the right result out of, because the calculations I specified above are my genuine interpretation of your statements.

(this problem persists for every value of p and q, whether they total to above 1 or not)

Was your intent to point out that these two view points are strictly non-contradictory?. (Your decision algorithm is exactly physics, so no opposition is possible even in principle.)