All of Henrik_Jonsson's Comments + Replies

Having worked for / talked to some people who became decamillionaires or higher through startups, a common theme seems to be just being really competitive. They don't care too much about money for money's sake - that's just what we currently use to send the signal "your startup is doing something we really like" in our society.

I tried it (for building muscle), kept to the instructions fairly strictly and saw improvements over my regular workout, but nowhere near the results described in the book. Much of the book makes sense, but it might be overly specific to his own physiology, and/or have non-functional components mixed in by mistake.

Very good post Louie! I agree with all the points, pretty much.

Number 11 seems especially important - it seems like a common trap for people in our crowd to try to over-optimize, so for me having an enjoyable life is a very high priority. A way of thinking that seems to work personally is to work on the margin rather than trying to reorganize my life top-down - to try to continually be a bit more awesome, work with more interesting people, get a bit more money, invest a bit more energy, etc, than yesterday.

In contrast, if I started out trying to allocate the resources I had access to / could gain access to in an optimal manner I suspect I would be paralyzed.

The discussion section sounds like a solid idea. As for making LW less intimidating, I'd rank it as the grace period > doing nothing > "karma coward", though I think users should be able to exit the grace period earlier by choice, and also possibly the score of comments on users in the grace period should be hidden (not just kept from affecting the total karma).

Seeing your comments plummet in score might be demoralizing, even if it doesn't affect your total score.

0[anonymous]
Seconding both.

Nick Tarleton said it well, but to try it another way: Depending on how you phrase things, both to yourself and others, the situation can appear to be as bleak as you describe it, or alternatively rather good indeed. If you were to phrase it as being stuck with a brain built for chasing deer across the savanna and caring about the few dozen members of your tribe, being able to try to gain money (because it's the most effective means to whatever your ends) and investing some appreciable fraction of it in the cause with highest expected future payoff, despit... (read more)

4Blueberry
That gives you an incentive to not try to change your psychology, or even see if it's possible to change. If seeing your psychology as immutable gives you a reason to get what you want, you'll be biased against seeing it any other way. It's perfectly fine to choose Starbucks lattes over strangers' lives, lest we end up like Zachary Baumkletterer, but let's at least be honest about our preferences.

My impression is that JVM is worse at concurrency than every other approach that's been tried so far.

Haskell and other functional programming languages has many promising ideas but isn't widely used in the industry AFAIK.

This presentation gives a good short overview of the current state of concurrency approaches.

I took part of the 2009 summer program during the vacation of my day job as a software developer in Sweden. This entailed spending five weeks with the smartest and most dedicated people I have ever met, working on a wide array of projects both short- and long-term, some of which were finished by the time I left and some of which are still on-going.

My biggest worry beforehand was that I would not be anywhere near talented enough to participate and contribute in the company of SIAI employees and supporters. That seems to not have occurred, though I don't cla... (read more)

0djcb
Thanks. Basically it seems that people are more likely to call harmful side-effects intentional, compared to beneficial ones. I'm not sure this really is a bias; the harmful/beneficial cases are not exact counterparts: in the harmful case one could assume that the actor needs to mentally do something -- namely, 'overcoming the moral problem' or 'silence his/her conscience', which makes the harmful case indeed a bit more 'intentional'.

I read the book, but found it rambling and poorly supported. The basic point about agents with hyperbolic discounting having dynamic inconsistencies is very important, but I wouldn't recommend the book over Ainslie's article. The only mental note I made of something new (for me) and interesting was a point about issues with a "bright line" being much easier to handle than those without. For example, it's easier to stop drinking alcohol completely than to drink less than a specific limit at each occasion, and even harder to eat a proper diet, when... (read more)

0Dojan
Same here! Are you still around?

Ainslie's answer is that he should set a hard-and-fast rule: "I will never drink alcoholism".

You probably meant to write "alcohol" here.

All data, even anecdotal, on how to beat akrasia is great, and this sounds like a method that might work well in many cases. If you wanted to raise your odds of succeeding even more you could probably make your oath in front of a group of friends or family members, or even include a rule about donating your money or time if you failed, preferably to a cause you hated for bonus motivation.

I'd like ... (read more)

1Henrik_Jonsson
I read the book, but found it rambling and poorly supported. The basic point about agents with hyperbolic discounting having dynamic inconsistencies is very important, but I wouldn't recommend the book over Ainslie's article. The only mental note I made of something new (for me) and interesting was a point about issues with a "bright line" being much easier to handle than those without. For example, it's easier to stop drinking alcohol completely than to drink less than a specific limit at each occasion, and even harder to eat a proper diet, when you obviously cannot make us of the only very bright line; no food at all. I have been busy (with the SIAI summer program), but I do think I actually would have found time to write the post if I had found more data that was both interesting and not obvious to the LW crowd. This might be rationalization, but I don't think the me of one month ago would have wanted a post written about the book if he had known the contents of the book.
1MichaelHoward
So... what happened? Why do you think your oath failed to work?
2byrnema
If it was a typo, it was a fortuitous one! I've quoted it several times in conversation while explaining that I read 'somewhere' that the key to quitting is in identifying the single, local instance (a beer) with the global bad (alcoholism) even if the connection isn't technically true. Because otherwise, without thinking about it this way, avoiding that single drink may seem silly or irrational, and that is what usually defeats me. (Not giving the small steps enough credit in whatever I'm trying to achieve.)

It would be interesting to see Searle debate anyone who didn't defer to his high status and common-sense-sounding arguments and pressed him to the wall on what exactly would happen if you, say, simulated a human brain in high resolution. His intuition pumps are powerful ("thought is just like digestion, you don't really believe a computer will digest food if you simulate gastric enzymes, do you?"), but he never really presents any argument on his views of consciousness or AI, at least what I've seen.

On the one hand it would be cool if the notes one jots down could self-organize somehow, even a little bit.

While I agree that it it would be cool, anything that doesn't keep your notes exactly like you left them is likely to be more annoying than productive unless it is very cleverly done. (Remember Microsoft Clippy?) You'd probably need to tag at least some things, like persons and places.

But even if the AI discovered some things about our physics, it does not significantly narrow the range of possible minds. It doesn't know if it's dealing with paperclippers or a pebblesorters. It might know roughly how smart we are.

You're using your (human) mind to predict what a postulated potentially smarter-than-human intelligence could and could not do.

It might not operate on the same timescales as us. It might do things that appear like pure magic. No matter how often you took snapshots and checked how far it had gotten in figuring out details abo... (read more)

4John_Maxwell
I don't think the number of Planck intervals is especially useful to cite... it seems like the relevant factor is CPU cycles, and while I'm not an expert on CPUs, I'm pretty sure that we're not bumping up on Planck intervals yet. Relatedly, if you were worried about self-improving superintelligence, you could give your AI a slow CPU.
2JamesAndrix
First, I feel like we're talking past each other a bit. Second, I edited this somewhat out of order, apologies if it doesn't flow. I am trying to look at this in a worst-case scenario, I'll grant that the AI is smart enough to solve any given solvable problem in a single iteration, that it's that smart even in the first experiment, and it would prioritze discovering it's true environment and paperclipping it. I'm proposing that there exists a sandbox which [provably] can't be gotten out of. And also a set of problems which do not convey information about our universe. You're using your (human) mind to predict what a postulated potentially smarter-than-human intelligence could and could not do. Isn't that required of FAI anyway? AI sitting inside thirty nestled sandboxes even 10 milliseconds (10^41 Planck intervals) of CPU time. Again talking past each other, I'm thinking in terms of giving the paperclipper hours. In the ideal, there isn't a provision for letting the AI out of the sandbox. thinking a bit more... None of it's problems/results need even be applicable to our universe, except for general principles of intelligence creation. Having it construct a CEV for itself might show our motives too much, or might not. (hmmmm, we should make sure any CEV we create finds, protects, and applies itself to any simulations used in its construction, in case our simulators use our CEV in their own universe :-) especially if you gave it motives for hiding that progress (such as pulling the plug every time it came close). But its existing self would never experience getting close, in the same way we have no records of the superweapons race of 1918. ;-) Between Iterations, we can retroactively withdraw information that turned out to be revealing, during iterations, it has no capacity to affect our universe. I think we can put strong brackets around what can be done with certain amounts of information, even by a superintelligence. Knowing all our physics doesn't imp

I think my other reply applies here too, if you read "communications channel" as all the information that might be inferred from the universe the AI finds itself in. Either the AI is not smart enough to be a worry without any sandboxing at all, or you have enough to worry about that you should not be relying on the sandbox to protect you.

Your point about our own simulation (if it is one) lacking a simple communications channel actually works against you - In our universe the simulation hypothesis has been proposed, despite the fact that we have only human intelligence to work with.

1JamesAndrix
But constructing the hypothesis isn't evidence that it's true, and if it is true, that still leaves us with (so far) no information about our simulators, and no way to guess their motives, let alone try to trick them. I've actually been considering the possibility of a process that would create random universes and challenges. But even if the AI discovered some things about our physics, it does not significantly narrow the range of possible minds. It doesn't know if it's dealing with paperclippers or a pebblesorters. It might know roughly how smart we are. The other half of the communication channel would be the solutions and self-modifications it provides at each iteration. These should not be emotionally compelling and would be subject to an arbitrary amount of review. There are other advantages to this kind of sandbox, we can present it the task of inferring our physics at various levels of its development, and archive any versions that have learned more than we are comfortable with. (anything) Keeping secrets from a hostile intelligence is something we already have formal and intuitive experience with. Controlling it's universe and peering into it mind are bonuses. Interesting Cognitive bias side note: While writing this, I was inclined to write in a style to make it seem silly that an AI could mindhack us based on a few bits. I do think that it's very unlikely, but if I wrote as I was thinking, it would probably have sounded dismissive. I do think a design goal should be zero bits.

We shall not cease from exploration and the end of our exploring shall be to return where we started and know the place for the first time.

-- T.S. Eliot

2Tyrrell_McAllister
Don't know why this was downvoted. The second clause is essentially Egan's Law: It all adds up to normality.

Once again, we are saddled with a Stone Age moral psychology that is appropriate to life in small, homogeneous communities in which all members share roughly the same moral outlook. Our minds trick us into thinking that we are absolutely right and that they are absolutely wrong because, once upon a time, this was a useful way to think. It is no more, though it remains natural as ever. We love our respective moral senses. They are as much a part of us as anything. But if we are to live together in the world we have created for ourselves, so unlike the one

... (read more)

If you had a program that might or might not be on a track to self-improve and initiate an Intelligence explosion you'd better be sure enough that it would remain friendly to, at the very least, give it a robot body, a scalpel, and stand with your throat exposed before it.

Surrounding it with a sandboxed environment couldn't be guaranteed to add any meaningful amount of security. Maybe the few bits of information you provide through your communications channel would be enough for this particular agent to reverse-engineer your psychology and find that corre... (read more)

0Vladimir_Nesov
Of course.

As long as you have a communications channel to the AI it would not be secure, since you are not a secure system and could be compromised by a sufficiently intelligent agent.

See http://yudkowsky.net/singularity/aibox

1JamesAndrix
I am familiar with he AI Box experiment. My short answer: So don't have a communications channel, in the same way that if anyone is running our simulation, they don't currently have a communications channel with us. The AI need only find itself in a series of universes with progressively more difficult challenges. (much like eurisko, actually) We can construct problems that have no bearing on our physics or our evolutionary history. (I'm not saying it's trivial, there would need to be a security review process) If a pure software intelligence explosion is feasible, then we should be able to get it to create and prove a CEV before it knows anything about us, or that it's possible to communicate with us. And just because humans aren't secure system doesn't mean we can't make secure systems.
0asciilifeform
Until Yudkowsky releases the chat transcripts for public review, the AI Box experiment proves nothing.
5Vladimir_Nesov
Intelligence is no help if you need to open a safe that only gets opened by one of the 10^10 possible combinations. You also need enough information about the correct combination to have any chance of guessing it. Humans likely have different compromising combinations, if any, so you'd also need to know a lot about a specific person, or even about their state of mind at the moment, the knowledge of human psychology in general might not be enough. (But apparently what would look to a human like almost no information about the correct combination might be more than enough to a sufficiently clever AI, so it's unsafe, but it's not magically unsafe.)

This is one of my all-time favourite posts of yours, Eliezer. I can recognize elements of what you're describing here in my own thinking over the last year or so, but you've made the processes so much more clear.

As I'm writing this, just a few minutes after finishing the post, it's increasingly difficult not to think of this as "obvious all along" and it's getting harder to pin down exactly what in the post that caused me to smile in recognition more than once.

Much of it may have been obvious to me before reading this post as well, but now the verbal imagery needed to clearly explain these things to myself (and hopefully to others) is available. Thank you for these new tools.