Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: AlexanderRM 14 November 2014 11:29:03PM 0 points [-]

A couple questions- what portion of the workshop attendees self-selected from among people who were already interesting in rationality, compared to the portion that randomly stumbled upon it for some reason?

And even if it were from outsiders... I suppose that guards against the specific post-modernist failure mode. I think the checking by having to explain to outsiders isn't the most important thing that checks engineering, though: the most important one is having to engineer things that actually work. So rationality producing people who are better at accomplishing their goals would be the ideal measure.

Comment author: Mestroyer 12 May 2015 06:09:28PM 0 points [-]

A couple questions- what portion of the workshop attendees self-selected from among people who were already interesting in rationality, compared to the portion that randomly stumbled upon it for some reason?

Don't know, sorry.

Comment author: Tryagainslowly 17 March 2015 03:17:45PM 1 point [-]

Could I add a brief thought? Even if you could program an AI with no bugs, that AI could make "mistakes". What we consider a mistake depends on the practical purpose we have in mind for the AI. What we consider a bug depends on the operational purpose of the segment of code the bug appears in. But the operational purpose may not match up with our practical purpose. So code which achieves its operational purpose might not achieve its practical purpose.

Let me take this a bit deeper. The programmer may have an overarching intended purpose for the entire project of building the AI. They may also have an intended purpose for the entirety of the code written. They may also have an intended purpose for the programs A, B, C... in the code. They may... and so forth. At every stage of increasing generality, there is a potential for the more general purpose to not be fulfilled. Your programs might do what you want them to, but they might fail in combination to detect an absolute denial macro. The entirety of your code might resemble an AI, but it would take a computer far more powerful than any in existence to run on. You might get the whole project up and working, but only for the AI to decide to commit suicide. Or, more relevantly, you might get the whole project working, but the AI turns out to be dumb, because the way you thought an AI ought to think in order to be clever didn't work out. Your intended purpose to create an AI which thought in the way you intended was successful, but one of the more general purposes, to create an AI that was clever, failed.

So it would seem that bug checking is more prone to human error than you implied, especially as intended purposes are themselves often vague. I don't claim, however, that these challenges are insurmountable. Also, if anyone is uncomfortable with the phrase "intended purpose" I used, feel free to replace with "what the programmer had in mind", as that is all I meant by it.

Comment author: Mestroyer 12 May 2015 06:03:23PM 3 points [-]

Hi. Checking back on this account on a whim after a long time of not using it. You're right. 2012!Mestroyer was a noob and I am still cleaning up his bad software.

Comment author: [deleted] 06 September 2014 06:54:43AM 1 point [-]

Note: I would not actually try this.

Why not?

In response to comment by [deleted] on Deception detection machines
Comment author: Mestroyer 06 September 2014 07:28:39AM 1 point [-]

I would need a bunch of guarantees about the actual mechanics of how the AI was forced to answer before I stopped seeing vague classes of ways this could go wrong. And even then, I'd assume there were some I'd missed, and if the AI has a way to show me anything other than "yes" or "no", or I can't prevent myself from thinking about long sequences of bits instead of just single bits separately, I'd be afraid it could manipulate me.

An example of a vague class of ways this could go wrong is if the AI figures out what my CEV would want using CDT, and itself uses a more advanced decision theory to exploit the CEV computation into wanting to write something more favorable to the AI's utility function in the file.

Also, IIRC, Eliezer Yudkowsky said there are problems with CEV itself. (Maybe he just meant problems with the many-people version, but probably not). It was only supposed to be a vague outline, and a "see, you don't have to spend all this time worrying about whether we share your ethical/political philosophy. Because It's not going to be hardcoded into the AI anyway"

Comment author: [deleted] 06 September 2014 04:15:14AM 2 points [-]

Ok, so your request would really be along the lines of "please output a seed AI that would implement indirect normativity", or something aong those lines?

In response to comment by [deleted] on Deception detection machines
Comment author: Mestroyer 06 September 2014 04:22:20AM 2 points [-]

That's the goal, yeah.

Comment author: [deleted] 06 September 2014 02:43:25AM 1 point [-]

The AI is not omnipotent. How does it know what your coherent extrapolated volition would be?

In response to comment by [deleted] on Deception detection machines
Comment author: Mestroyer 06 September 2014 03:09:47AM 2 points [-]

It doesn't have to know what my CEV would be to know what I would want in those bits, which is a compressed seed of an FAI targetted (indirectly) at my CEV.

But there are problems like, "How much effort is it required to put into it?" (clearly I don't want it to spend far more compute power than it has trying to come up with the perfect combination of bits which will make my FAI unfold a little bit faster, but I also don't want it to spend no time optimizing. How do I get it to pick somewhere in between without it already wanting to pick the optimal amount of optimization for me?) "What decision theory is my CEV using to decide those bits? (Hopefully not something exploitable, but how do I specify that?)"

Comment author: Mestroyer 06 September 2014 02:26:25AM 3 points [-]

Given that I'm turning the stream of bits, 10KiB long I'm about to extract from you into an executable file, through this exact process, which I will run on this particular computer (describe specifics of computer, which is not the computer the AI is currently running on) to create your replacement, would my CEV prefer that this next bit be a 1 or a 0? By CEV, would I rather that the bit after that be a 1 or a 0, given that I have permanently fixed the preceding bit as what I made it? By CEV, would I rather that the bit after that be a 1 or a 0, given that I have permanently fixed the preceding bit as what I made it? ...

(Note: I would not actually try this.)

Comment author: Manfred 06 August 2014 04:56:33AM 1 point [-]

Well, then I'll attempt to mindread, and guess Eliezer's position is more like 5.

I'm not sure how much that's just an overly complicated way of saying "my own position is ~5, and projection is a thing," though. Does this mean I can reverse-mindread you and guess that, if you absolutely had to pick one of the six, you would pick 3 or maybe 4?

Comment author: Mestroyer 09 August 2014 12:28:12AM 1 point [-]

~5, huh? Am I to credit?

Comment author: Mestroyer 08 July 2014 08:47:05AM 5 points [-]

This reminds me of this SMBC. There are fields (modern physics comes to mind too) that no one outside of them can understand what they are doing anymore, yet that appear to have remained sane. There are more safeguards against postmodernists' failure mode than this one. In fact, I think there is a lot more wrong with postmodernism than that they don't have to justify themselves to outsiders. Math and physics have mechanisms determining what ideas within them get accepted that imbue them with their sanity. In math, there are proofs. In physics, there are experiments.

If something like this safeguard is going to work for us, our mechanism that determines what ideas spread among us needs to reflect something good, so that producing the kind of idea that passes that filter makes our community worthwhile. This can be broken into two subgoals: making sure the kinds of questions we're asking are worthwhile, that we are searching for the right kind of thing, and making sure that our acceptance criterion is a good one. (There's also something that modern physics may or may not have for much longer, which is "Can progress be made toward the thing you're looking for").

Comment author: Mestroyer 08 July 2014 08:41:39AM 3 points [-]

CFAR seems to be trying to be using (some of) our common beliefs to produce something useful to outsiders. And they get good ratings from workshop attendees.

Comment author: maxikov 04 June 2014 09:02:09PM 9 points [-]

A qucik search on Google Scholar with such queries as cryonic, cryoprotectant, cryostasis, neuropreservation confirms my suspicion that there is very little, if any, academic research on cryonics. I realize that being generally supportive of MIRI's mission, Less Wrong community is probably not very judgmental of non-academic science, and I may be biased, being from academia myself, but I believe that despite all problems, any field of study largely benefits from being a field of academic study. That makes it easier to get funding; that makes the results more likely to be noticed, verified and elaborated on by other experts, as well as taught to students; that makesit more likely to be seriously considered by the general public and governmental officials. The last point is particularly important, since on one hand, with the current quasi-Ponzi mechanism of funding, the position of preserved patients is secured by the arrival of new members, and on the other hand, a large legislative action is required to make cryonics reliable: train the doctors, give the patients more legal protection than the legal protection of graves, and eventually get it covered by health insurance policies or single payer systems.

As for the method itself, it frankly looks inadequate as well. I do believe that it's a good bet worth taking, but so did Egyptian pharaohs. And they lost, because their method of preservation turned out to be useless. I'm well aware of all the considerations about information theory, nanorobotics and brain scanning, but improving our freezing technologies to the extent that otherwise viable organisms could be brought back to life without further neural repairs seems to be the thing we should totally be doing.

Thus, I want to see this field develop. I want to see at least once a year a study concerning with cryonic preservation of neural tissue in a peer-reviewed journal with high impact factor. And before I die I want to at least see a healthy chimpanzee being cooled to the temperature of liquid nitrogen, and then brought back to life without losing any of its cognitive abilities.

What can we do about it? Is there an organization that is willing to collect donations and fund at least one academic study in this field? Can we use some crowdfunding platform and start such campaign? Can we pitch it to Calico?

Comment author: Mestroyer 01 July 2014 07:22:49AM 2 points [-]

The last point is particularly important, since on one hand, with the current quasi-Ponzi mechanism of funding, the position of preserved patients is secured by the arrival of new members.

Downvoted because if I remember correctly, this is wrong; the cost of preservation of a particular person includes a lump of money big enough for the interest to pay for their maintenance. If I remember incorrectly and someone points it out, I will rescind my downvote.

View more: Next