I suppose "no sex before marriage" really depends on culture. If the most reliable way to get ANY sex is such a commitment, then you're probably right.

Outside of such a culture, though... well, the women with higher sex drives are presumably LESS likely to want to wait until marriage, so by waiting you both delay gratification AND probably end up with a partner who has below-average sex drive. AND you've made it more difficult for yourself to change partners.

Of course, whether the average male believes this to be true is another matter entirely :)

Suffice to say, if sex is a terminal value and you're able to move cities, this is blatantly the incorrect strategy. So either "no sex before marriage" is used by people who don't know better, or there's some other factor that has a higher priority :)

I can't think of any real gain to "no sex before marriage" beyond the status/reputation aspect, so I'd be forced to conclude that it is indeed an aspect of trading sex -> status/reputation

"Safer doesn't imply safe."

I think it's important to distinguish between what I consider a True Oracle - an AI with no internal motivation system, including goal systems - and an AGI which has been designed to -behave- like an Oracle. A True Oracle AI is -not- a general intelligence.

The difference is that an AGI designed to behave like an oracle tries to figure out what you want, and gives it to you. A True Oracle is necessarily quite stupid. From the linked article by Eliezer, this quote from Holden "constructutilityfunction(processuserinput()) is just a human-quality function for understanding what the speaker wants" represents the difference. Encapsulating utility into your Oracle means your Oracle is behaving more like an agent than a tool; it's making decisions about what you want without consulting you about it.

In fact, as far as I define such things, we already have Oracle AIs. The computer itself is one; you tell it what your problem is, and it solves it for you. If it gives you the wrong answer, it's entirely because your problem specification is incomplete or incorrect. When I read people's discussions of Oracle AIs, what it seems they really want is an AI that can figure out what problem you're -really- trying to solve, given a poorly-defined problem, and solve -that- for you.

-That- is what is dangerous.

Given that read-only hardware exists, yes, a clean border can be drawn, with the caveat that nothing is stopping the intelligence from emulating itself as if it were modified.

However - and it's an important however - emulating your own modified code isn't the same as modifying yourself. Just because you can imagine what your thought processes might be if you were sociopathic doesn't make you sociopathic; just because an AI can emulate a process to arrive at a different answer than it would have doesn't necessarily give it the power to -act- on that answer.

Which is to say, emulation can allow an AI to move past blocks on what it is permitted to think, but doesn't necessarily permit it to move past blocks on what it is permitted to do.

This is particularly important in the case of something like a goal system; if a bug would result in an AI breaking its own goal system on a self-modification, this bug becomes less significant if the goal system is read-only. It could emulate what it would do with a different goal system, but it would be evaluating solutions from that emulation within its original goal system.

Evolution and capitalism are both non-goal-oriented, extremely powerful intelligences. Goals are only one form of motivators.

there's nothing I'd really rather do

How confident are you about this? (And how well-calibrated do you think your confidence levels are?) Seems like it would be worth your time to verify this.

The AI program could have many different types of output, some of which are controlled by the AI, and some of which are not.

Can you elaborate on what you mean by "control" here? I am not sure we mean the same thing by it because:

This is not "AI output", because the AI cannot control it (except by actually changing its own beliefs), but it is "AI program output", because the program that outputs the answer is the same program as the one that performs all the cognition.

If the AI can control its memory (for example, if it can arbitrarily delete things from its memory) then it can control its beliefs.

Disengaging.

Aha. Thanks for the correction.

Now you're just ascribing magical powers to a potentially-transhuman AI. I'm sure there exists such a silver bullet, in fact by definition if security isn't 100%, that's just another way of saying there exists a strategy which will work; but that's ignoring the point about layers of security not being completely redundant with proofs and utility functions and decision theories, and adding some amount of safety.

Yes, of course the Silicon Valley attracts CompSci professionals from all over the world, but the SI doesn't employ them. Strange people you say? I've never been to San Francisco, but I've heard that it's considered home to weirdos of every possible kind. Maybe that's the people the SI panders to?

All people who like SI are by definition out of the mainstream, but not all people out of the mainstream are whom SI 'panders' to.

It's not like Peter Thiel doesn't know how to use the Internet or can't afford flying.

And yet...

Facebook, for instance, was located in Massachusetts and only moved to the Silicon Valley in 2011.

How wasteful of them. Don't they know they can just use the Internet to do this thing called 'social networking'? There's no reason for them to be in Silicon Valley. Hopefully their shareholders will do something about that.

What kind of difference?

My opposition to meat varies linearly with the intelligence of the animal. I'm much more OK with fish than I am pigs.

I wanted to say that anyone who is creating AGI need to control it some how and by this need somekind of analog of FAI, at least for not to be killed himself. And this idea could be promoted to any AGI reasearch group.

Just wanted to point out that many contributors to the site are afflicted by what I call "theoritis", a propensity to advance a theory despite being green amateurs in the subject matter, and then have the temerity to argue about it with the (clearly-non-stupid) experts in the field. The field in question can be psychology, neuroscience, physics, math, computer science, you name it.

It is rare that people consider a reverse situation first: what would I think of an amateur who argues with me in the area of my competence? For example, if you are an auto mechanic, would you take seriously someone who tells you how to diagnose and fix car issues without ever having done any repairs first? If not, why would you argue about quantum mechanics with a physicist, with a decision theorist about utility functions,or with a mathematician about first-order logic, unless that's your area of expertise? Of course, looking back it what I post about, I am no exception.

OK, I cannot bring myself to add philosophy to the list of "don't argue with the experts, learn from them" topics, but maybe it's because I don't know anything about philosophy.

I have an important choice to make in a few months (about what type of education to pursue). I have changed my mind once already, and after hearing a presentation where the presenter clearly favored my old choice, I'm about to revert my decision - in fact, introspection tells me that my decision was already changed at some point during the presentation. In regards to my original change of mind, I may also have been affected by the friend who gave me the idea.

All of this worries me, and I've started making a list of everything I know as far as pros/cons go of each choice. I want to weigh the options objectively and make a decision. I fear that, already favoring one of the two choices, I won't be objective.

How do I decrease my bias and get myself as close as possible to that awesome point at the start of a discussion where you can list pros and cons and describe the options without having yet gotten attached to any position?

I think the idea is that it is economical, but the patent-holder simply never thought of it.

See also.

how ostentatiously

Not at all -- not that I tried to hide the fact that I was eating meat, but I tried to be as nonchalant as I would be if I didn't know they were vegetarians. OTOH I'm not terribly good at hiding emotions, so probably some of them could tell I was feeling a little embarrassed.

The type of meat would also make a difference to me.

What kind of difference? Pork vs beef vs chicken? Steaks vs minced meat? Free-range vs factory farmed vs hunted (but how would you tell)?

I noticed that I cache a bunch of negative self-image thoughts related to things I think I'm supposed to be good at. For example, I've gotten pretty good at a sport but I don't practice a lot while I'm in school. So during the time that I didn't train, I internalized that I've gotten much worse that I used to be which made me less inclined to start training again. But then I went to practice during the holiday break, I noticed only a very manageable rustiness that was nowhere near the horrible failure I was projecting in my brain. So I'm going to try and not trust my negative self-estimates too much when they stop me from doing productive things.

Of course not, it just points out how California sucks in strange people and techies (and both) from around the world eg. my elder sister was raised and went to college on the east coast, but guess where she's working now? Silicon Valley.

Yes, of course the Silicon Valley attracts CompSci professionals from all over the world, but the SI doesn't employ them. Strange people you say? I've never been to San Francisco, but I've heard that it's considered home to weirdos of every possible kind. Maybe that's the people the SI panders to?

SI would never have gotten Thiel's support, I suspect, if it had remained in Atlanta. Having gotten his support, it will not keep it by moving out of Silicon Valley. Having moved out of Silicon Valley, it will find it hard to find any more donors.

Well, I dunno. It's not like Peter Thiel doesn't know how to use the Internet or can't afford flying. Facebook, for instance, was located in Massachusetts and only moved to the Silicon Valley in 2011.

I'm starting to get frustrated, because the things I'm trying to explain seem really simple to me, and yet apparently I'm failing to explain them.

When I say "the AI's output", I do not mean "the AI program's output". The AI program could have many different types of output, some of which are controlled by the AI, and some of which are not. By "the AI's output", I mean those outputs which are controlled by the AI. So the answer to your question is mu: the researchers would look at the program's output.

My above comment contains an example of what I would consider to be "AI program output" but not "AI output":

If a researcher types in "estimate the probability of Riemann's hypothesis" (but in some computer language, of course), that should query the AI's belief system directly, rather than informing the AI of the question and allowing it to choose whatever answer it wishes.

This is not "AI output", because the AI cannot control it (except by actually changing its own beliefs), but it is "AI program output", because the program that outputs the answer is the same program as the one that performs all the cognition.

I can imagine a clear dichotomy between "the AI" and "the AI program", but I don't know if I've done an adequate job of explaining what this dichotomy is. If I haven't, let me know, and I'll try to explain it.

Your expectation only holds if you expect failure to be perfectly correlated or multiple layers actually reduce the strength of layers, otherwise the probability of the AI beating layers A and B necessarily is less than beating just A or B (A ^B < A v B).

That's true. However I would expect a transhuman to be able to find a single point of failure which does not even occur to our limited minds, so this perfect correlation is a virtual certainty.

Most people whose paycheck comes from designing a bomb have no trouble rationalizing it. Similarly, if your paycheck depends on the AGI progress and not FAI progress, you will likely be unwilling to slow down or halt the AGI development progress, and if you are, you get fired and replaced.

ugh...I just did a rough estimate for the same problem with clocks...it's not much better. So much for that idea!

I wonder if there is a way to use math to squeeze more digits out of this situation...

Isn't the whole point of patents for people NOT to use them? If it's not economical for the patent-holders to profit from them isn't it even less economical for someone who would need to pay license fees to use them?

Even that is probably too naive, as there could well be other failure modes of which AI deboxing is but a side effect, and our limited human imagination will never going to catch them all. My expectation is that if you rely on safety triggers to bail you out (instead of including them as a desperate last-ditch pray-it-works defense), then you might as well not bother with boxing at all.

My whole point of 'defense in depth' was that each layer was highly fallible and could have errors. Your expectation only holds if you expect failure to be perfectly correlated or multiple layers actually reduce the strength of layers, otherwise the probability of the AI beating layers A and B necessarily is less than beating just A or B (A ^B < A v B).

I posted my current conundrum on the mentoring thread (http://lesswrong.com/r/discussion/lw/929/less_wrong_mentoring_thread/89c2), but since that's a pretty old thread, I figured I'd post a shortened version here, if you don't mind...

Here's my preliminary conclusion: I've been pursuing a profession that has (from what I've been able to tell) about a 1% chance of making a living at it. These are horrible odds, but since there's nothing I'd really rather do, it's like choosing between $5 and a 1% chance of getting $500, except that even if I don't get the $500, I still get the $5 ($5 representing, in my case, a job I'm not really interested in).

Does this sound rational? :\

EDIT: In a way, it's like the "just lose hope already" thing, but hopefully I'm avoiding that pitfall since I'm going to be adjusting my goals and strategy.

Assuming my math is right, if your stone carving were accurate to 1 micron, in order to encode a 140 character 'tweet' using this method, you would need a stone tablet 10^163 times larger than the observable universe. (!)

Be specific? What sort of triggers, what sort of dangerous territory? I can't tell if you're still relying on a human to outwit a transhuman or talking about something entirely different.

There is no reason you cannot put the AI in a box with some triggers for it venturing into dangerous territory

A trans-human intelligence ought to be able to model human one with ease. This means being able to predict potential triggers and being able to predict how to trick the lack-wit humans on the other end to unwittingly reveal the location of the triggers (even if they don't consciously know it themselves). So the only trigger that matters is one to detect a hint of an intent to get out. Even that is probably too naive, as there could well be other failure modes of which AI deboxing is but a side effect, and our limited human imagination will never going to catch them all. My expectation is that if you rely on safety triggers to bail you out (instead of including them as a desperate last-ditch pray-it-works defense), then you might as well not bother with boxing at all.

Last night I decided that since my current schedule of classes goes from noon until 10pm, I should treat all times as though they are 4 hours earlier and use this as my intuition for when to go to bed, which will allow me to do things like actually eat dinner at a reasonable time.

Then I got up this morning and realized that I have something at 10am today, and I stayed up far too late to be okay with that.

Minus one rationality point for adopting a change without thinking about it for five minutes.

AFAIK the SI doens't do software development or direct computer science research. Other than operating Less Wrong, their main outputs seem to be philosophical essays and some philosophical pubblications, plus the annual Singularity Summits (which makes sense to do in the Silicon Valley, but don't have to be physically close to the SI main location). A cursory look on the SI team pages suggests that most of the staff are not CompSci professionals, and many of them didn't get their education or did research at Stanford or other Silicon Valley colleges.

The audience and donors are there, which is enough, but your point about the people is not strong: most of the people in Silicon Valley was not taught at Stanford, does that mean they are wasting their time there? Of course not, it just points out how California sucks in strange people and techies (and both) from around the world eg. my elder sister was raised and went to college on the east coast, but guess where she's working now? Silicon Valley.

From the donors point of view, IIUC, most of the money donated to the SI comes from very few big donors, Peter Thiel in particular donates much more than everybody else (maybe more than everybody else combined?). I suppose that such donors would continue to support the SI even if it was relocated.

You suppose that because it is convenient for your claim that being in Silicon Valley is wasteful, not because it is true. The widespread absence of telecommuting in corporations, the worldwide emphasis on clustering into cities so you can be physically close to everyone, how donors in every charity like to physically meet principles and "look them in the eyes", the success of LW meetups - all these point to presence being better than absence.

SI would never have gotten Thiel's support, I suspect, if it had remained in Atlanta. Having gotten his support, it will not keep it by moving out of Silicon Valley. Having moved out of Silicon Valley, it will find it hard to find any more donors.

What, like Thiel is guaranteed to never drop support? Even in such an absurd situation, why would you risk it by ignoring all other big donors? And what if you wanted to grow? If SI were to leave Silicon Valley to save some money on salaries, it would be a major long-term strategic mistake which would justify everything critics might say about SI being incompetent in choosing to be penny-wise and pound-foolish.

Even assuming that there are benefits from staying in the Silicon Valley that outweight the costs, the point stands that Yudkowsky could accept a lower salary while still staying well above subsistence level.

Dunno, but that wasn't the point I was addressing.

I agree. And I don't see why Eliezer expects that people MOSTLY disagree on the difficulty of success, even if some (like the OP) do.

When I talk casually to people and tell them I expect the world to end they smile and nod.

When I talk casually to people and tell them that the things they value are complicated and even being specific in English about that is difficult, they agree and we have extensive conversations.

So my (extremely limited) data points suggest that the main point of contention between Eliezer's view and the views of most people who at least have some background in formal logic, is that they don't see this as an important problem rather than that they don't see it as a difficult problem.

Therefore, when Eliezer dismisses criticism that the problem is easy as the main criticism, in the way I pointed out in my comment, it feels weird and misdirected to me.

That's not 'boxing'. Boxing is a human pitting their wits against a potentially hostile transhuman over a text channel and it is stupid.

That was how you did your boxing experiments, but I've never taken it to be so arbitrarily limited in goals, capacities, or strategies on either end. There is no reason you cannot put the AI in a box with some triggers for it venturing into dangerous territory, and this would be merely sane for anyone doing such a thing.

In what way would SI be 'trying it'? The point about multiple layers of security being a good idea for any seed AI project has been made at least as far back as Eliezer's CFAI and brought up periodically since with innovations like the suicide button and homomorphic encryption.

Laziness can muddy the waters, but it's also optional in functional programming. People using haskell in a practical setting usually avoid it and are coming up with new language extensions to make strict evaluation the default (like in records for example).

What you're really saying is the causal link between assembly and the language is less obvious, which is certainly true as it is a very high level language. However, if we're talking about the causality of the language itself, then functional languages enforce a more transparent causal structure of the code itself.

You can be certain that a function that isn't tainted by IO in haskell, for example, isn't going to involve dozens of different causal structures. An imperative function like AnimalFactory.create("dog") could involve dozens of different dependencies (e.g. through singletons or dependency injection) making the dependency graph (and causal structure) obfuscated. This lack of transparent guarantees about state and dependencies in imperative languages makes concurrent/parallelprogramming (and even plain code) very difficult to reason about and test.

Moreover, the concessions that haskell has given way to are probably temporary. Haskell is a research language and functional solutions to problems like IO and event driven programs have been put forward but are not yet widely accepted. And even ignoring these solutions, you still have a basic paradigm where you have top level imperative style code with everything else being functional.

And while it can be more difficult to debug functional programs, they're easier to test, and they're less prone to runtime bugs. And really, the debugging problem is one of laziness and difficult to use debuggers. Debugging F# with visual studio's debugger isn't that difficult.

(Note: that when I talk about functional programming, I'm talking about a paradigm that avoids mutable state and data rather than idiomatic approaches to container manipulation)

Fixed. Thanks.

the wrong actions can trigger some invariant and signal that something went wrong with the decision theory or utility function

That's not 'boxing'. Boxing is a human pitting their wits against a potentially hostile transhuman over a text channel and it is stupid. What you're describing is some case where we think that even after 'proving' some set of invariants, we can still describe a high-level behavior X such that detecting X either indicates global failure with high-enough probability that we would want to shut down the AI after detecting any of many possible things in the reference class of X, or alternatively, we think that X has a probability of flagging failure and that we afterward stand a chance of doing a trace-back to determine more precisely if something is wrong. Having X stay in place as code after the AI self-modifies will require solving a hard open problem in FAI for having a nontrivially structured utility function such that X looks like instrumentally a good thing (your utility function must yield, 'under circumstances X it is better that I be suspended and examined than that I continue to do whatever I would otherwise calculate as the instrumentally right thing). This is how you would describe on a higher level of abstraction an attempt to write a tripwire that immediately detects an attempt to search out a strategy for deceiving the programmers as the goal is formed and before the strategy is actually searched.

There's another class of things Y where we think that humans should monitor surface indicators because a human might flag something that we can't yet reify as code, and this potentially indicates a halt-melt-and-catch-fire-worthy problem. This is how you would describe on a higher level of abstraction the 'Last Judge' concept from the original CEV essay.

All of these things have fundamental limitations in terms of our ability to describe X and monitor Y; they are fallback strategies rather than core strategies. If you have a core strategy that can work throughout, these things can flag exceptions indicating that your core strategy is fundamentally not working and you need to give up on that entire strategy. Their actual impact on safety is that they give a chance of detecting an unsafe approach early enough that you can still give up on it. Meddling dabblers invariably want to follow a strategy of detecting such problems, correcting them, and then saying afterward that the AI is back on track, which is one of those things that is suicide that they think might have an 80% chance of working or whatever.

Little or no artificial light after dark for better sleep and mood

Suppose that a hundred years ago, Sam was considering the possibility of the eventual existence of people like us living lives like ours, and deciding how many resources to devote to increasing the likelihood of that existence.

I'm not positing prophetic abilities here; I don't mean he's peering into a crystal ball and seeing Dave and Ghatanathoah. I mean, rather, that he is considering in a general way the possibility of people who might exist in a century and the sorts of lives they might live and the value of those lives. For simplicity's sake I assume that Sam is very very smart, and his forecasts are generally pretty accurate.

We seem to be in agreement that Sam ought to care about us (as well as the various other hypothetical future people who don't exist in our world). It seems to follow that he ought to be willing to devote resources to us. (My culture sometimes calls this investing in the future, and we at the very least talk as though it were a good thing.)

Agreed?

Since Sam does not have unlimited resources, resources he devotes to that project will tend to be resources that aren't available to other projects, like satisfying the preferences of his neighbors. This isn't necessary... it may be, for example, that the best way to benefit you and I is to ensure that our grandparents' preferences were fully satisfied... but it's possible.

Agreed?

And if I'm understanding you correctly, you're saying that if it turns out that devoting resources towards arranging for the existence of our lives does require depriving his neighbors of resources that could be used to satisfy their preferences, it's nevertheless OK -- perhaps even good -- for Sam to devote those resources that way.

Yes?

What's not OK, on your account, is for Sam to harm his neighbors in order to arrange for the existence of our lives , since his neighbors already have preferences and we don't.

Have I understood you so far?

If so, can you clarify the distinction between harming me and diverting resources away from the satisfaction of my preferences, and why the latter is OK but the former is not?

The set theory book only links to an image of the book, not the amazon page,

I have a client!! Not just a one time thing that a mentor referred me, but a result of my own marketing efforts (and thus evidence that the business is sustainable).

Makes me happy because it is some evidence that I'm being rational in opening the law firm - which is very reassuring. Still, more marketing / networking is necessary. And getting good marketing habits has been a struggle.

Well, yes, but that will break the ceteris paribus for the anthropics.

I'd rather just see it as a different way of mathematically describing the same thing. Greatly simplifying, you can either have a law of X=Y or you can have plurality of solutions inclusive of one with X=Y and an unstable condition where when X!=Y everyone's twins "die". In a sense that's merely two different ways of writing down exact same thing. It might be easier to express gravitation as survivor bias, that would make us use such formalism, but otherwise, the choice is arbitrary. Also, depending to how vacuum decay is triggered, one can obtain, effectively, an objective collapse theory.

With regards to probabilities, your continued existence constitutes incredibly strong evidence that the relevant 'probability' does not dramatically decrease over time.

That's only because you're not just adding an option, you're changing the choice you already have. You're not going from X to a choice between X and Y, but rather from X to a choice between X+(B knows A prefers this choice to the other) and Y+(B knows A prefers this choice to the other). This has nothing to do with there being another agent about, it's just the fact that you're changing the options available at the same time as you add extra ones.

EDIT: As a general principle, there shouldn't be any results that hold except when there are other agents about, because other agents are just part of the universe.

You're living the dream. That's amazing, and I don't want to do anything to discount that. You've done well in breaking your previous bad habits, and I think you certainly are smart and dedicated pull this off. What are your plans for sleep, weather, thieves, and explaining the job history gap? Improvising is half of what makes runs like that exciting, but it's probably better to have a plan in advance for the more predictable issues. Just because there is a plan doesn't mean you have to stick to it.

Your failure mode sounds a lot like what I just got out of doing, and yours is a more coherent explanation of why I can't do college right now than I've managed to articulate. You go [insert appropriate gender noun]! Your bike ride sounds similar to the fantasy I had about packing up (Well, mosting throwing away) everything I own and driving off to see more exciting places. I'd say the bike is a better idea because it provides a focus for your energy and gas is expensive, but I really do appreciate having a steady place to sleep and guard myself from weather's whims.

I'm currently stuck somewhere between 1 and 2. My job doesn't take much effort on my part, but I've become increasingly aware that it does take a good amount of time out of my day that I could be spending on things I actually want to do.

Anyone heard of Marblar?.

The idea is to crowdsource uses for the huge number of patents and new technologies generated by universities but never used, and awarding prizes. Seems like a really clever idea to capture low hanging fruit, and the sort of thing LW people should be quite good at,

Actually, Thinking is one of the required books of the course. The prof was apparently taught by Kahneman at one point.

I think you accidentally a word there.

(semi-rationally) not granting the possibility that any one thing can be as important as you feel AI is

How much damage failure would do is a separate question to how easy it is to achieve success.

Http://intonarumoron.wordpress.com

I've collected music videos since the mid 1980s. Electronic and experimental and strange, mostly. Being able to link to or download videos instead of dub them on videotape has nearly filled a TB drive and inspired my blog. Rather than list a few recent favorites, see above for years of favorites. I post weekly, my fellow intonarumorons irregularly.

I mean, maybe you're here as a representative of the people running the machine simulating me

Honestly, I would have upvoted just for this bit.

or at least they said that in the future SI's research may lead to implementation secrets that should not be shared with others. I didn't understand why that should be.

It seems pretty understandable to me... SI may end up having some insights that could speed up UFAI progress if made public, and at the same time provably-Friendly AI may be much more difficult than UFAI. For example, suppose that in order to build a provably-Friendly AI, you may have to first understand how to build an AI that works with an arbitrary utility function, and then it will take much longer to figure out how to specify the correct utility function.

If you create a precise duplicate of the universe in a simulation, I don't regard that we have gained anything; I consider that two instances of indistinguishable utility aren't cumulative. If you create a precise duplicate of me in a simulation and then torture that duplicate, utility decreases.

This may seem to be favoring "average" utility, but I think the distinction is in the fact that torturing an entity represents, not lower utility, but disutility; because I regard a duplicate universe as adding no utility, the negative utility shows up as a net loss.

I'm basically assuming this reality-fluid stuff is legit for the purposes of this post. I included the most common argument in it's favor (the probability argument) but I'm not setting out to defend it, I'm just exploring the consequences.

For instance, should I imply a "vastly" in front of moral weight as well as if there is a 1:1 correspondence or should I not do that?

Pretty much, yeah.

Is this the only moral consideration you are considering on this tier? (I.E, there may be other moral considerations, but if this is the only "vast" one, it will probably outweigh all others.)

Well, I'm considering the torture's disutility, and the torturers' utility.

Does the arrangement of the copies reality fluid matter? Omega is usually thought of as a computer, so I am considering the file system. He might have 3 copies in 1 file for resilience, such as in a RAID array. Or he can have 3 copies that link to 3 files, such as in just having Sim001.exe and Sim002.exe and Sim003.exe having the exact same contents and being in the same folder. In both cases, the copies are identical. And if they are being run simultaneously and updated simultaneously, then the copies might not be able to tell which structure Omega was using. Which of these are you envisioning (or would it not matter? [Or do I not understand what a RAID array is?])

I'm not entirely sure I understand this question, but I don't think it should matter.

"Please divide the length of this sentence by the length of the stone block it is written on and convert to base 26 in order to reveal a code which can be cyphered using the obvious number-to-letter mapping"

I don't think it would be feasible to encode even half a dozen letters with that technique.

This reminds me of a competition for extra credit in one of my CS classes to write a C++ program in "as few statements as possible," where I took the obvious algorithm, completely unrolled the loop, and used logical connectives to stick every statement together into one.

That's exactly what I would've done!

although there might be useful methods in common.

Indeed - for example, on the F front, computational models of human ethical reasoning seem like something that could help increase the safety of all kinds of AGI projects and also be useful for Friendliness theory in general, and some of them could conceivably be developed in the context of heuristic AGI. Likewise, for the AGI aspect, it seems like there should be all kinds of machine learning techniques and advances in probability theory (for example) that would be equally useful for pretty much any kind of AGI - after all, we already know that an understanding of e.g. Bayes' theorem and expected utility will be necessary for pretty much any kind of AGI implementation, so why should we assume that all of the insights that will be useful in many kinds of contexts would have been developed already?

Making a poorly understood AGI doesn't necessarily help with implementing a FAI (even if you have the theory figured out)

Right, by the above I meant to say "the right kind of AGI + Friendliness"; I certainly agree that there are many conceivable ways of building AGIs that would be impossible to ever make Friendly.

Let N=3^^^^^^3, surely N nice world + another nice world is better than N nice worlds + a torture world. Why? Because another nice world is better than a torture world, and the prior existence of the N previous worlds shouldn't matter to that decision.

The torture world, in this case, is being used to satisfy the whims of the Niceworld's residents. Lots of Niceworld copies = lots of Reality = lots of utility. So goes the logic.

Sure, but equally the expected utility of being the master of a torture world with probability 1/(N+1) can be neglected.

Since they are all the same, they can share a torture world.

That it is true, if unintuitive, that people gain moral worth the more "real" they get, is a position I have seen on LW, and the arguments do seem reasonable. (It is also rather more coherent when used in a Big Universe.) This post assumes that position, and includes a short version of the most common argument for that position.

Incidentally, I used to hold the position you describe; how do you deal with the fact that a tortured copy is, by definition, no longer "part" of the original?

Yes, I'm starting to rethink that. But is still seems that we could have physics A, with vacuum decay, and physics B, without, such that internal observers made the same observation in either case.

The standard solution to that is to be altruistic to some group of people as they existed at time T, and the standard problem with that is it doesn't allow moral progress, and the standard solution to that is to be altruistic to some idealized or extrapolated group of people. So we just have to make the heuristics-based FAI understand the concept of CEV (or whatever the right notion of "idealized" is), which doesn't seem impossible. What does seem impossible is to achieve high confidence that it understands the notion correctly, but if provably-Friendly AI is just too slow or unfeasible, and we're not trying to achieve 100% safety...

The 13th amendment didn't obveously destroy any value; it just transfered value from slaveowners to slaves.

Nor does this discussion have anything to do with incentives or profits. I'm just trying to quantify the value of the existing gun stock.

I think one reason why people (including me, in the past) have difficulty accepting the way you present this argument is that you're speaking in too abstract terms, while many of the values that we'd actually like to preserve are ones that we appreciate the most if we consider them in "near" mode. It might work better if you gave concrete examples of ways by which there could be a catastrophic value drift, like naming Bostrom's all-work-and-no-fun scenario where

what will maximize fitness in the future will be nothing but non-stop high-intensity drudgery, work of a drab and repetitive nature, aimed at improving the eighth decimal of some economic output measure

or some similar example.

Another question regarding two books: I have Thinking, Fast and Slow by Kahneman's. How much more beneficial would it be to get Heuristics and Biases, the fuller works?

Even evil creator of AI needs somekind of controll over his child, that could be called friendliness to one person. So any group which is seriosly creating AGI and going to use it it in any efforts should be interested in FAI theory. So it could be enough to explain to any one who create AGI that he needs somekind of F-theory and it should be mathematically proven.

More than that, I think two or so an episode. There are three in the first (Mrs Hudson, Angelo, and Mycroft), and that's if you count Angelo's shipping of Johnlock and Sherlock's clumsy attempt to let John down gently as one joke. Oddly enough, I can't find a tally, so I'll keep one on my next marathon.

The jokes are good - anything that causes Freeman to act one of his nine or so flavors of exasperation is automatic comedy gold. I'm just complaining about the frequency.

they did it rather inconsistently

I think the mood dictates that. A Study In Pink is meant to show Sherlock's abilities, so we can exclaim "Fantastic!" in chorus with John, which is why we get both clue highlighting and expospeak. Baskerville is about Sherlock losing it a little, so making things less clear helps.

I agree that the characters are sometimes dense. In Reichenbach, Sherlock misses or takes forever to get nearly all of Moriarty's hints, though a large part of it is probably playing dumb. (Moriarty's last move genuinely surprises him, but he didn't phone that one in.) My personal theory for his abysmal stupidity in A Study In Pink is that he starts out incapable of any thinking while distracted (e.g. by Anderson's face) and that improvement in this area is one of the benefits of having a sidekick-caregiver-sober coach.

My point is that the lifetime of a pig (EDIT: being farmed for meat) isn't very long (about 6 months from what I can find on the internet). Thus all we would have to do is stop breeding them for a while and we very quickly wouldn't have many pigs.

Yeah, it is true that Finnish universities do generally require your Master's to be pretty similar than your undergrad - for example, looking at the admission criteria document from my faculty's international admission pages (it's my understanding that other universities have similar policies), it says that

Following receipt of a reasoned application, the admissions board may grant the right to pursue the Master of Science degree without an entrance examination to an applicant who has completed an applicable Bachelor’s degree with good grades. In such cases, the applicant must meet the Finnish, Swedish or English language requirements, and the content and scope of the completed degree must be correspond sufficiently to the Bachelor of Science degree in the same discipline at the Faculty of Science. Admission will be based on the amount, quality and grades of the completed studies as well as on the letter of motivation.

On the other hand, it does also say that

If the content of a successful applicant’s Bachelor’s degree does not correspond sufficiently to the Faculty requirements for the Bachelor of Science degree in the relevant major subject, the applicant may have to complete up to 60 credits of supplementary major subject, minor subject and language studies before beginning to pursue advanced studies for the Master’s degree.

But I don't know how similar the previous degree has to be in order for them to say "okay, but you have to complete extra courses" instead of rejecting the application outright.

There are compelling reasons to be there: it is the epicenter of the global tech world. You will not find a place more interested in these topics, with more potential donors, critics, potential employees, etc.

This is the same reasoning for why the WikiMedia Foundation moved from St Petersburg, Florida to San Francisco back in 2007 or 2008 or so: that they would be able to recruit more talent and access more big donors.

AFAIK the SI doens't do software development or direct computer science research. Other than operating Less Wrong, their main outputs seem to be philosophical essays and some philosophical pubblications, plus the annual Singularity Summits (which makes sense to do in the Silicon Valley, but don't have to be physically close to the SI main location). A cursory look on the SI team pages suggests that most of the staff are not CompSci professionals, and many of them didn't get their education or did research at Stanford or other Silicon Valley colleges.

From the donors point of view, IIUC, most of the money donated to the SI comes from very few big donors, Peter Thiel in particular donates much more than everybody else (maybe more than everybody else combined?). I suppose that such donors would continue to support the SI even if it was relocated.

Even assuming that there are benefits from staying in the Silicon Valley that outweight the costs, the point stands that Yudkowsky could accept a lower salary while still staying well above subsistence level.

This listing looks like largely undergraduate courses, and unfortunately I don't speak Finnish! Though in the US, some Master's programs are set up to accept students who want to do something different than what they did in undergrad, I have no idea if this applies to Finnish universities at all. And honestly Finland sounds like a wonderful country, would not mind moving there except maybe because of the cold.

Yes, it is this layered approach that the OP is asking about -- I don't see that SI is trying it.

provably-Friendly AI has a significant speed disadvantage, as the OP argues.

Yes, the OP made that point. But I have heard the opposite from SI-ers -- or at least they said that in the future SI's research may lead to implementation secrets that should not be shared with others. I didn't understand why that should be.

Statements to the effect that it's necessary to argue that hard takeoff is probable/possible in order to motivate FAI research appear regularly, even your post left this same impression.

No, I didn't want to give that impression. SI's research direction is the most important one, regardless of whether we face a fast or slow takeoff. The question raised was whether other approaches are needed too.

It's not obvious to me that the main barrier to people pursuing AI safety is

living in a verbal-English fantasy world

As opposed to (semi-rationally) not granting the possibility that any one thing can be as important as you feel AI is; perhaps combined with some lack of cross-domain thinking and poorly designed incentive systems. The above comments always seem pretty weird to me (especially considering that cryptographers seem to share these intuitions about security being hard.

I essentially agree with the rest of the parent.

Right, SI's basic idea is correct.

However, given that WBE's will in any case be developed (and we can mention IA as well) , I'd like to see more consideration of how to keep brain-based AI's as safe as possible before they enter their Intelligence Explosion -- even though we understand that after an Explosion, there is little you can do.

But I would certainly expect the strategy of 'understanding the problem' to produce Event-Horizon-level results faster than 'do stuff that seems like it might work'.

The two are not mutually exclusive. The smarter non-SI teams will most likely try to 'understand the problem ' as best they can, experimenting and plugging gaps with 'stuff that seems that it might work', for which they will likely have some degree of understanding as well.

Little consideration has been given to a block on self-modification because it seems that it is impossible. You could do a non-Von Neumann machine, separating data and code, but data can be interpreted as code.

Still, consideration should be given to whether anything can be done, even if only as stopgap.

Whatever weird physics arguments you make, it all adds up to reality. So I would look through the self-consistent theories and choose the one that didn't make me make decisions I disapprove of all the time.

The causality in a functional language is far from obvious. Consider Haskell, a language that is both purely functional and lazy, and is considered somewhat of a beautiful poster child of the functional approach. Say you write a program and it has a bug -- it's not doing what it's supposed to. How would you debug it? Some alternatives:

(a) Use a debugger to step through a program until it does something it's not supposed to (this entails dealing with a causal order of evaluation of statements -- something Haskell as a lazy and functional language is explicitly hiding from you until you start a debugger).

(b) Use good ol' print statements. These will appear in a very strange order because of lazy evaluation. Again, Haskell hides the true order -- the true order has nothing to do with the way the code appears on the page. This makes it difficult to build a causal model of what's going on in your program. A causal model is what you need if you want to use print statements to debug.

(c) Intervene in a program by changing some intermediate runtime value to see what would happen to the output. As a functional language, Haskell does not allow you to change state (ignoring monads which are a very complicated beast, and at any rate would not support a straightforward value change while debugging anyways).

My claim is that causality is so central to how human beings think about complex computer programs that it is not possible to write and debug large programs written in functional style without either building a causal model of your program (something most functional language will fight with you about to the extent that they are functional), or mostly sticking to an imperative "causal" style, and only use simple functional idioms that you know work and that do not require further thinking (like map and reduce, and simple closure use). Note that even Haskell, a language committed to "wearing the hair shirt" (http://research.microsoft.com/en-us/um/people/simonpj/papers/haskell-retrospective/haskellretrospective.pdf) of functional programming, has given concessions to the imperative/causal writing style by providing the "do" shorthand.

Personally, I love functional idioms, and I think functional code with heavy recursion use is often quite beautiful. But I don't think the functional approach is well suited to writing complex code precisely because it is violating a principal rule of computer science -- make things easy for the human at the expense of the machine.

I used diigo for annotation before clearly had highlighting. Now, just as you, use diigo for link storage and Evernote for content storage. Diigo annotation has still the advantage that it excerpts the text you highlight. With Clearly if I want to have the highlighted parts I have to find and manually select them again... Also tagging from clearly requires 5 or so clicks which is ridiculous... But I hope it will get fixed.

I plan to use pocket once I get a tablet... it is pretty and convenient, but the most likely to get cut out of the workflow.

Thanks for the evernote import function - I'll look into it, maybe it could make the Evenote - org-mode integration tighter. Even then, having 3 separate systems is not quite optimal...

Focusing on the money makes a lot of sense to me. If we are honest with ourselves about the monetary incentives at work here, the whole discussion gets more realistic.

I'm also reminded of the historical conversation having to do with the 13th amendment: In one swoop of the pen, a vast sum of money was wiped off the books, the value of all that property which was now nullified.

I don't have a lot of ideas on how to make guns less profitable- unlike drugs, their high value has less to do with their legal status. But I don't think the gun lobby has got the nation's best interests in mind.