Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: denkenberger 19 September 2017 04:15:16PM *  6 points [-]

Thank you, Jennifer, for the introduction. Some more background on me: I have read the sequences and the foom debate. In 2011, I tried to do cost-effectiveness scoping for all causes inspired by Yudkowsky's scope and neglectedness framework (the scope, neglectedness, and tractability framework had not yet been invented). I am concerned about AI risk, and have been working with Alexey Turchin. I am primarily motivated by existential risk reduction. If we lose anthropological civilization (defined by cooperation outside the clan), we may not recover for the following reasons:

• Easily accessible fossil fuels and minerals exhausted

• Don’t have the stable climate of last 10,000 years

• Lose trust or IQ permanently

• Endemic disease prevents high population density

• Permanent loss of grains precludes high population density

Not recovering is a form of existential risk (not realizing our potential), and we might actually go extinct because of a supervolcano or asteroid after losing civilization. Because getting prepared (research and development of non-sunlight dependent foods such as mushrooms and natural gas digesting bacteria, and planning) is so cost-effective for the present generation, I think it will be a very cost effective way of reducing existential risk.

Comment author: dogiv 19 September 2017 06:28:51PM 0 points [-]

Thanks for posting!

I haven't read your book yet but I find your work pretty interesting. I hope you won't mind a naive question... you've mentioned non-sunlight-dependent foods like mushrooms and leaf tea. Is it actually possible for a human to survive on foods like this? Has anybody self-experimented with it?

By my calculation, a person who needs 1800 kcals/day would have to eat about 5 kg of mushrooms. Tea (the normal kind, anyway) doesn't look any better.

Bacteria fed by natural gas seems like a very promising food source--and one that might even be viable outside of catastrophe scenarios. Apparently it's being used for fish feed already.

Comment author: Erfeyah 30 August 2017 10:19:12AM *  3 points [-]

It is extremely interesting to see the attempts of the community to justify through or extract values from rationality. I have been pointing to the alternative perspective, based on the work of Jordan Peterson, in which morality is grounded on evolved behavioral patterns. It is rationally coherent and strongly supported by evidence. The only 'downside' if you can call it that is that it turns out that morality is not based on rationality and the "ought from an is" problem is an accurate portrayal of our current (and maybe general) situation.

I am not going to expand on this unless you are interested but I have a question. What does the rationalist community in general, and your article, try to get at? I can think of two possibilities:

[1] that morality is based on rational thought as expressed through language

[2] that morality has a computational basis implemented somewhere in the brain and accessed through the conscious mind as an intuition

I do not see how [1] can be true since we can observe the emergence of moral values in cultures in which rationality is hardly developed. Furthermore, even today as your article shows, we are straggling to extract value from rational argument so, our intuition can not be stemming from something we haven't even succeeded at. As for [2], it is a very interesting proposal but I haven't seen any scientific evidence that link it to structures in the human brain.

I feel the rationalist community is resistant in entertaining the alternative because, if true, it would show that rationality is not the foundation of everything but a tool of assessing and manipulating. Maybe further resistance is caused because (in an slightly embarrassing turn of events) it brings stories, myth and religion into the picture again, albeit in a very different manner. But even if that proves to be the case so what? What is our highest priority here? Rationality or Truth?

Comment author: dogiv 30 August 2017 08:00:19PM 1 point [-]

I think many of us "rationalists" here would agree that rationality is a tool for assessing and manipulating reality. I would say much the same about morality. There's not really a dichotomy between morality being "grounded on evolved behavioral patterns" and having "a computational basis implemented somewhere in the brain and accessed through the conscious mind as an intuition". Rather, the moral intuitions we have are computed in our brains, and the form of that computation is determined both by the selection pressures of evolution and the ways that our evolved brain structures interact with our various environments.

So what is our highest priority here? It's neither Rationality nor Truth, but Morality in the broad sense--the somewhat arbitrary and largely incoherent set of states of reality that our moral intuition prefers. I say arbitrary because our moral intuition does not aim entirely at the optimization target of the evolutionary process that generated it--propagating our genes. Call that moral relativism if you want to.

Comment author: dogiv 25 July 2017 09:14:02PM *  9 points [-]

I think this is an interesting and useful view, if applied judiciously. In particular, it will always tend to be most relevant for crony beliefs--beliefs that affect the belief-holder's life mainly through other people's opinions of them, like much of politics and some of religion. When it comes to close-up stuff that can cause benefit or harm directly, you will find that most people really do have a model of the world. When you ask someone whether so-and-so would make a good president, the answer is often a signal about their cultural affiliations. Ask them which is the fastest way to get to where they work, and the answer reflects what they've learned about rush-hour traffic patterns. Ask people if they believe in God, and the answer is a signal. Ask them if they believe pre-marital sex is ever acceptable, and the answer you get is a lot more practical.

It's also worth unpacking the us-vs-them terminology you employ here. Many of us may tend to be more literal than the average person (especially those who fall on the spectrum) but in my experience we are still prone to this same behavior. In most cases, there's nothing wrong with that. Understanding the difference can help us avoid trying to cooperatively world-model with people who are just expressing social beliefs, and can also help us recognize world-modeling when we see it, so that we can reduce our tendency to make snap judgements about people on the basis of the beliefs they express.

Comment author: Stabilizer 22 July 2017 05:04:03AM *  4 points [-]

I don't have time to refute each of arguments, because there're too many. But consider number 5 in your list. He describes a laser experiment that he claims cannot be accounted for on the current picture of the Earth. But if you think it through, it is perfectly well accounted for.

Here's the version of the experiment performed by the two Polish guys on a lake. They place two stakes 2km apart. The stakes have lasers attached to them at 30 cm height from the surface of the water. They measure the height above the surface of the point at which the laser beams meet and find it to be 39-40cm above the surface of the water.

Wild Heretic claims, on the basis of this diagram, that on the convex Earth theory (i.e., the widely accepted theory) one should expect the height from the water at the point where the lasers meet to be smaller that the height at which the lasers are mounted. But Wild Heretic's diagram misrepresents the state of affairs. Here is a better representation I drew and associated calculations that I did, which show that the convex Earth theory correctly predicts that the laser beams would meet approximately 38cm above the surface, which is very close to the observed 39-40cm.

EDIT: As dogiv points out below, I mis-interpreted the experiment. So the argument above is not a refutation of the experiment as described.

Comment author: dogiv 24 July 2017 03:22:50PM 1 point [-]

This doesn't actually seem to match the description. They only talk about having used one laser, with two stakes, whereas your diagram requires using two lasers. Your setup would be quite difficult to achieve, since you would somehow have to get both lasers perfectly horizontal; I'm not sure a standard laser level would give you this kind of precision. In the version they describe, they level the laser by checking the height of the beam on a second stake. This seems relatively easy.

My guess is they just never did the experiment, or they lied about the result. But it would be kind of interesting to repeat it sometime.

Comment author: turchin 26 June 2017 09:50:47PM *  1 point [-]

Thanks for interesting post. I think that there are two types of self-modification. In the first, an agent is working on lower level parts of itself, for example, by adding hardware or connecting modules. It produces evolutionary development with small returns and is relatively safe.

Another type is high-level self-modification, where the second agent is created, as you describe. Its performance should be mathematically proved (that is difficult) or tested in many simulated environments (which is also risky, as a superior agent will be able to break through it.) We could call it a revolutionary way of self-improvement. Such self-modification will provide higher returns if successful.

Knowing all this, most agents will prefer evolutionary development, that is gaining the same power by lower-level changes. But risk-hungry agents will still prefer revolutionary methods, in case if they are time constrained.

Early stage AI will be time constrained by arms race with other (possible) AIs, and it will prefer risky revolutionary ways of development, even if its probability of failure will be very high.

(It was TL;DR of my text "Levels of self-improvement".)

Comment author: dogiv 28 June 2017 08:22:55PM 1 point [-]

Thanks, that's an interesting perspective. I think even high-level self-modification can be relatively safe with sufficient asymmetry in resources--simulated environments give a large advantage to the original, especially if the successor can be started with no memories of anything outside the simulation. Only an extreme difference in intelligence between the two would overcome that.

Of course, the problem of transmitting values to a successor without giving it any information about the world is a tricky one, since most of the values we care about are linked to reality. But maybe some values are basic enough to be grounded purely in math that applies to any circumstances.

Comment author: cousin_it 27 June 2017 03:45:54PM *  1 point [-]

Yeah, Schelling's "Strategy of Conflict" deals with many of the same topics.

A: "I would have an advantage in war so I demand a bigger share now" B: "Prove it" A: "Giving you the info would squander my advantage" B: "Let's agree on a procedure to check the info, and I precommit to giving you a bigger share if the check succeeds" A: "Cool"

Comment author: dogiv 28 June 2017 07:53:08PM 0 points [-]

If visible precommitment by B requires it to share the source code for its successor AI, then it would also be giving up any hidden information it has. Essentially both sides have to be willing to share all information with each other, creating some sort of neutral arbitration about which side would have won and at what cost to the other. That basically means creating a merged superintelligence is necessary just to start the bargaining process, since they each have to prove to the other that the neutral arbiter will control all relevant resources to prevent cheating.

Realistically, there will be many cases where one side thinks its hidden information is sufficient to make the cost of conflict smaller than the costs associated with bargaining, especially given the potential for cheating.

Comment author: dogiv 26 June 2017 01:37:27AM *  5 points [-]

I've read a couple of Lou Keep's essays in this series and I find his writing style very off-putting. It seems like there's a deep idea about society and social-economic structures buried in there, but it's obscured by a hodgepodge of thesis-antithesis and vague self-reference.

As best I can tell, his point is that irrational beliefs like belief in magic (specifically, protection from bullets) can be useful for a community (by encouraging everyone to resist attackers together) even though it is not beneficial to the individual (since it doesn't prevent death when shot). He relates this to Seeing Like A State, in that any attempt by the state to increase legibility by clarifying the benefits makes them disappear.

He further points out that political and economic policies tend to focus on measurable effects, whereas the ultimate point of governments and economies is to improve the subjective wellbeing of people (happiness, although he says that's just a stand-in for something else he doesn't feel like explaining).

Extending that, he thinks we have probably lost some key cultural traditions that were very important to the quality of people's lives, but weren't able to thrive in a modern economic setting. He doesn't give any examples of that, although he mentions marriages and funerals as examples of traditions that have survived. Still, it seems plausible.

Overall, it reminds me of Scott Alexander's essay How the West was Won, about the advance of universalist (capitalist) culture and its ability to out-compete traditional systems whether or not it actually improves people's lives. Moloch is also relevant.

It's very likely I've missed a key aspect here. If anyone knows what it is, please let me know.

Comment author: Screwtape 20 June 2017 04:09:15AM 2 points [-]

There have been a couple of community building projects put forward that got me thinking about this, and then over in the post about ways to make the community better it was suggested that some people might want to get to know other lesswrongers through D&D*. I love that idea. Tabletop RPGs are the fastest way I know of to build a connection with someone that doesn't leave scars. While the concept of an 'expert' in those games is sort of goofy, I figure I've got plenty of experience and interest in them to run something or organize a LessWrong RPG group. I haven't been terribly active around LessWrong itself, but I'm the guy who ran the Dungeons and Discourse game with the boring machine about a year ago and I'm also the guy who infodumped about Exalted at the NY Solstice afterparty if you wanted nerd credentials.

Who would want to play?

*D&D is used here as a generic name for these sorts of games, sort of like how if I ask for a Kleenex I mean any sort of disposable thing I can sneeze in. I've got a bunch of systems and genres I could grab from, depending on what people thought sounded cool.

Comment author: dogiv 23 June 2017 03:50:09PM 0 points [-]

Are you talking about a local game in NY or a correspondence thing?

Comment author: turchin 22 June 2017 02:15:32PM 2 points [-]

Wireheading bomb – a putative mechanism to stop dangerous AI.

If a reward function of an AI is presented openly in its source code, any attempt to self-improve by AI will result in its own immediate wireheading, as when it reaches its own source code, it will become able to modify it in order to get maximum reward. So we could create an AI architecture in the way that as soon as it gets access to its own source code, it stops, and use it as a way of reaching passive safety and self-limited self-improving capacity.

We also could do exactly opposite, and put a reward function into the remote impenetrable cryptographically protected box, so AI will not be able to wirehead itself in an unexpected moment. However, we could keep some solutions of the reward functions as a string of numbers, which are equal to maximum reward for this black box. If AI finds this string of numbers, it will reach its maximum utility and stop. The benefit of this stop switch is that AI will not be against it, as it would give it infinite reward. So it would actively cooperate in an attempt to stop it, if it will know that such stop-code exists.

Comment author: dogiv 23 June 2017 02:29:17PM 0 points [-]

I like the first idea. But can we really guarantee that after changing its source code to give itself maximum utility, it will stop all other actions? If it has access to its own source code, what ensures that its utility is "maximum" when it can change the limit arbitrarily? And if all possible actions have the same expected utility, an optimizer could output any solution--"no action" would be the trivial one but it's not the only one.

An AI that has achieved all of its goals might still be dangerous, since it would presumably lose all high-level executive function (its optimization behavior) but have no incentive to turn off any sub-programs that are still running.

Both proposals have the possible failure mode that the AI will discover or guess that this mechanism exists, and then it will only care about making sure it gets activated--which might mean doing bad enough things that humans are forced to open the box and shut it down.

Comment author: Viliam 22 June 2017 09:38:32AM *  0 points [-]

The idea is not to ignore "social games" completely, but rather that some people - specifically, upper-class people - are in a risk of going too far, and seeing the world consisting of "social games" only. Mostly because they are liberated from forces that make lower classes play the "games with nature", such as having to bake your bread or having to keep a job.

Yes, division of labor is a good thing. Problem is, with any division, you need some kind of coordination: whether a person, or an impersonal market. But when you successfully do the revolution, you may kill the competent people and make the market illegal. Then, there may be many people who know how to grow grain and bake bread, but some activities necessary for this process may be made illegal and punished by death. The result is shortage of bread.

The king does not have to know how to make bread, but should not be so insane that he prevents anyone in his kingdom from making bread. And believing e.g. that "objective reality does not exist and everything is socially constructed" seems like a royal road to insanity; but at the same time it is easy to imagine how a person who only ever plays "social games" might find that credible.

Comment author: dogiv 22 June 2017 02:06:42PM 1 point [-]

It seems like the ideal leisure activities, then, should combine the social games with games against nature. Sports do this to some extent, but the "game against nature" part is mostly physical rather than intellectual.

Maybe we could improve on that. I'm envisioning some sort of combination of programming and lacrosse, where the field reconfigures itself according to the players' instructions with a 10-second delay...

But more realistically, certain sports are more strategic and intellectual than others. I've seen both tennis and fencing mentioned as sports that involve quick strategic thinking and predicting your opponent, although they lack the team element that lets you build coordination skills. Maybe some kind of group fencing would be good... or doubles tennis?

View more: Next