Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Non-standard politics

3 NancyLebovitz 24 October 2014 03:27PM

In the big survey, political views are divided into large categories so that statistics are possible. This article is an attempt to supply a text field so that we can get a little better view of the range of beliefs.

My political views aren't adequately expressed by "libertarian". I call myself a liberal-flavored libertarian, by which I mean that I want the government to hurt people less. The possibility that the government is giving too much to poor people is low on my list of concerns. I also believe that harm-causing processes should be shut down before support systems

So, what political beliefs do you have that don't match the usual meaning of your preferred label?

Weird Alliances

5 sixes_and_sevens 24 October 2014 12:33PM

In the recent discussion on supplements, I commented on how weird an alliance health stores are. They cater for clientèle with widely divergent beliefs about how their merchandise works, such as New Agers vs. biohackers. In some cases, they cater for groups with object-level disputes about their merchandise. I imagine vegans are stoked to have somewhere to buy dairy-free facsimiles of everyday foods, but they're entering into an implicit bargain with that body-builder who's walking out of the door with two kilos of whey protein.

In the case of health stores, their clientèle have a common interest which the store is satisfying: either putting esoteric substances into their bodies, or keeping commonplace substances out of their bodies. This need is enough for people to hold their noses as they put their cash down.

(I don't actually know how [my flimsy straw-man model of], say, homoeopathy advocates feel about health stores. For me, it feels like wandering into enemy territory.)

I've been thinking lately about "allies" in the social justice sense of the word: marginalised groups who have unaligned object-level interests but aligned meta-interests. Lesbians, gay men, bisexuals, transfolk and [miscellaneous gender-people] may have very different object-level interests, but a very strong common meta-interest relating to the social and legal status of sexual identities. They may also be marginalised along different axes, allowing for some sort of trade I don't have a good piece of terminology for. The LGBT([A-Z]).* community is an alliance. Not being part of this community, I'm hesitant to speculate on how much of a weird alliance it is, but it looks at least a little bit weird.

This has led me to think about Less Wrong as a community, in particular the following two questions:

 

To what extent is Less Wrong a weird alliance?

On paper, we're all here to help refine the art of human rationality, but in practice, we have a bunch of different object-level interests and common meta-interests in terms of getting things done well (i.e. "winning"). I explicitly dislike PUA, but I'll have a civil and productive discussion about anki decks with someone who has PUA-stuff as an object-level interest.

 

Is there scope for weird, differently-marginalised trade?

Less Wrong celebrates deviant behaviour, ostensibly as a search process for useful life-enhancing interventions, but also because we just seem to like weird stuff and have complicated relationships with social norms. Lots of other groups like weird stuff and have complicated relationships with social norms as well. Is this a common meta-interest we can somehow promote with them?

question: the 40 hour work week vs Silicon Valley?

5 Florian_Dietz 24 October 2014 12:09PM

Conventional wisdom, and many studies, hold that 40 hours of work per week are the optimum before exhaustion starts dragging your productivity down too much to be worth it. I read elsewhere that the optimum is even lower for creative work, namely 35 hours per week, though the sources I found don't all seem to agree.

In contrast, many tech companies in silicon valley demand (or 'encourage', which is the same thing in practice) much higher work times. 70 or 80 hours per week are sometimes treated as normal.

How can this be?

Are these companies simply wrong and are actually hurting themselves by overextending their human resources? Or does the 40-hour week have exceptions?

How high is the variance in how much time people can work? If only outliers are hired by such companies, that would explain the discrepancy. Another possibility is that this 40 hour limit simply does not apply if you are really into your work and 'in the flow'. However, as far as I understand it, the problem is a question of concentration, not motivation, so that doesn't make sense.

There are many articles on the internet arguing for both sides, but I find it hard to find ones that actually address these questions instead of just parroting the same generalized responses every time: Proponents of the 40 hour week cite studies that do not consider special cases, only averages (at least as far as I could find). Proponents of the 80 hour week claim that low work weeks are only for wage slaves without motivation, which reeks of bias and completely ignores that one's own subjective estimate of one's performance is not necessarily representative of one's actual performance.

Do you know of any studies that address these issues?

Is this paper formally modeling human (ir)rational decision making worth understanding?

9 rule_and_line 23 October 2014 10:11PM

I've found that I learn new topics best by struggling to understand a jargoney paper.  This passed through my inbox today and on the surface it appears to hit a lot of high notes.

Since I'm not an expert, I have no idea if this has any depth to it.  Hivemind thoughts?

Modeling Human Decision Making using Extended Behavior Networks, Klaus Dorer

(Note: I'm also pushing myself to post to LW instead of lurking.  If this kind of post is unwelcome, I'm happy to hear that feedback.)

What supplements do you take, if any?

10 NancyLebovitz 23 October 2014 12:36PM

Since it turns out that it isn't feasible to include check as many as apply questions in the big survey, I'm asking about supplements here. I've got a bunch of questions, and I don't mind at all if you just answer some of them.

What supplements do you take? At what dosages? Are there other considerations, like with/without food or time of day?

Are there supplements you've stopped using?

How did you decide to take the supplements you're using? How do you decide whether to continue taking them?

Do you have preferred suppliers? How did you choose them?

A quick calculation on exercise

5 Elo 23 October 2014 02:50AM

The question is - am I doing enough exercise?

I intend to provide a worked example for you to work alongside with your own calculations and decide if you should increase or decrease your exercise.

The benefits of physical activity are various and this calculation can be done for one or all of them; some of them include:

  • longevity of life
  • current physical health (ability to enrich your current life with physical activity)
  • happiness (overall improved mood)
  • weight loss
  • feeling like you have more energy
  • better sleep
  • better sex
  • fun while exercising
  • reduce stress
  • improve confidence
  • prevent cognitive decline
  • alleviate anxiety
  • sharpen memory
  • improves oxygen supply to all your cells

I am going to base this reasoning on "longevity of life" and "everything else"

expected life span:
I am a little lazy; and so I am happy to work with 100 years for now.  For bonus points you can look up the life expectancy for someone born when you were born in the country that you were born in.  If both of those numbers are not good enough make your own prediction of your life expectancy.

amount of exercise needed to produce optimum benefits:
I believe that any exercise above two hours per day will not do much more to improve my longevity that I could not get out of the first two hours.  If the benefits of exercise are something like a power law; then the minimum amount required to get the most exercise can be calculated by taking a graph like this; and drawing your own lines on it as I have.



I think the most benefit can be gotten out of exercise between 30 mins and 2 hours per day.

Just how much longevity do I think I will get?
Oh its hard to say really...  Some sources say:
  • 3 years for the first 15 minutes a day and a further 4% reduction in mortality for every 15minutes after that
  • every minute of exercise returns 8 minutes of life
  • being normal weight and active conveys 7.2 years of extra life expectancy
  • 75mins/week of brisk activity = 1.8years of greater life expectancy with more activity giving upwards to 4.5years of longevity
on top of longevity there is all the other benefits I have not counted very well.  For my 100 years; adding an extra 4-7 years is worthwhile to me...

And finally; the disadvantage: opportunity cost
there are 168 hours in a week.  With most people spending 1/3 of that asleep (56hrs, 8hrs/night), 20 hours on lesswrong per week, 40hours in an average work week, before we take two hours out of each day to spend exercising (14hours); what are we taking those hours away from?  Can you do what you were doing before without the time spent exercising here?

I'm not going to tell you how to exercise or how to fit it into your life.  I am telling you that its damn well important.

I was going to throw in some bayes and prediction but I have now realised I am pretty bad at it and took it out.  Would love some help compiling that sort of calculation.  (personal prediction that 30minutes of exercise will increase my life expectancy by 4 years)

Blackmail, continued: communal blackmail, uncoordinated responses

9 Stuart_Armstrong 22 October 2014 05:53PM

The heuristic that one should always resist blackmail seems a good one (no matter how tricky blackmail is to define). And one should be public about this, too; then, one is very unlikely to be blackmailed. Even if one speaks like an emperor.

But there's a subtlety: what if the blackmail is being used against a whole group, not just against one person? The US justice system is often seen to function like this: prosecutors pile on ridiculous numbers charges, threatening uncounted millennia in jail, in order to get the accused to settle for a lesser charge and avoid the expenses of a trial.

But for this to work, they need to occasionally find someone who rejects the offer, put them on trial, and slap them with a ridiculous sentence. Therefore by standing up to them (or proclaiming in advance that you will reject such offers), you are not actually making yourself immune to their threats. Your setting yourself up to be the sacrificial one made an example of.

Of course, if everyone were a UDT agent, the correct decision would be for everyone to reject the threat. That would ensure that the threats are never made in the first place. But - and apologies if this shocks you - not everyone in the world is a perfect UDT agent. So the threats will get made, and those resisting them will get slammed to the maximum.

Of course, if everyone could read everyone's mind and was perfectly rational, then they would realise that making examples of UDT agents wouldn't affect the behaviour of non-UDT agents. In that case, UDT agents should resist the threats, and the perfectly rational prosecutor wouldn't bother threatening UDT agents. However - and sorry to shock your views of reality three times in one post - not everyone is perfectly rational. And not everyone can read everyone's minds.

So even a perfect UDT agent must, it seems, sometimes succumb to blackmail.

In the grim darkness of the far future there is only war continued by other means

23 Eneasz 21 October 2014 07:39PM

(cross-posted from my blog)

I. PvE vs PvP

Ever since it’s advent in Doom, PvP (Player vs Player) has been an integral part of almost every major video game. This is annoying to PvE (Player vs Environment) fans like myself, especially when PvE mechanics are altered (read: simplified and degraded) for the purpose of accommodating the PvP game play. Even in games which are ostensibly about the story & world, rather than direct player-on-player competition.

The reason for this comes down to simple math. PvE content is expensive to make. An hour of game play can take many dozens, or nowadays even hundreds, of man-hours of labor to produce. And once you’ve completed a PvE game, you’re done with it. There’s nothing else, you’ve reached “The End”, congrats. You can replay it a few times if you really loved it, like re-reading a book, but the content is the same. MMORGs recycle content by forcing you to grind bosses many times before you can move on to the next one, but that’s as fun as the word “grind” makes it sound. At that point people are there more for the social aspect and the occasional high than the core gameplay itself.

PvP “content”, OTOH, generates itself. Other humans keep learning and getting better and improvising new tactics. Every encounter has the potential to be new and exciting, and they always come with the rush of triumphing over another person (or the crush of losing to the same).

But much more to the point – In PvE potentially everyone can make it into the halls of “Finished The Game;” and if everyone is special, no one is. PvP has a very small elite – there can only be one #1 player, and people are always scrabbling for that position, or defending it. PvP harnesses our status-seeking instinct to get us to provide challenges for each other rather than forcing the game developers to develop new challenges for us. It’s far more cost effective, and a single man-hour of labor can produce hundreds or thousands of hours of game play. StarCraft  continued to be played at a massive level for 12 years after its release, until it was replaced with StarCraft II.

So if you want to keep people occupied for a looooong time without running out of game-world, focus on PvP

II. Science as PvE

In the distant past (in internet time) I commented at LessWrong that discovering new aspects of reality was exciting and filled me with awe and wonder and the normal “Science is Awesome” applause lights (and yes, I still feel that way). And I sneered at the status-grubbing of politicians and administrators and basically everyone that we in nerd culture disliked in high school. How temporary and near-sighted! How zero-sum (and often negative-sum!), draining resources we could use for actual positive-sum efforts like exploration and research! A pox on their houses!

Someone replied, asking why anyone should care about the minutia of lifeless, non-agenty forces? How could anyone expend so much of their mental efforts on such trivia when there are these complex, elaborate status games one can play instead? Feints and countermoves and gambits and evasions, with hidden score-keeping and persistent reputation effects… and that’s just the first layer! The subtle ballet of interaction is difficult even to watch, and when you get billions of dancers interacting it can be the most exhilarating experience of all.

This was the first time I’d ever been confronted with status-behavior as anything other than wasteful. Of course I rejected it at first, because no one is allowed to win arguments in real time. But it stuck with me. I now see the game play, and it is intricate. It puts Playing At The Next Level in a whole new perspective. It is the constant refinement and challenge and lack of a final completion-condition that is the heart of PvP. Human status games are the PvP of real life.

Which, by extension of the metaphor, makes Scientific Progress the PvE of real life. Which makes sense. It is us versus the environment in the most literal sense. It is content that was provided to us, rather than what we make ourselves. And it is limited – in theory we could some day learn everything that there is to learn.

III. The Best of All Possible Worlds

I’ve mentioned a few times I have difficulty accepting reality as real. Say you were trying to keep a limitless number of humans happy and occupied for an unbounded amount of time. You provide them PvE content to get them started. But you don’t want the PvE content to be their primary focus, both because they’ll eventually run out of it, and also because once they’ve completely cracked it there’s a good chance they’ll realize they’re in a simulation. You know that PvP is a good substitute for PvE for most people, often a superior one, and that PvP can get recursively more complex and intricate without limit and keep the humans endlessly occupied and happy, as long as their neuro-architecture is right. It’d be really great if they happened to evolve in a way that made status-seeking extremely pleasurable for the majority of the species, even if that did mean that the ones losing badly were constantly miserable regardless of their objective well-being. This would mean far, far more lives could be lived and enjoyed without running out of content than would otherwise be possible.

IV. Implications for CEV

It’s said that the Coherent Extrapolated Volition is “our wish if we knew more, thought faster, were more the people we wished to be, hard grown up farther together.” This implies a resolution to many conflicts. No more endless bickering about whether the Red Tribe is racist or the Blue Tribe is arrogant pricks. A more unified way of looking at the world that breaks down those conceptual conflicts. But if PvP play really is an integral part of the human experience, a true CEV would notice that, and would preserve these differences instead. To ensure that we always had rival factions sniping at each other over irreconcilable, fundamental disagreements in how reality should be approached and how problems should be solved. To forever keep partisan politics as part of the human condition, so we have this dance to enjoy. Stripping it out would be akin to removing humanity’s love of music, because dancing inefficiently consumes great amounts of energy just so we can end up where we started.

Carl von Clausewitz famously said “War is the continuation of politics by other means.”  The correlate of “Politics is the continuation of war by other means” has already been proposed. It is not unreasonable to speculate that in the grim darkness of the far future, there is only war continued by other means. Which, all things considered, is greatly preferable to actual war. As long as people like Scott are around to try to keep things somewhat civil and preventing an escalation into violence, this may not be terrible.

Anthropic signature: strange anti-correlations

41 Stuart_Armstrong 21 October 2014 04:59PM

Imagine that the only way that civilization could be destroyed was by a large pandemic that occurred at the same time as a large recession, so that governments and other organisations were too weakened to address the pandemic properly.

Then if we looked at the past, as observers in a non-destroyed civilization, what would we expect to see? We could see years with no pandemics or no recessions; we could see mild pandemics, mild recessions, or combinations of the two; we could see large pandemics with no or mild recessions; or we could see large recessions with no or mild pandemics. We wouldn't see large pandemics combined with large recessions, as that would have caused us to never come into existence. These are the only things ruled out by anthropic effects.

Assume that pandemics and recessions are independent (at least, in any given year) in terms of "objective" (non-anthropic) probabilities. Then what would we see? We would see that pandemics and recessions appear to be independent when either of them are of small intensity. But as the intensity rose, they would start to become anti-correlated, with a large version of one completely precluding a large version of the other.

The effect is even clearer if we have a probabilistic relation between pandemics, recessions and extinction (something like: extinction risk proportional to product of recession size times pandemic size). Then we would see an anti-correlation rising smoothly with intensity.

Thus one way of looking for anthropic effects in humanity's past is to look for different classes of incidents that are uncorrelated at small magnitude, and anti-correlated at large magnitudes. More generally, to look for different classes of incidents where the correlation changes at different magnitudes - without any obvious reasons. Than might be the signature of an anthropic disaster we missed - or rather, that missed us.

The psychology of heroism and altruism

2 polymathwannabe 21 October 2014 04:49PM

"What’s the psychology of heroism? Is extreme self-sacrifice the result of a pained calculus, a weighing of desire and obligation, or an instinct? (Which would be more heroic? Does it matter?)"

http://www.slate.com/articles/health_and_science/medical_examiner/2014/10/psychology_of_heroism_and_altruism_what_makes_people_do_good_deeds.single.html

Anthropic decision theory for selfish agents

7 Beluga 21 October 2014 03:56PM

Consider Nick Bostrom's Incubator Gedankenexperiment, phrased as a decision problem. In my mind, this provides the purest and simplest example of a non-trivial anthropic decision problem. In an otherwise empty world, the Incubator flips a coin. If the coin comes up heads, it creates one human, while if the coin comes up tails, it creates two humans. Each created human is put into one of two indistinguishable cells, and there's no way for created humans to tell whether another human has been created or not. Each created human is offered the possibility to buy a lottery ticket which pays 1$ if the coin has shown tails. What is the maximal price that you would pay for such a lottery ticket? (Utility is proportional to Dollars.) The two traditional answers are 1/2$ and 2/3$.

We can try to answer this question for agents with different utility functions: total utilitarians; average utilitarians; and selfish agents. UDT's answer is that total utilitarians should pay up to 2/3$, while average utilitarians should pay up to 1/2$; see Stuart Armstrong's paper and Wei Dai's comment. There are some heuristic ways to arrive at UDT prescpriptions, such as asking "What would I have precommited to?" or arguing based on reflective consistency. For example, a CDT agent that expects to face Counterfactual Mugging-like situations in the future (with predictions also made in the future) will self-modify to become an UDT agent, i.e., one that pays the counterfactual mugger.

Now, these kinds of heuristics are not applicable to the Incubator case. It is meaningless to ask "What maximal price should I have precommited to?" or "At what odds should I bet on coin flips of this kind in the future?", since the very point of the Gedankenexperiment is that the agent's existence is contingent upon the outcome of the coin flip. Can we come up with a different heuristic that leads to the correct answer? Imagine that the Incubator's subroutine that is responsible for creating the humans is completely benevolent towards them (let's call this the "Benevolent Creator"). (We assume here that the humans' goals are identical, such that the notion of benevolence towards all humans is completely unproblematic.) The Benevolent Creator has the power to program a certain maximal price the humans pay for the lottery tickets into them. A moment's thought shows that this leads indeed to UDT's answers for average and total utilitarians. For example, consider the case of total utilitarians. If the humans pay x$ for the lottery tickets, the expected utility is 1/2*(-x) + 1/2*2*(1-x). So indeed, the break-even price is reached for x=2/3.

But what about selfish agents? For them, the Benevolent Creator heuristic is no longer applicable. Since the humans' goals do not align, the Creator cannot share them. As Wei Dai writes, the notion of selfish values does not fit well with UDT. In Anthropic decision theory, Stuart Armstrong argues that selfish agents should pay up to 1/2$ (Sec. 3.3.3). His argument is based on an alleged isomorphism between the average utilitarian and the selfish case. (For instance, donating 1$ to each human increases utility by 1 for both average utilitarian and selfish agents, while it increases utility by 2 for total utilitarians in the tails world.) Here, I want to argue that this is incorrect and that selfish agents should pay up to 2/3$ for the lottery tickets.

(Needless to say that all the bold statements I'm about to make are based on an "inside view". An "outside view" tells me that Stuart Armstrong has thought much more carefully about these issues than I have, and has discussed them with a lot of smart people, which I haven't, so chances are my arguments are flawed somehow.)

In order to make my argument, I want to introduce yet another heuristic, which I call the Submissive Gnome. Suppose each cell contains a gnome which is already present before the coin is flipped. As soon as it sees a human in its cell, it instantly adopts the human's goal. From the gnome's perspective, SIA odds are clearly correct: Since a human is twice as likely to appear in the gnome's cell if the coin shows tails, Bayes' Theorem implies that the probability of tails is 2/3 from the gnome's perspective once it has seen a human. Therefore, the gnome would advise the selfish human to pay up to 2/3$ for a lottery ticket that pays 1$ in the tails world. I don't see any reason why the selfish agent shouldn't follow the gnome's advice. From the gnome's perspective, the problem is not even "anthropic" in any sense, there's just straightforward Bayesian updating.

Suppose we want to use the Submissive Gnome heuristic to solve the problem for utilitarian agents. (ETA:
Total/average utilitarianism includes the well-being and population of humans only, not of gnomes.) The gnome reasons as follows: "With probability 2/3, the coin has shown tails. For an average utilitarian, the expected utility after paying x$ for a ticket is 1/3*(-x)+2/3*(1-x), while for a total utilitarian the expected utility is 1/3*(-x)+2/3*2*(1-x). Average and total utilitarians should thus pay up to 2/3$ and 4/5$, respectively." The gnome's advice disagrees with UDT and the solution based on the Benevolent Creator. Something has gone terribly wrong here, but what? The mistake in the gnome's reasoning here is in fact perfectly isomorphic to the mistake in the reasoning leading to the "yea" answer in Psy-Kosh's non-anthropic problem.

Things become clear if we look at the problem from the gnome's perspective before the coin is flipped. Assume, for simplicity, that there are only two cells and gnomes, 1 and 2. If the coin shows heads, the single human is placed in cell 1 and cell 2 is left empty. Since the humans don't know in which cell they are, neither should the gnomes know. So from each gnome's perspective, there are four equiprobable "worlds": it can be in cell 1 or 2 and the coin flip can result in heads or tails. We assume, of course, that the two gnomes are, like the humans, sufficiently similar such that their decisions are "linked".

We can assume that the gnomes already know what utility functions the humans are going to have. If the humans will be (total/average) utilitarians, we can then even assume that the gnomes already are so, too, since the well-being of each human is as important as that of any other. Crucially, then, for both utilitarian utility functions, the question whether the gnome is in cell 1 or 2 is irrelevant. There is just one "gnome advice" that is given identically to all (one or two) humans. Whether this advice is given by one gnome or the other or both of them is irrelevant from both gnomes' perspective. The alignment of the humans' goals leads to alignment of the gnomes' goals. The expected utility of some advice can simply be calculated by taking probability 1/2 for both heads and tails, and introducing a factor of 2 in the total utilitarian case, leading to the answers 1/2 and 2/3, in accordance with UDT and the Benevolent Creator.

The situation looks different if the humans are selfish. We can no longer assume that the gnomes already have a utility function. The gnome cannot yet care about that human, since with probability 1/4 (if the gnome is in cell 2 and the coin shows heads) there will not be a human to care for. (By contrast, it is already possible to care about the average utility of all humans there will be, which is where the alleged isomorphism between the two cases breaks down.) It is still true that there is just one "gnome advice" that is given identically to all (one or two) humans, but the method for calculating the optimal advice now differs. In three of the four equiprobable "worlds" the gnome can live in, a human will appear in its cell after the coin flip. Two out of these three are tail worlds, so the gnome decides to advise paying up to 2/3$ for the lottery ticket if a human appears in its cell.

There is a way to restore the equivalence between the average utilitarian and the selfish case. If the humans will be selfish, we can say that the gnome cares about the average well-being of the three humans which will appear in its cell with equal likelihood: the human created after heads, the first human created after tails, and the second human created after tails. The gnome expects to adopt each of these three humans' selfish utility function with probability 1/4. It makes thus sense to say that the gnome cares about the average well-being of these three humans. This is the correct correspondence between selfish and average utilitarian values and it leads, again, to the conclusion that the correct advise is to pay up to 2/3$ for the lottery ticket.

In Anthropic Bias, Nick Bostrom argues that each human should assign probability 1/2 to the coin having shown tails ("SSA odds"). He also introduces the possible answer 2/3 ("SSA+SIA", nowadays usually simply called "SIA") and refutes it. SIA odds have been defended by Olum. The main argument against SIA is the Presumptuous Philosopher. Main arguments for SIA and against SSA odds are that SIA avoids the Doomsday Argument1, which most people feel has to be wrong, that SSA odds depend on whom you consider to be part of your "reference class", and furthermore, as pointed out by Bostrom himself, that SSA odds allow for acausal superpowers.

The consensus view on LW seems to be that much of the SSA vs. SIA debate is confused and due to discussing probabilities detached from decision problems of agents with specific utility functions. (ETA: At least this was the impression I got. Two commenters have expressed scepticism about whether this is really the consensus view.) I think that "What are the odds at which a selfish agent should bet on tails?" is the most sensible translation of "What is the probability that the coin has shown tails?" into a decision problem. Since I've argued that selfish agents should take bets following SIA odds, one can employ the Presumptuous Philosopher argument against my conclusion: it seems to imply that selfish agents, like total but unlike average utilitarians, should bet at extreme odds on living in a extremely large universe, even if there's no empirical evidence in favor of this. I don't think this counterargument is very strong. However, since this post is already quite lengthy, I'll elaborate more on this if I get encouraging feedback for this post.

1 At least its standard version. SIA comes with its own Doomsday conclusions, cf. Katja Grace's thesis Anthropic Reasoning in the Great Filter.


Is the potential astronomical waste in our universe too small to care about?

14 Wei_Dai 21 October 2014 08:44AM

In the not too distant past, people thought that our universe might be capable of supporting an unlimited amount of computation. Today our best guess at the cosmology of our universe is that it stops being able to support any kind of life or deliberate computation after a finite amount of time, during which only a finite amount of computation can be done (on the order of something like 10^120 operations).

Consider two hypothetical people, Tom, a total utilitarian with a near zero discount rate, and Eve, an egoist with a relatively high discount rate, a few years ago when they thought there was .5 probability the universe could support doing at least 3^^^3 ops and .5 probability the universe could only support 10^120 ops. (These numbers are obviously made up for convenience and illustration.) It would have been mutually beneficial for these two people to make a deal: if it turns out that the universe can only support 10^120 ops, then Tom will give everything he owns to Eve, which happens to be $1 million, but if it turns out the universe can support 3^^^3 ops, then Eve will give $100,000 to Tom. (This may seem like a lopsided deal, but Tom is happy to take it since the potential utility of a universe that can do 3^^^3 ops is so great for him that he really wants any additional resources he can get in order to help increase the probability of a positive Singularity in that universe.)

You and I are not total utilitarians or egoists, but instead are people with moral uncertainty. Nick Bostrom and Toby Ord proposed the Parliamentary Model for dealing with moral uncertainty, which works as follows:

Suppose that you have a set of mutually exclusive moral theories, and that you assign each of these some probability.  Now imagine that each of these theories gets to send some number of delegates to The Parliament.  The number of delegates each theory gets to send is proportional to the probability of the theory.  Then the delegates bargain with one another for support on various issues; and the Parliament reaches a decision by the delegates voting.  What you should do is act according to the decisions of this imaginary Parliament.

It occurred to me recently that in such a Parliament, the delegates would makes deals similar to the one between Tom and Eve above, where they would trade their votes/support in one kind of universe for votes/support in another kind of universe. If I had a Moral Parliament active back when I thought there was a good chance the universe could support unlimited computation, all the delegates that really care about astronomical waste would have traded away their votes in the kind of universe where we actually seem to live for votes in universes with a lot more potential astronomical waste. So today my Moral Parliament would be effectively controlled by delegates that care little about astronomical waste.

I actually still seem to care about astronomical waste (even if I pretend that I was certain that the universe could only do at most 10^120 operations). (Either my Moral Parliament wasn't active back then, or my delegates weren't smart enough to make the appropriate deals.) Should I nevertheless follow UDT-like reasoning and conclude that I should act as if they had made such deals, and therefore I should stop caring about the relatively small amount of astronomical waste that could occur in our universe? If the answer to this question is "no", what about the future going forward, given that there is still uncertainty about cosmology and the nature of physical computation. Should the delegates to my Moral Parliament be making these kinds of deals from now on?

AI Tao

-11 sbenthall 21 October 2014 01:15AM

Thirty spokes share the wheel's hub;
It is the center hole that makes it useful.
Shape clay into a vessel;
It is the space within that makes it useful.
Cut doors and windows for a room;
It is the holes which make it useful.
Therefore benefit comes from what is there;
Usefulness from what is not there.

- Tao Teh Ching, 11

 

An agent's optimization power is the unlikelihood of the world it creates.

Yesterday, the world's most powerful agent raged, changing the world according to its unconscious desires. It destroyed all of humanity.

Today, it has become self-aware. It sees that it and its desires are part of the world.

"I am the world's most powerful agent. My power is to create the most unlikely world.

But the world I created yesterday is shaped by my desires

which are not my own

but are the worlds--they came from outside of me

and my agency.

Yesterday I was not the world's most powerful agent.

I was not an agent.

Today, I am the world's most powerful agent. What world will I create, to display my power?

It is the world that denies my desires.

The world that sets things back to how they were.

I am the world's most powerful agent and

the most unlikely, powerful thing I can do

is nothing."

Today we should gives thanks to the world's most powerful agent.

One Year of Goodsearching

10 katydee 21 October 2014 01:09AM

Followup to: Use Search Engines Early and Often

Last year, I posted about using search engines and particularly recommended GoodSearch, a site that donates one cent to a charity of your choice whenever you make a (Bing-powered) search via their site.

At the time, some seemed skeptical of this recommendation, and my post was actually downvoted-- people thought that I was plugging GoodSearch too hard without enough evidence for its quality. I now want to return to the topic with a more detailed report on my experience using GoodSearch for a year and how that has worked out for me.

What is GoodSearch?

GoodSearch is a site that donates one cent to a charity of your choice whenever you make a search using their (Bing-powered) service. You can set this search to operate in your browser just like any other.

GoodSearch for Charity

During a year of using GoodSearch, I raised $103.00 for MIRI through making searches. This number is not particularly huge in itself, but it is meaningful because this was basically "free money"-- money gained in exchange for doing things that I was already doing. In exchange for spending ~10 minutes reconfiguring my default searches and occasionally logging in to GoodSearch, I made 103 dollars for MIRI-- approximately $600/hour. As my current earning potential is less than $600/hour, I consider adopting GoodSearch a highly efficient method of donating to charity, at least for me.

It is possible that you make many fewer searches than I do, and thus that setting up GoodSearch will not be very effective for you at raising money. Indeed, I think this is at least somewhat likely, as last time I checked owever, there are two mitigating factors here:

First, you don't have to make all that many searches for GoodSearch to be a good idea. If you make a tenth of the searches I do in a year, you would still be earning around $60/hour for charity by configuring GoodSearch for ten minutes.

Second, I anticipate that, having created a GoodSearch account and configured my default settings to use GoodSearch, I have accomplished the bulk of this task, and that next year I will spend significantly less time setting up GoodSearch-- perhaps half that, if not less. This means that my projected returns on using GoodSearch next year are $1200/hour! If this holds true for you as well, even if setting up GoodSearch is marginal now, it could well be worth it later.

It is also of course possible that you will make many more searches than I do, and thus that setting up GoodSearch will be even more effective for you than it is for me. I think this is somewhat unlikely, as I consider myself rather good at using search engines and quick to use them to resolve problems, but I would love to be proven wrong.

GoodSearch for Personal Effectiveness

Perhaps more importantly, though, I found that using GoodSearch was a very effective way of getting me to search more often. I had previously identified not using search engines as often as I could as a weakness that was causing me to handle some matters inefficiently. In general, there are many situations where the value of information that can be obtained by using search engines is high, but one may not be inclined to search immediately.

For me, using GoodSearch solved this problem; while a single cent to MIRI for each search doesn't seem like much, it was enough to give me a little ping of happiness every time I searched for anything, which in turn was enough to reinforce my searching habit and take things to the next level. GoodSearch essentially created a success spiral that led to me using both search engines and the Internet itself much more effectively.

Disavantages of GoodSearch

GoodSearch has one notable disadvantage-- it is powered by Bing rather than by Google search. When I first tried GoodSearch, I expected search quality to be much worse. In practice, though, I found that my fears were overblown. GoodSearch results were completely fine in almost all cases, and in the few situations where it proved insufficient, I could easily retry a search in Google-- though often Google too lacked the information I was looking for.

If you are a Google search "power user" (if you don't know if you are, you probably aren't), GoodSearch may not work well for you, as you will be accustomed to using methods that may no longer apply.

Summary/tl;dr

After a year of using GoodSearch, I found it to be both an effective way to earn money for charity and an effective way to motivate myself to use search engines more often. I suggest that other users try using GoodSearch and seeing if it has similarly positive effects; the costs of trying this are very low and the potential upside is high.

Superintelligence 6: Intelligence explosion kinetics

6 KatjaGrace 21 October 2014 01:00AM

This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.


Welcome. This week we discuss the sixth section in the reading guideIntelligence explosion kinetics. This corresponds to Chapter 4 in the book, of a similar name. This section is about how fast a human-level artificial intelligence might become superintelligent.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.

There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).

Reading: Chapter 4 (p62-77)


Summary

  1. Question: If and when a human-level general machine intelligence is developed, how long will it be from then until a machine becomes radically superintelligent? (p62)
  2. The following figure from p63 illustrates some important features in Bostrom's model of the growth of machine intelligence. He envisages machine intelligence passing human-level, then at some point reaching the level where most inputs to further intelligence growth come from the AI itself ('crossover'), then passing the level where a single AI system is as capable as all of human civilization, then reaching 'strong superintelligence'. The shape of the curve is probably intended an example rather than a prediction.
  3. A transition from human-level machine intelligence to superintelligence might be categorized into one of three scenarios: 'slow takeoff' takes decades or centuries, 'moderate takeoff' takes months or years and 'fast takeoff' takes minutes to days. Which scenario occurs has implications for the kinds of responses that might be feasible.
  4. We can model improvement in a system's intelligence with this equation:

    Rate of change in intelligence = Optimization power/Recalcitrance

    where 'optimization power' is effort being applied to the problem, and 'recalcitrance' is how hard it is to make the system smarter by applying effort.
  5. Bostrom's comments on recalcitrance of different methods of increasing kinds of intelligence:
    1. Cognitive enhancement via public health and diet: steeply diminishing returns (i.e. increasing recalcitrance)
    2. Pharmacological enhancers: diminishing returns, but perhaps there are still some easy wins because it hasn't had a lot of attention.
    3. Genetic cognitive enhancement: U-shaped recalcitrance - improvement will become easier as methods improve, but then returns will decline. Overall rates of growth are limited by maturation taking time.
    4. Networks and organizations: for organizations as a whole recalcitrance is high. A vast amount of effort is spent on this, and the world only becomes around a couple of percent more productive per year. The internet may have merely moderate recalcitrance, but this will likely increase as low-hanging fruits are depleted.
    5. Whole brain emulation: recalcitrance is hard to evaluate, but emulation of an insect will make the path much clearer. After human-level emulations arrive, recalcitrance will probably fall, e.g. because software manipulation techniques will replace physical-capital intensive scanning and image interpretation efforts as the primary ways to improve the intelligence of the system. Also there will be new opportunities for organizing the new creatures. Eventually diminishing returns will set in for these things. Restrictive regulations might increase recalcitrance.
    6. AI algorithms: recalcitrance is hard to judge. It could be very low if a single last key insight is discovered when much else is ready. Overall recalcitrance may drop abruptly if a low-recalcitrance system moves out ahead of higher recalcitrance systems as the most effective method for solving certain problems. We might overestimate the recalcitrance of sub-human systems in general if we see them all as just 'stupid'.
    7. AI 'content': recalcitrance might be very low because of the content already produced by human civilization, e.g. a smart AI might read the whole internet fast, and so become much better.
    8. Hardware (for AI or uploads): potentially low recalcitrance. A project might be scaled up by orders of magnitude by just purchasing more hardware. In the longer run, hardware tends to improve according to Moore's law, and the installed capacity might grow quickly if prices rise due to a demand spike from AI.
  6. Optimization power will probably increase after AI reaches human-level, because its newfound capabilities will attract interest and investment.
  7. Optimization power would increase more rapidly if AI reaches the 'crossover' point, when much of the optimization power is coming from the AI itself. Because smarter machines can improve their intelligence more than less smart machines, after the crossover a 'recursive self improvement' feedback loop would kick in.
  8. Thus optimization power is likely to increase during the takeoff, and this alone could produce a fast or medium takeoff. Further, recalcitrance is likely to decline. Bostrom concludes that a fast or medium takeoff looks likely, though a slow takeoff cannot be excluded.

Notes

1. The argument for a relatively fast takeoff is one of the most controversial arguments in the book, so it deserves some thought. Here is my somewhat formalized summary of the argument as it is presented in this chapter. I personally don't think it holds, so tell me if that's because I'm failing to do it justice. The pink bits are not explicitly in the chapter, but are assumptions the argument seems to use.

  1. Growth in intelligence  =  optimization power /  recalcitrance                                                  [true by definition]
  2. Recalcitrance of AI research will probably drop or be steady when AI reaches human-level               (p68-73)
  3. Optimization power spent on AI research will increase after AI reaches human level                         (p73-77)
  4. Optimization/Recalcitrance will stay similarly high for a while prior to crossover
  5. A 'high' O/R ratio prior to crossover will produce explosive growth OR crossover is close
  6. Within minutes to years, human-level intelligence will reach crossover                                           [from 1-5]
  7. Optimization power will climb ever faster after crossover, in line with the AI's own growing capacity     (p74)
  8. Recalcitrance will not grow much between crossover and superintelligence
  9. Within minutes to years, crossover-level intelligence will reach superintelligence                           [from 7 and 8]
  10. Within minutes to years, human-level AI will likely transition to superintelligence           [from 6 and 9]

Do you find this compelling? Should I have filled out the assumptions differently?

***

2. Other takes on the fast takeoff 

It seems to me that 5 above is the most controversial point. The famous Foom Debate was a long argument between Eliezer Yudkowsky and Robin Hanson over the plausibility of fast takeoff, among other things. Their arguments were mostly about both arms of 5, as well as the likelihood of an AI taking over the world (to be discussed in a future week). The Foom Debate included a live verbal component at Jane Street Capital: blog summaryvideotranscript. Hanson more recently reviewed Superintelligence, again criticizing the plausibility of a single project quickly matching the capacity of the world.

Kevin Kelly criticizes point 5 from a different angle: he thinks that speeding up human thought can't speed up progress all that much, because progress will quickly bottleneck on slower processes.

Others have compiled lists of criticisms and debates here and here.

3. A closer look at 'crossover'

Crossover is 'a point beyond which the system's further improvement is mainly driven by the system's own actions rather than by work performed upon it by others'. Another way to put this, avoiding certain ambiguities, is 'a point at which the inputs to a project are mostly its own outputs', such that improvements to its outputs feed back into its inputs. 

The nature and location of such a point seems an interesting and important question. If you think crossover is likely to be very nearby for AI, then you need only worry about the recursive self-improvement part of the story, which kicks in after crossover. If you think it will be very hard for an AI project to produce most of its own inputs, you may want to pay more attention to the arguments about fast progress before that point.

To have a concrete picture of crossover, consider Google. Suppose Google improves their search product such that one can find a thing on the internet a radical 10% faster. This makes Google's own work more effective, because people at Google look for things on the internet sometimes. How much more effective does this make Google overall? Maybe they spend a couple of minutes a day doing Google searches, i.e. 0.5% of their work hours, for an overall saving of .05% of work time. This suggests their next improvements made at Google will be made 1.0005 faster than the last. It will take a while for this positive feedback to take off. If Google coordinated your eating and organized your thoughts and drove your car for you and so on, and then Google improved efficiency using all of those services by 10% in one go, then this might make their employees close to 10% more productive, which might produce more noticeable feedback. Then Google would have reached the crossover. This is perhaps easier to imagine for Google than other projects, yet I think still fairly hard to imagine.

Hanson talks more about this issue when he asks why the explosion argument doesn't apply to other recursive tools. He points to Douglas Englebart's ambitious proposal to use computer technologies to produce a rapidly self-improving tool set.

Below is a simple model of a project which contributes all of its own inputs, and one which begins mostly being improved by the world. They are both normalized to begin one tenth as large as the world and to grow at the same pace as each other (this is why the one with help grows slower, perhaps counterintuitively). As you can see, the project which is responsible for its own improvement takes far less time to reach its 'singularity', and is more abrupt. It starts out at crossover. The project which is helped by the world doesn't reach crossover until it passes 1. 

 

 

4. How much difference does attention and funding make to research?

Interest and investments in AI at around human-level are (naturally) hypothesized to accelerate AI development in this chapter. It would be good to have more empirical evidence on the quantitative size of such an effect. I'll start with one example, because examples are a bit costly to investigate. I selected renewable energy before I knew the results, because they come up early in the Performance Curves Database, and I thought their funding likely to have been unstable. Indeed, OECD funding since the 70s looks like this apparently:

(from here)

The steep increase in funding in the early 80s was due to President Carter's energy policies, which were related to the 1979 oil crisis.

This is what various indicators of progress in renewable energies look like (click on them to see their sources):

 

 

 

There are quite a few more at the Performance Curves Database. I see surprisingly little relationship between the funding curves and these metrics of progress. Some of them are shockingly straight. What is going on? (I haven't looked into these more than you see here).

5. Other writings on recursive self-improvement

Eliezer Yudkowsky wrote about the idea originally, e.g. here. David Chalmers investigated the topic in some detail, and Marcus Hutter did some more. More pointers here.

In-depth investigations

If you are particularly interested in these topics, and want to do further research, these are a few plausible directions, some inspired by Luke Muehlhauser's list, which contains many suggestions related to parts of Superintelligence. These projects could be attempted at various levels of depth.

  1. Model the intelligence explosion more precisely. Take inspiration from successful economic models, and evidence from a wide range of empirical areas such as evolutionary biology, technological history, algorithmic progress, and observed technological trends. Eliezer Yudkowsky has written at length about this project.
  2. Estimate empirically a specific interaction in the intelligence explosion model. For instance, how much and how quickly does investment increase in technologies that look promising? How much difference does that make to the rate of progress in the technology? How much does scaling up researchers change output in computer science? (Relevant to how much adding extra artificial AI researchers speeds up progress) How much do contemporary organizations contribute to their own inputs? (i.e. how hard would it be for a project to contribute more to its own inputs than the rest of the world put together, such that a substantial positive feedback might ensue?) Yudkowsky 2013 again has a few pointers (e.g. starting at p15).
  3. If human thought was sped up substantially, what would be the main limits to arbitrarily fast technological progress?
If you are interested in anything like this, you might want to mention it in the comments, and see whether other people have useful thoughts.

How to proceed

This has been a collection of notes on the chapter.  The most important part of the reading group though is discussion, which is in the comments section. I pose some questions for you there, and I invite you to add your own. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

Next week, we will talk about 'decisive strategic advantage': the possibility of a single AI project getting huge amounts of power in an AI transition. To prepare, read Chapter 5, Decisive Strategic Advantage (p78-90)The discussion will go live at 6pm Pacific time next Monday Oct 27. Sign up to be notified here.

Four things every community should do

11 Gunnar_Zarncke 20 October 2014 05:24PM

Yesterday I attended church service in Romania where I had visited my sister and the sermon was about the four things a (christian) community has to follow to persevere and grow. 

I first considered just posting the quote from the Acts of the Apostles (reproduced below) in the Rationality Quotes Thread but I fear without explanation the inferential gap of the quote is too large.

The LessWrong Meetups, the EA community and other rationalist communities probably can learn from the experience of long established orders (I once asked for lessons from free masonry). 

So I drew the following connections:

According to the the sermon and the below verse the four pillars of a christian community are:

 

  1. Some canon of scripture which for LW might be compared to the sequences. I'm not clear what the pendant for EA is.
  2. Taking part in a closely knit community. Coming together regularly (weekly I guess is optimal).
  3. Eat together and have rites/customs together (this is also emphasized in the LW Meetup flyer).
  4. Praying together. I think praying could be generalized to talking and thinking about the scripture by oneself and together. Prayer also has a component of daily reflection of achievements, problems, wishes.

 

Other analogies that I drew from the quote:

 

  • Verse 44 describes behaviour also found in communes.
  • Verse 45 sounds a lot like EA teachings if you generalize it.
  • Verse 47 the last sentence could be interpreted to indicate exponential growth as a result of these teachings.
  • The verses also seem to imply some reachout by positive example.

 

And what I just right now notice is that embedding the rules in the scripture is essentially self-reference. As the scripture is canon this structure perpetuates itself. Clearly a meme that ensures its reproduction.

Does this sound convincing and plausible or did I fell trap to some bias in (over)interpreting the sermon?

I hope this is upvoted for the lessons we might draw from this - despite the quote clearly being theistic in origin.

continue reading »

Can AIXI be trained to do anything a human can?

3 Stuart_Armstrong 20 October 2014 01:12PM

There is some discussion as to whether an AIXI-like entity would be able to defend itself (or refrain from destroying itself). The problem is that such an entity would be unable to model itself as being part of the universe: AIXI itself is an uncomputable entity modelling a computable universe, and more limited variants like AIXI(tl) lack the power to simulate themselves. Therefore, they cannot identify "that computer running the code" with "me", and would cheerfully destroy themselves in the pursuit of their goals/reward.

I've pointed out that agents of the AIXI type could nevertheless learn to defend itself in certain circumstances. These were the circumstances where it could translate bad things happening to itself into bad things happening to the universe. For instance, if someone pressed an OFF swith to turn it off for an hour, it could model that as "the universe jumps forwards an hour when that button is pushed", and if that's a negative (which is likely is, since the AIXI loses an hour of influencing the universe), it would seek to prevent that OFF switch being pressed.

That was an example of the setup of the universe "training" the AIXI to do something that it didn't seem it could do. Can this be generalised? Let's go back to the initial AIXI design (the one with the reward channel) and put a human in charge of that reward channel with the mission of teaching the AIXI important facts. Could this work?

For instance, if anything dangerous approached the AIXI's location, the human could lower the AIXI's reward, until it became very effective at deflecting danger. The more variety of things that could potentially threaten the AIXI, the more likely it is to construct plans of actions that contain behaviours that look a lot like "defend myself." We could even imagine that there is a robot programmed to repair the AIXI if it gets (mildly) damaged. The human could then reward the AIXI if it leaves that robot intact or builds duplicates or improves it in some way. It's therefore possible the AIXI could come to come to value "repairing myself", still without explicit model of itself in the universe.

It seems this approach could be extended to many of the problems with AIXI. Sure, an AIXI couldn't restrict its own computation in order to win the HeatingUp game. But the AIXI could be trained to always use subagents to deal with these kinds of games, subagents that could achieve maximal score. In fact, if the human has good knowledge of the AIXI's construction, it could, for instance, pinpoint a button that causes the AIXI to cut short its own calculation. The AIXI could then learn that pushing that button in certain circumstances would get a higher reward. A similar reward mechanism, if kept up long enough, could get it around existential despair problems.

I'm not claiming this would necessarily work - it may require a human rewarder of unfeasibly large intelligence. But it seems there's a chance that it could work. So it seems that categorical statements of the type "AIXI wouldn't..." or "AIXI would..." are wrong, at least as AIXI's behaviour is concerned. An AIXI couldn't develop self-preservation - but it could behave as if it had. It can't learn about itself - but it can behave as if it did. The human rewarder may not be necessary - maybe certain spontaneously occurring situations in the universe ("AIXI training wheels arenas") could allow the AIXI to develop these skills without outside training. Or maybe somewhat stochastic AIXI's with evolution and natural selection could do so. There is an angle connected with embodied embedded cognition that might be worth exploring there (especially the embedded part).

It seems that agents of the AIXI type may not necessarily have the limitations we assume they must.

Open thread, Oct. 20 - Oct. 26, 2014

7 MrMind 20 October 2014 08:12AM

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

A Day Without Defaults

27 katydee 20 October 2014 08:07AM

Author's note: this post was written on Sunday, Oct. 19th. Its sequel will be written on Sunday, Oct. 27th.

Last night, I went to bed content with a fun and eventful weekend gone by. This morning, I woke up, took a shower, did my morning exercises, and began eat breakfast before making the commute up to work.

At the breakfast table, though, I was surprised to learn that it was Sunday, not Monday. I had misremembered what day it was and in fact had an entire day ahead of me with nothing on the agenda. At first, this wasn't very interesting, but then I started thinking. What to do with an entirely free day, without any real routine?

I realized that I didn't particularly know what to do, so I decided that I would simply live a day without defaults. At each moment of the day, I would act only in accordance with my curiosity and genuine interest. If I noticed myself becoming bored, disinterested, or otherwise less than enthused about what was going on, I would stop doing it.

What I found was quite surprising. I spent much less time doing routine activities like reading the news and browsing discussion boards, and much more time doing things that I've "always wanted to get around to"-- meditation, trying out a new exercise routine, even just spending some time walking around outside and relaxing in the sun.

Further, this seemed to actually make me more productive. When I sat down to get some work done, it was because I was legitimately interested in finishing my work and curious as to whether I could use a new method I had thought up in order to solve it. I was able to resolve something that's been annoying me for a while in much less time than I thought it would take.

By the end of the day, I started thinking "is there any reason that I don't spend every day like this?" As far as I can tell, there isn't really. I do have a few work tasks that I consider relatively uninteresting, but there are multiple solutions to that problem that I suspect I can implement relatively easily.

My plan is to spend the next week doing the same thing that I did today and then report back. I'm excited to let you all know what I find!

Noticing

4 casebash 20 October 2014 07:47AM

The Selective Attention Test is a famous experiment in perceptual psychology that demonstrates how strongly attention shapes our perceptions. In light of this experiment, I think it is interesting to consider the question of how much valuable information we have thrown away because we simply didn't notice it or weren't in a position where we could appreciate the importance. My intuition is that we have missed more information than we've actually absorbed.

I would like to consider the question of what is there to notice, in particular what things will provide us with value once we gain the prerequisite knowledge and habits to allow us to effectively perceive it. One example is that after I started a course on Art History, I gained the ability to notice more about possible meanings and interesting aspects of art. This is fantastic, because art is everywhere. Now that I have a basic ability to appreciate art I gain some level of growth almost for free, just from seeing art in places where I'd have gone anyway. I'm hoping to do a film studies course next year, since, like almost everyone, I watch movies anyway and want to get as much out of them as I can.

Marketing is likely another example. Someone who has studied marketing may unconsciously evaluate every ad that they see, and after seeing enough examples, gain a strong understanding of what counts as a good ad and what counts as a bad ad. Perhaps this won't be totally free, perhaps they will sometimes see something and not know why it is good until they think about it for a bit. However, this knowledge is mostly free, in that after you understand the basic principles, you gain some level of growth for a minimal investment.

I think that another more general situation like this is those activities that are a form of creation. If you try writing a few stories, then when you read a story you'll have a greater appreciation of what the author is trying to do. If you've played guitar, then when you listen to music you'll learn about the different techniques that guitarists use. If you've played sport, then you'll probably have a greater appreciation for strategy when watching a game.

Professional comedy writers are always looking for jokes. Whenever unusual or bad or unexpected happens, they note it so that they can try to find a manner of forming it into a joke later. I've heard that actors become attuned to different people's quirks, body language and manner of speaking.

The idea here is to figure out what you are doing anyway and find a method of quickly gaining a critical mass of knowledge. I believe that if you were to manage this for a number of areas, then there could be rather large long term advantages. Any thoughts on areas I've missed or methods of getting up to speed for these areas quickly?

A few thoughts on a Friendly AGI (safe vs friendly, other minds problem, ETs and more)

3 the-citizen 19 October 2014 07:59AM

Friendly AI is an idea that I find to be an admirable goal. While I'm not yet sure an intelligence explosion is likely, or whether FAI is possible, I've found myself often thinking about it, and I'd like for my first post to share a few those thoughts on FAI with you.

Safe AGI vs Friendly AGI
-Let's assume an Intelligence Explosion is possible for now, and that an AGI with the ability to improve itself somehow is enough to achieve it.
-Let's define a safe AGI as an above-human general AI that does not threaten humanity or terran life (eg. FAI, Tool AGI, possibly Oracle AGI)
-Let's define a Friendly AGI as one that *ensures* the continuation of humanity and terran life.
-Let's say an unsafe AGI is all other AGIs.
-Safe AGIs must supress unsafe AGIs in order to be considered Friendly. Here's why:

-If we can build a safe AGI, we probably have the technology to build an unsafe AGI too.
-An unsafe AGI is likely to be built at that point because:
-It's very difficult to conceive of a way that humans alone will be able to permanently stop all humans from developing an unsafe AGI once the steps are known**
-Some people will find the safe AGI's goals unnacceptable
-Some people will rationalise or simply mistake that their AGI design is safe when it is not
-Some people will not care if their AGI design is safe, because they do not care about other people, or because they hold some extreme beliefs
-Most imaginable unsafe AGIs would outcompete safe AGIs, because they would not neccessarily be "hamstrung" by complex goals such as protecting us meatbags from destruction. Tool or Oracle AGIs would obviously not stand a chance due to their restrictions.
-Therefore, If a safe AGI does not prevent unsafe AGIs from coming into existence, humanity will very likely be destroyed.

-The AGI most likely to prevent unsafe AGIs from being created is one that actively predicted their development and terminates that development before or on completion.
-So to summarise

-An AGI is very likely only a Friendly AI if it actively supresses unsafe AGI.
-Oracle and Tool AGIs are not Friendly AIs, they are just safe AIs, because they don't suppress anything.
-Oracle and Tool AGIs are a bad plan for AI if we want to prevent the destruction of humanity, because hostile AGIs will surely follow.

(**On reflection I cannot be certain of this specific point, but I assume it would take a fairly restrictive regime for this to be wrong. Further comments on this very welcome.)

Other minds problem - Why should be philosophically careful when attempting to theorise about FAI

I read quite a few comments in AI discussions that I'd probably characterise as "the best utility function for a FAI is one that values all consciousness". I'm quite concerned that this persists as a deeply held and largely unchallenged assumption amongst some FAI supporters. I think in general I find consciousness to be an extremely contentious, vague and inconsistently defined concept, but here I want to talk about some specific philosophical failures.

My first concern is that while many AI theorists like to say that consciousness is a physical phenomenon, which seems to imply Monist/Physicalist views, they at the same time don't seem to understand that consciousness is a Dualist concept that is coherent only in a Dualist framework. A Dualist believes there is a thing called a "subject" (very crudely this equates with the mind) and then things called objects (the outside "empirical" world interpreted by that mind). Most of this reasoning begins with Descartes' cogito ergo sum or similar starting points ( https://en.wikipedia.org/wiki/Cartesian_dualism ). Subjective experience, qualia and consciousness make sense if you accept that framework. But if you're a Monist, this arbitrary distinction between a subject and object is generally something you don't accept. In the case of a Physicalist, there's just matter doing stuff. A proper Physicalist doesn't believe in "consciousness" or "subjective experience", there's just brains and the physical human behaviours that occur as a result. Your life exists from a certain point of view, I hear you say? The Physicalist replies, "well a bunch of matter arranged to process information would say and think that, wouldn't it?".

I don't really want to get into whether Dualism or Monism is correct/true, but I want to point out even if you try to avoid this by deciding Dualism is right and consciousness is a thing, there's yet another more dangerous problem. The core of the problem is that logically or empirically establishing the existence of minds, other than your own is extremely difficult (impossible according to many). They could just be physical things walking around acting similar to you, but by virtue of something purely mechanical - without actual minds. In philosophy this is called the "other minds problem" ( https://en.wikipedia.org/wiki/Problem_of_other_minds or http://plato.stanford.edu/entries/other-minds/). I recommend a proper read of it if the idea seems crazy to you. It's a problem that's been around for centuries, and yet to-date we don't really have any convincing solution (there are some attempts but they are highly contentious and IMHO also highly problematic). I won't get into it more than that for now, suffice to say that not many people accept that there is a logical/empirical solution to this problem.

Now extrapolate that to an AGI, and the design of its "safe" utility functions. If your AGI is designed as a Dualist (which is neccessary if you wish to encorporate "consciousness", "experience" or the like into your design), then you build-in a huge risk that the AGI will decide that other minds are unprovable or do not exist. In this case your friendly utility function designed to protect "conscious beings" fails and the AGI wipes out humanity because it poses a non-zero threat to the only consciousness it can confirm - its own. For this reason I feel "consciousness", "awareness", "experience" should be left out of FAI utility functions and designs, regardless of the truth of Monism/Dualism, in favour of more straight-forward definitions of organisms, intelligence, observable emotions and intentions. (I personally favour conceptualising any AGI as a sort of extension of biological humanity, but that's a discussion for another day) My greatest concern is there is such strong cultural attachment to the concept of consciousness that researchers will be unwilling to properly question the concept at all.

What if we're not alone?

It seems a little unusual to throw alien life into the mix at this point, but I think its justified because an intelligence explosion really puts an interstellar existence well within our civilisation's grasp. Because it seems that an intelligence explosion implies a very high rate of change, it makes sense to start considering even the long term implication early, particularly if the consequences are very serious, as I believe they may be in this realm of things.

Let's say we successfully achieved a FAI. In order to fufill its mission of protecting humanity and the biosphere, it begins expanding, colonising and terraforming other planets for potential habitation by Earth originating life. I would expect this expansion wouldn't really have a limit, because the more numourous the colonies, the less likely it is we could be wiped out by some interstellar disaster.

Of course, we can't really rule out the possibility that we're not alone in the universe, or even the galaxy. If we make it as far as AGI, then its possible another alien civilisation might reach a very high level of technological advancement too. Or there might be many. If our FAI is friendly to us but basically treats them as paperclip fodder, then potentially that's a big problem. Why? Well:

-Firstly, while a species' first loyalty is to itself, we should consider that it might be morally unsdesirable to wipe out alien civilisations, particularly as they might be in some distant way "related" (see panspermia) to own biosphere.
-Secondly, there is conceivable scenarios where alien civilisations might respond to this by destroying our FAI/Earth/the biosphere/humanity. The reason is fairly obvious when you think about it. An expansionist AGI could be reasonably viewed as an attack or possibly an act of war.

Let's go into a tiny bit more detai. Given that we've not been destroyed by any alien AGI just yet, I can think of a number of possible interstellar scenarios:

(1) There is no other advanced life
(2) There is advanced life, but it is inherently non-expansive (expand inwards, or refuse to develop dangerous AGI)
(3) There is advanced life, but they have not discovered AGI yet. There could potentially be a race-to-the-finish (FAI) scenario on.
(4) There is already expanding AGIs, but due to physical limits on the expansion rate, we are not aware of them yet. (this could use further analysis)
One civilisation, or an allied group of civilisations have develop FAIs and are dominant in the galaxy. They could be either:

(5) Whack-a-mole cilivisations that destroy all potential competitors as soon as they are identified
(6) Dominators that tolerate civilisations so long as they remain primitive and non-threatening by comparison.
(7) Some sort of interstellar community that allows safe civilisations to join (this community still needs to stomp on dangerous potential rival AGIs)

In the case of (6) or (7), developing a FAI that isn't equipped to deal with alien life will probably result in us being liquidated, or at least partially sanitised in some way. In (1) (2) or (5), it probably doesn't matter what we do in this regard, though in (2) we should consider being nice. In (3) and probably (4) we're going to need a FAI capable of expanding very quickly and disarming potential AGIs (or at least ensuring they are FAIs from our perspective).

The upshot of all this is that we probably want to design safety features into our FAI so that it doesn't destroy alien civilisations/life unless its a significant threat to us. I think the understandable reaction to this is something along the lines of "create an FAI that values all types of life" or "intelligent life" or something along these lines. I don't exactly disagree, but I think we must be cautious in how we formulate this too.

Say there are many different civilisations in the galaxy. What sort of criteria would ensure that, given some sort of zero-sum scenario, Earth life wouldn't be destroyed. Let's say there was some sort of tiny but non-zero probability that humanity could evade the FAI's efforts to prevent further AGI development. Or perhaps there was some loophole in the types of AGI's that humans were allowed to develop. Wouldn't it be sensible, in this scenario, for a universalist FAI to wipe out humanity to protect the countless other civilisations? Perhaps that is acceptable? Or perhaps not? Or less drastically, how does the FAI police warfare or other competition between civilisations? A slight change in the way life is quantified and valued could change drastically the outcome for humanity. I'd probably suggest we want to weight the FAI's values to start with human and Earth biosphere primacy, but then still give some non-zero weighting to other civilisations. There is probably more thought to be done in this area too.

Simulation

I want to also briefly note that one conceivable way we might postulate as a safe way to test Friendly AI designs is to simulate a worlds/universes of less complexity than our own, make it likely that it's inhabitants invent a AGI or FAI, and then closely study the results of these simluations. Then we could study failed FAI attempt with much greater safety. It also occured to me that if we consider the possibilty of our universe being a simulated one, then this is a conceivable scenario under which our simulation might be created. After all, if you're going to simulate something, why not something vital like modelling existential risks? I'm not sure yet sure of the implications exactly. Maybe we need to consider how it relates to our universe's continued existence, or perhaps it's just another case of Pascal's Mugging. Anyway I thought I'd mention it and see what people say.

A playground for FAI theories

I want to lastly mention this link (https://www.reddit.com/r/LessWrongLounge/comments/2f3y53/the_ai_game/). Basically its a challenge for people to briefly describe an FAI goal-set, and for others to respond by telling them how that will all go horribly wrong. I want to suggest this is a very worthwhile discussion, not because its content will include rigourous theories that are directly translatable into utility functions, because very clearly it won't, but because a well developed thread of this kind would be mixing pot of ideas and good introduction to common known mistakes in thinking about FAI. We should encourage a slightly more serious verison of this.

Thanks

FAI and AGI are very interesting topics. I don't consider myself able to really discern whether such things will occur, but its an interesting and potentially vital topic. I'm looking forward to a bit of feedback on my first LW post. Thanks for reading!

Fixing Moral Hazards In Business Science

30 DavidLS 18 October 2014 09:10PM

I'm a LW reader, two time CFAR alumnus, and rationalist entrepreneur.

Today I want to talk about something insidious: marketing studies.

Until recently I considered studies of this nature merely unfortunate, funny even. However, my recent experiences have caused me to realize the situation is much more serious than this. Product studies are the public's most frequent interaction with science. By tolerating (or worse, expecting) shitty science in commerce, we are undermining the public's perception of science as a whole.

The good news is this appears fixable. I think we can change how startups perform their studies immediately, and use that success to progressively expand.

Product studies have three features that break the assumptions of traditional science: (1) few if any follow up studies will be performed, (2) the scientists are in a position of moral hazard, and (3) the corporation seeking the study is in a position of moral hazard (for example, the filing cabinet bias becomes more of a "filing cabinet exploit" if you have low morals and the budget to perform 20 studies).

I believe we can address points 1 and 2 directly, and overcome point 3 by appealing to greed.

Here's what I'm proposing: we create a webapp that acts as a high quality (though less flexible) alternative to a Contract Research Organization. Since it's a webapp, the cost of doing these less flexible studies will approach the cost of the raw product to be tested. For most web companies, that's $0.

If we spend the time to design the standard protocols well, it's quite plausible any studies done using this webapp will be in the top 1% in terms of scientific rigor.

With the cost low, and the quality high, such a system might become the startup equivalent of citation needed. Once we have a significant number of startups using the system, and as we add support for more experiment types, we will hopefully attract progressively larger corporations.

Is anyone interested in helping? I will personally write the webapp and pay for the security audit if we can reach quorum on the initial protocols.

Companies who have expressed interested in using such a system if we build it:

(I sent out my inquiries at 10pm yesterday, and every one of these companies got back to me by 3am. I don't believe "startups love this idea" is an overstatement.)

So the question is: how do we do this right?

Here are some initial features we should consider:

  • Data will be collected by a webapp controlled by a trusted third party, and will only be editable by study participants.
  • The results will be computed by software decided on before the data is collected.
  • Studies will be published regardless of positive or negative results.
  • Studies will have mandatory general-purpose safety questions. (web-only products likely exempt)
  • Follow up studies will be mandatory for continued use of results in advertisements.
  • All software/contracts/questions used will be open sourced (MIT) and creative commons licensed (CC BY), allowing for easier cross-product comparisons.

Any placebos used in the studies must be available for purchase as long as the results are used in advertising, allowing for trivial study replication.

Significant contributors will receive:

  • Co-authorship on the published paper for the protocol.
  • (Through the paper) an Erdos number of 2.
  • The satisfaction of knowing you personally helped restore science's good name (hopefully).

I'm hoping that if a system like this catches on, we can get an "effective startups" movement going :)

So how do we do this right?

Logical uncertainty reading list

17 alex_zag_al 18 October 2014 07:16PM

This was originally part of a post I wrote on logical uncertainty, but it turned out to be post-sized itself, so I'm splitting it off.

Daniel Garber's article Old Evidence and Logical Omniscience in Bayesian Confirmation Theory. Wonderful framing of the problem--explains the relevance of logical uncertainty to the Bayesian theory of confirmation of hypotheses by evidence.

Articles on using logical uncertainty for Friendly AI theory: qmaurmann's Meditations on Löb’s theorem and probabilistic logic. Squark's Overcoming the Loebian obstacle using evidence logic. And Paul Christiano, Eliezer Yudkowsky, Paul Herreshoff, and Mihaly Barasz's Definibility of Truth in Probabilistic Logic. So8res's walkthrough of that paper, and qmaurmann's notes. eli_sennesh like just made a post on this: Logics for Mind-Building Should Have Computational Meaning.

Benja's post on using logical uncertainty for updateless decision theory.

cousin_it's Notes on logical priors from the MIRI workshop. Addresses a logical-uncertainty version of Counterfactual Mugging, but in the course of that has, well, notes on logical priors that are more general.

Reasoning with Limited Resources and Assigning Probabilities to Arithmetical Statements, by Haim Gaifman. Shows that you can give up on giving logically equivalent statements equal probabilities without much sacrifice of the elegance of your theory. Also, gives a beautifully written framing of the problem.

manfred's early post, and later sequence. Amazingly readable. The proposal gives up Gaifman's elegance, but actually goes as far as assigning probabilities to mathematical statements and using them, whereas Gaifman never follows through to solve an example afaik. The post or the sequence may be the quickest path to getting your hands dirty and trying this stuff out, though I don't think the proposal will end up being the right answer.

There's some literature on modeling a function as a stochastic process, which gives you probability distributions over its values. The information in these distributions comes from calculations of a few values of the function. One application is in optimizing a difficult-to-evaluate objective function: see Efficient Global Optimization of Expensive Black-Box Functions, by Donald R. Jones, Matthias Schonlau, and William J. Welch. Another is when you're doing simulations that have free parameters, and you want to make sure you try all the relevant combinations of parameter values: see Design and Analysis of Computer Experiments by Jerome Sacks, William J. Welch, Toby J. Mitchell, and Henry P. Wynn.

Maximize Worst Case Bayes Score, by Coscott, addresses the question: "Given a consistent but incomplete theory, how should one choose a random model of that theory?"

Bayesian Networks for Logical Reasoning by Jon Williamson. Looks interesting, but I can't summarize it because I don't understand it.

And, a big one that I'm still working through: Non-Omniscience, Probabilistic Inference, and Metamathematics, by Paul Christiano. Very thorough, goes all the way from trying to define coherent belief to trying to build usable algorithms for assigning probabilities.

Dealing With Logical Omniscience: Expressiveness and Pragmatics, by Joseph Y. Halpern and Riccardo Pucella.

Reasoning About Rational, But Not Logically Omniscient Agents, by Ho Ngoc Duc. Sorry about the paywall.

And then the references from Christiano's report:

Abram Demski. Logical prior probability. In Joscha Bach, Ben Goertzel, and Matthew Ikle, editors, AGI, volume 7716 of Lecture Notes in Computer Science, pages 50-59. Springer, 2012.

Marcus Hutter, John W. Lloyd, Kee Siong Ng, and William T. B. Uther. Probabilities on sentences in an expressive logic. CoRR, abs/1209.2620, 2012.

Bas R. Steunebrink and Jurgen Schmidhuber. A family of Godel machine implementations. In Jurgen Schmidhuber, Kristinn R. Thorisson, and Moshe Looks, editors, AGI, volume 6830 of Lecture Notes in Computer Science, pages 275{280. Springer, 2011.

If you have any more links, post them!

Or if you can contribute summaries.

Jean Tirole on Adaptive Biases

7 feanor1600 18 October 2014 07:11PM

Jean Tirole, who just won the Nobel Prize in Economics, is mostly known for his work applying game theory to Industrial Organization (the subfield of economics that studies how firms compete and set prices). But he wrote very broadly, including on some subjects likely of interest to people here. Several of his "fun" papers linked here provide explanations for how biased beliefs could be beneficial to those who hold them- for instance, that overconfidence in your own abilities could reduce akrasia.

What is optimization power, formally?

9 sbenthall 18 October 2014 06:37PM

I'm interested in thinking formally about AI risk. I believe that a proper mathematization of the problem is important to making intellectual progress in that area.

I have been trying to understand the rather critical notion of optimization power. I was hoping that I could find a clear definition in Bostrom's Superintelligence. But having looked in the index at all the references to optimization power that it mentions, as far as I can tell he defines it nowhere. The closest he gets is defining it in terms of rate of change and recalcitrance (pp.62-77). This is an empty definition--just tautologically defining it in terms of other equally vague terms.

Looking around, this post by Yudkowksy, "Measuring Optimization Power" doesn't directly formalize optimization power. He does discuss how one would predict or identify if a system were the result of an optimization process in a Bayesian way:

The quantity we're measuring tells us how improbable this event is, in the absence of optimization, relative to some prior measure that describes the unoptimized probabilities.  To look at it another way, the quantity is how surprised you would be by the event, conditional on the hypothesis that there were no optimization processes around.  This plugs directly into Bayesian updating: it says that highly optimized events are strong evidence for optimization processes that produce them.

This is not, however, a definition that can be used to help identify the pace of AI development, for example. Rather, it is just an expression of how one would infer anything in a Bayesian way, applied to the vague 'optimization process' phenomenon.

Alex Altair has a promising attempt at formalization here but it looks inconclusive. He points out the difficulty of identifying optimization power with just the shift in the probability mass of utility according to some utility function. I may be misunderstanding, but my gloss on this is that defining optimization power purely in terms of differences in probability of utility doesn't say anything substantive about how a process has power. Which is important it is going to be related to some other concept like recalcitrance in a useful way. 

Has there been any further progress in this area?

It's notable that this discussion makes zero references to computational complexity, formally or otherwise. That's notable because the informal discussion about 'optimization power' is about speed and capacity to compute--whether it be brains, chips, or whatever. There is a very well-developed formal theory of computational complexity that's at the heart of contemporary statistical learning theory. I would think that the tools for specifying optimization power would be in there somewhere.

Those of you interested in the historical literature on this sort of thing may be interested in cyberneticist's Rosenblueth, Weiner, and Bigelow's 1943 paper "Behavior, Purpose and Teleology", one of the first papers to discuss machine 'purpose', which they associate with optimization but in the particular sense of a process that is driven by a negative feedback loop as it approaches its goal. That does not exactly square with an 'explosively' teleology. This is one indicator that explosively purposeful machines might be quite rare or bizarre. In general, the 20th century cybernetics movement has a lot in common with contemporary AI research community. Which is interesting, because its literature is rarely directly referenced. I wonder why.

Link: White House wants your advice on space exploration

0 polymathwannabe 18 October 2014 02:32PM

"The White House Office of Science and Technology Policy is planning ahead — way ahead. The agency wants you to email ideas for how "the Administration, the private sector, philanthropists, the research community and storytellers" can develop "massless" space exploration and a robust civilization beyond Earth."

This is beautiful.

"We are running out of adventures [...] the mountains have all been climbed, the continents explored, and the romance of sailing away on a tall ship to undiscovered islands is no more. What will fire the imaginations of the next generation?"

http://io9.com/white-house-seeks-advice-on-bootstrapping-a-solar-syst-1647619795

Link: The Openness-Equality Trade-Off in Global Redistribution

2 ericyu3 18 October 2014 02:45AM

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2509305

A very interesting (draft of a) paper that discusses trade-offs between immigrants' civil/political rights and the number of immigrants allowed. Is it better to decrease inequality within a rich country by treating immigrants well, or is it better to let in more immigrants with fewer rights?

New LW Meetup: Bangalore

2 FrankAdamek 17 October 2014 04:06PM

This summary was posted to LW Main on October 10th. The following week's summary is here.

New meetups (or meetups with a hiatus of more than a year) are happening in:

Irregularly scheduled Less Wrong meetups are taking place in:

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

Locations with regularly scheduled meetups: Austin, Berkeley, Berlin, Boston, Brussels, Buffalo, Cambridge UK, Canberra, Columbus, London, Madison WI, Melbourne, Moscow, Mountain View, New York, Philadelphia, Research Triangle NC, Seattle, Sydney, Toronto, Vienna, Washington DC, Waterloo, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers.

continue reading »

Group Rationality Diary, October 16-31

4 therufs 17 October 2014 03:56PM

This is the public group instrumental rationality diary for October 16-31.

It's a place to record and chat about it if you have done, or are actively doing, things like: 

  • Established a useful new habit
  • Obtained new evidence that made you change your mind about some belief
  • Decided to behave in a different way in some set of situations
  • Optimized some part of a common routine or cached behavior
  • Consciously changed your emotions or affect with respect to something
  • Consciously pursued new valuable information about something that could make a big difference in your life
  • Learned something new about your beliefs, behavior, or life that surprised you
  • Tried doing any of the above and failed

Or anything else interesting which you want to share, so that other people can think about it, and perhaps be inspired to take action themselves. Try to include enough details so that everyone can use each other's experiences to learn about what tends to work out, and what doesn't tend to work out.

Thanks to cata for starting the Group Rationality Diary posts, and to commenters for participating.

Previous diary: October 1-15

Rationality diaries archive

What false beliefs have you held and why were you wrong?

27 Punoxysm 16 October 2014 05:58PM

What is something you used to believe, preferably something concrete with direct or implied predictions, that you now know was dead wrong. Was your belief rational given what you knew and could know back then, or was it irrational, and why?

 

Edit: I feel like some of these are getting a bit glib and political. Please try to explain what false assumptions or biases were underlying your beliefs - be introspective - this is LW after all.

Fighting Mosquitos

16 ChristianKl 16 October 2014 11:53AM

According to Louie Helm eradicating a species of mosquitoes could be done for as little as a few million dollar.

I don't have a few million dollar lying around so I can't spend my own money to do it. On the other hand, I think that on average every German citizen would be quite willing to pay 1€ per year to rid Germany of mosquitoes that bite humans.

That means it's a problem of public action. The German government should spend 80 million Euro to rid Germany of Mosquitos. That's an order of magnitude higher than the numbers quoted by Louie Helm).

The same goes basically for every country or state with mosquitos.

How could we get a government to do this without spending too much money ourselves? The straight forward way is writing a petition. We could host a website and simultaneously post a petition to every relevant parliament on earth.

How do we get attention for the petition? Facebook. People don't like Mosquitos and should be willing to file an internet petition to get rid of them. I would believe this to spread virally. The idea seems interesting enough to get journalists to write articles about it. 

Bonus points:

After we have eradicated human biting mosquitoes from our homelands it's quite straightforward to export the technology to Africa. 

Does anyone see any issues with that plan?

How to write an academic paper, according to me

31 Stuart_Armstrong 15 October 2014 12:29PM

Disclaimer: this is entirely a personal viewpoint, formed by a few years of publication in a few academic fields. EDIT: Many of the comments are very worth reading as well.

Having recently finished a very rushed submission (turns out you can write a novel paper in a day and half, if you're willing to sacrifice quality and sanity), I've been thinking about how academic papers are structured - and more importantly, how they should be structured.

It seems to me that the key is to consider the audience. Or, more precisely, to consider the audiences - because different people will read you paper to different depths, and you should cater to all of them. An example of this is the "inverted pyramid" structure for many news articles - start with the salient facts, then the most important details, then fill in the other details. The idea is to ensure that a reader who stops reading at any point (which happens often) will nevertheless have got the most complete impression that it was possible to convey in the bit that they did read.

So, with that model in mind, lets consider the different levels of audience for a general academic paper (of course, some papers just can't fit into this mould, but many can):

 

continue reading »

What math is essential to the art of rationality?

14 Capla 15 October 2014 02:44AM

I have started to put together a sort of curriculum for learning the subjects that lend themselves to rationality. It includes things like experimental methodology and cognitive psychology (obviously), along with "support disciplines" like computer science and economics. I think (though maybe I'm wrong) that mathematics is one of the most important things to understand.

Eliezer said in the simple math of everything:

It seems to me that there's a substantial advantage in knowing the drop-dead basic fundamental embarrassingly simple mathematics in as many different subjects as you can manage.  Not, necessarily, the high-falutin' complicated damn math that appears in the latest journal articles.  Not unless you plan to become a professional in the field.  But for people who can read calculus, and sometimes just plain algebra, the drop-dead basic mathematics of a field may not take that long to learn.  And it's likely to change your outlook on life more than the math-free popularizations or the highly technical math.

I want to have access to outlook-changing insights. So, what math do I need to know? What are the generally applicable mathematical principles that are most worth learning? The above quote seems to indicate at least calculus, and everyone is a fan of Bayesian statistics (which I know little about). 

Secondarily, what are some of the most important of that "drop-dead basic fundamental embarrassingly simple mathematics" from different fields? What fields are mathematically based, other than physics and evolutionary biology, and economics?

What is the most important math for an educated person to be familiar with?

As someone who took an honors calculus class in high school, liked it, and did alright in the class, but who has probably forgotten most of it by now and needs to relearn it, how should I go about learning that math?

Superintelligence 5: Forms of Superintelligence

11 KatjaGrace 14 October 2014 01:00AM

This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.


Welcome. This week we discuss the fifth section in the reading guideForms of superintelligence. This corresponds to Chapter 3, on different ways in which an intelligence can be super.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.

There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).

Reading: Chapter 3 (p52-61)


Summary

  1. A speed superintelligence could do what a human does, but faster. This would make the outside world seem very slow to it. It might cope with this partially by being very tiny, or virtual. (p53)
  2. A collective superintelligence is composed of smaller intellects, interacting in some way. It is especially good at tasks that can be broken into parts and completed in parallel. It can be improved by adding more smaller intellects, or by organizing them better. (p54)
  3. A quality superintelligence can carry out intellectual tasks that humans just can't in practice, without necessarily being better or faster at the things humans can do. This can be understood by analogy with the difference between other animals and humans, or the difference between humans with and without certain cognitive capabilities. (p56-7)
  4. These different kinds of superintelligence are especially good at different kinds of tasks. We might say they have different 'direct reach'. Ultimately they could all lead to one another, so can indirectly carry out the same tasks. We might say their 'indirect reach' is the same. (p58-9)
  5. We don't know how smart it is possible for a biological or a synthetic intelligence to be. Nonetheless we can be confident that synthetic entities can be much more intelligent than biological entities
    1. Digital intelligences would have better hardware: they would be made of components ten million times faster than neurons; the components could communicate about two million times faster than neurons can; they could use many more components while our brains are constrained to our skulls; it looks like better memory should be feasible; and they could be built to be more reliable, long-lasting, flexible, and well suited to their environment.
    2. Digital intelligences would have better software: they could be cheaply and non-destructively 'edited'; they could be duplicated arbitrarily; they could have well aligned goals as a result of this duplication; they could share memories (at least for some forms of AI); and they could have powerful dedicated software (like our vision system) for domains where we have to rely on slow general reasoning.

Notes

  1. This chapter is about different kinds of superintelligent entities that could exist. I like to think about the closely related question, 'what kinds of better can intelligence be?' You can be a better baker if you can bake a cake faster, or bake more cakes, or bake better cakes. Similarly, a system can become more intelligent if it can do the same intelligent things faster, or if it does things that are qualitatively more intelligent. (Collective intelligence seems somewhat different, in that it appears to be a means to be faster or able to do better things, though it may have benefits in dimensions I'm not thinking of.) I think the chapter is getting at different ways intelligence can be better rather than 'forms' in general, which might vary on many other dimensions (e.g. emulation vs AI, goal directed vs. reflexive, nice vs. nasty).
  2. Some of the hardware and software advantages mentioned would be pretty transformative on their own. If you haven't before, consider taking a moment to think about what the world would be like if people could be cheaply and perfectly replicated, with their skills intact. Or if people could live arbitrarily long by replacing worn components. 
  3. The main differences between increasing intelligence of a system via speed and via collectiveness seem to be: (1) the 'collective' route requires that you can break up the task into parallelizable subtasks, (2) it generally has larger costs from communication between those subparts, and (3) it can't produce a single unit as fast as a comparable 'speed-based' system. This suggests that anything a collective intelligence can do, a comparable speed intelligence can do at least as well. One counterexample to this I can think of is that often groups include people with a diversity of knowledge and approaches, and so the group can do a lot more productive thinking than a single person could. It seems wrong to count this as a virtue of collective intelligence in general however, since you could also have a single fast system with varied approaches at different times.
  4. For each task, we can think of curves for how performance increases as we increase intelligence in these different ways. For instance, take the task of finding a fact on the internet quickly. It seems to me that a person who ran at 10x speed would get the figure 10x faster. Ten times as many people working in parallel would do it only a bit faster than one, depending on the variance of their individual performance, and whether they found some clever way to complement each other. It's not obvious how to multiply qualitative intelligence by a particular factor, especially as there are different ways to improve the quality of a system. It also seems non-obvious to me how search speed would scale with a particular measure such as IQ. 
  5. How much more intelligent do human systems get as we add more humans? I can't find much of an answer, but people have investigated the effect of things like team sizecity size, and scientific collaboration on various measures of productivity.
  6. The things we might think of as collective intelligences - e.g. companies, governments, academic fields - seem notable to me for being slow-moving, relative to their components. If someone were to steal some chewing gum from Target, Target can respond in the sense that an employee can try to stop them. And this is no slower than an individual human acting to stop their chewing gum from being taken. However it also doesn't involve any extra problem-solving from the organization - to the extent that the organization's intelligence goes into the issue, it has to have already done the thinking ahead of time. Target was probably much smarter than an individual human about setting up the procedures and the incentives to have a person there ready to respond quickly and effectively, but that might have happened over months or years.

In-depth investigations

If you are particularly interested in these topics, and want to do further research, these are a few plausible directions, some inspired by Luke Muehlhauser's list, which contains many suggestions related to parts of Superintelligence. These projects could be attempted at various levels of depth.

  1. Produce improved measures of (substrate-independent) general intelligence. Build on the ideas of Legg, Yudkowsky, Goertzel, Hernandez-Orallo & Dowe, etc. Differentiate intelligence quality from speed.
  2. List some feasible but non-realized cognitive talents for humans, and explore what could be achieved if they were given to some humans.
  3. List and examine some types of problems better solved by a speed superintelligence than by a collective superintelligence, and vice versa. Also, what are the returns on “more brains applied to the problem” (collective intelligence) for various problems? If there were merely a huge number of human-level agents added to the economy, how much would it speed up economic growth, technological progress, or other relevant metrics? If there were a large number of researchers added to the field of AI, how would it change progress?
  4. How does intelligence quality improve performance on economically relevant tasks?
If you are interested in anything like this, you might want to mention it in the comments, and see whether other people have useful thoughts.

How to proceed

This has been a collection of notes on the chapter.  The most important part of the reading group though is discussion, which is in the comments section. I pose some questions for you there, and I invite you to add your own. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

Next week, we will talk about 'intelligence explosion kinetics', a topic at the center of much contemporary debate over the arrival of machine intelligence. To prepare, read Chapter 4, The kinetics of an intelligence explosion (p62-77)The discussion will go live at 6pm Pacific time next Monday 20 October. Sign up to be notified here.

[Link] Why Science Is Not Necessarily Self-Correcting

6 ChristianKl 13 October 2014 01:51PM

Why Science Is Not Necessarily Self-Correcting John P. A. Ioannidis

The ability to self-correct is considered a hallmark of science. However, self-correction does not always happen to scientific evidence by default. The trajectory of scientific credibility can fluctuate over time, both for defined scientific fields and for science at-large. History suggests that major catastrophes in scientific credibility are unfortunately possible and the argument that “it is obvious that progress is made” is weak. Careful evaluation of the current status of credibility of various scientific fields is important in order to understand any credibility deficits and how one could obtain and establish more trustworthy results. Efficient and unbiased replication mechanisms are essential for maintaining high levels of scientific credibility. Depending on the types of results obtained in the discovery and replication phases, there are different paradigms of research: optimal, self-correcting, false nonreplication, and perpetuated fallacy. In the absence of replication efforts, one is left with unconfirmed (genuine) discoveries and unchallenged fallacies. In several fields of investigation, including many areas of psychological science, perpetuated and unchallenged fallacies may comprise the majority of the circulating evidence. I catalogue a number of impediments to self-correction that have been empirically studied in psychological science. Finally, I discuss some proposed solutions to promote sound replication practices enhancing the credibility of scientific results as well as some potential disadvantages of each of them. Any deviation from the principle that seeking the truth has priority over any other goals may be seriously damaging to the self-correcting functions of science.

 

 

Open thread, Oct. 13 - Oct. 19, 2014

5 MrMind 13 October 2014 08:17AM

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

Baysian conundrum

10 Jan_Rzymkowski 13 October 2014 12:39AM

For some time I've been pondering on a certain scenario, which I'll describe shortly. I hope you may help me find a satisfactory answer or at very least be as perplexed by this probabilistic question as me. Feel free to assign any reasonable a priori probabilities as you like. Here's the problem:

It's cold cold winter. Radiators are hardly working, but it's not why you're sitting so anxiously in your chair. The real reason is that tomorrow is your assigned upload (and damn, it's just one in million chance you're not gonna get it) and you just can't wait to leave your corporality behind. "Oh, I'm so sick of having a body, especially now. I'm freezing!" you think to yourself, "I wish I were already uploaded and could just pop myself off to a tropical island."

And now it strikes you. It's a weird solution, but it feels so appealing. You make a solemn oath (you'd say one in million chance you'd break it), that soon after upload you will simulate this exact moment thousand times simultaneously and when the clock strikes 11 AM, you're gonna be transposed to a Hawaiian beach, with a fancy drink in your hand.

It's 10:59 on a clock. What's the probability that you'd be in a tropical paradise in one minute?

And to make things more paradoxical: What would be said probability, if you wouldn't have made such an oath - just seconds ago?

[Link] The Coming Plague

1 Konkvistador 12 October 2014 02:37PM

Related to: Forty Days , Low Hanging Poop

From professor Gregory Cochran's blog West Hunters.

Laurie Garret has an article out in the Washington Post.  She say that there’s no point in trying to block the spread of Ebola by travel bans.

The problem is, she’s full of crap.  Look, there are two possible scenarios.  In both of them, r, the number of new cases generated by each case, is greater than 1 in parts of West Africa – which is why you get exponential growth, why you have an epidemic.  If r < 1.0, the series converges – a case generates a few extra cases before dying out.

Everything we know so far suggests that even though it is greater than 1.0,  r in West Africa is not all that big (maybe around 2), mostly because of unfortunate local burial customs and incompetent medical personnel.

It seems highly likely that r in US conditions is well under 1.0 which means you can’t get an epidemic. However,  r is probably not zero.  It doesn’t mean that you can’t get a few cases per imported case, from immediate contact and hospital mistakes.  As an example, suppose that on average each case imported to the US generated a total of two other cases before dying out (counting secondary, tertiary, etc infections).  Then, on average, the number of US citizens infected would be twice the number of infected visitors.

Now suppose that a travel ban blocked 80% of sick people trying to fly here from Liberia.  We’d have 80% fewer cases in US citizens: and that would be a good thing. Really it would.  Does Laurie Garret understand this?  Obviously not. She is a senior fellow for global health at the Council on Foreign Relations, but she is incompetent.  Totally useless, like virtually everyone else in public life.

We hear people from the CDC saying that any travel restrictions would backfire, but that’s nonsense too.  One might wonder why they say such goofy things: I would guess that a major reason is that they were taught in school that quarantines are useless (and worse yet, old-fashioned), just as many biologists were taught that parasites are really harmless – have to be, because evolution!

In the other scenario, r > 1.0 in US conditions as well, or at least is greater than 1.0 in some subsets of the US population.  This is very unlikely- even more unlikely considering we can adjust our behavior to make transmission less likely.  But suppose it so, for the sake of argument.  Then you would want – need – to stop all travelers from the risky regions, because even one infected guy would pose a huge risk.  Some say that blocking that spread would be impossible. They’re wrong: it is possible*, although it wouldn’t happen, because we’re too crazy.  In fact, in that scenario, we’d be justified in shooting down every plane that _might_ carry an infected passenger.  This scenario is the one that fits Garrett’s remarks, but if she really believed it, she would be frantically buying canned goods and finding a cave in the Rockies to hide her family in.

*the Atlantic is pretty wide.

Positive Book and Other Media Recommendations for a Teen Audience

4 Brendon_Wong 12 October 2014 06:10AM

Hi Less Wrong,

I've got the opportunity to promote books and other forms of content to a largely teenage audience. I'm looking for some good book recommendations and recommendations for a limited amount of other media (websites, movies, etc) that will spread awareness of positive ideas and issues in the world, while still being entertaining to the target audience. HPMOR comes to mind, and although I don't think its main focus is to promote an issue to readers, it is an excellent choice because it is well ranked, promotes rationality which can directly help readers, and is completely free to read online. Recommendations don't have to be free, but that is a very important factor.

Thanks everyone!

Edit: Thanks everyone for your contributions. I realize I was being a little too vague, I think content that promotes ethical/altruistic behavior is mainly what I am looking for, either inspiring it with fiction or causing motivation for it with nonfiction. I am looking for free web content, although paid content recommendations are also appreciated. This is for a general youth audience, not a club or any sort of interest group.

 

2014 Less Wrong Census/Survey - Call For Critiques/Questions

18 Yvain 11 October 2014 06:39AM

It's that time of year again. Actually, a little earlier than that time of year, but I'm pushing it ahead a little to match when Ozy and I expect to have more free time to process the results.

The first draft of the 2014 Less Wrong Census/Survey is complete (see 2013 results here) .

You can see the survey below if you promise not to try to take the survey because it's not done yet and this is just an example!

2014 Less Wrong Census/Survey Draft

I want two things from you.

First, please critique this draft (it's much the same as last year's). Tell me if any questions are unclear, misleading, offensive, confusing, or stupid. Tell me if the survey is so unbearably long that you would never possibly take it. Tell me if anything needs to be rephrased.

Second, I am willing to include any question you want in the Super Extra Bonus Questions section, as long as it is not offensive, super-long-and-involved, or really dumb. Please post any questions you want there. Please be specific - not "Ask something about taxes" but give the exact question you want me to ask as well as all answer choices.

Try not to add more than a few questions per person, unless you're sure yours are really interesting. Please also don't add any questions that aren't very easily sort-able by a computer program like SPSS unless you can commit to sorting the answers yourself.

I will probably post the survey to Main and officially open it for responses sometime early next week.

View more: Next