"The XAI program will focus the development of multiple systems on addressing challenges problems in two areas: (1) machine learning problems to classify events of interest in heterogeneous, multimedia data; and (2) machine learning problems to construct decision policies for an autonomous system to perform a variety of simulated missions."

"At the end of the program, the final delivery will be a toolkit library consisting of machine learning and human-computer interface software modules that could be used to develop future explainable AI systems. After the program is complete, these toolkits would be available for further refinement and transition into defense or commercial applications"

http://www.darpa.mil/program/explainable-artificial-intelligence

Here's a puzzle based on something I used to be confused about:

It is known that utility functions are equivalent (i.e. produce the same preferences over actions) up to a positive affine transformation: u'(x) = au(x) + b where a is positive.

Suppose I have u(vanilla) = 3, u(chocolate) = 8. I prefer an action that yields a 50% chance of chocolate over an action that yields a 100% chance of vanilla, because 0.5(8) > 1.0(3).

Under the positive affine transformation a = 1, b = 4; we get that u'(vanilla) = 7 and u'(chocolate) = 12. Therefore I now prefer the action that yields a 100% chance of vanilla, because 1.0(7) > 0.5(12).

How to resolve the contradiction?

"The evidence of harm would be the evidence that you can hurt some cognitive functions with the same stimulation protocols that help another cognitive function. But they're completely correct that we don't have any evidence saying you're definitely hurting yourself. We do have evidence that you're definitely changing your brain."

interview:

http://www.wbur.org/commonhealth/2016/07/11/caution-brain-hacking

Paper:

http://onlinelibrary.wiley.com/doi/10.1002/ana.24689/references

I was aware of the variability of responses to stim, but not the paper that leveraging one brain function could impair another. This was also written to give the docs some info to help inform their patients.

edit

I'll also tuck this in here, as i posted it to open thread.

Texting changes brain waves to new, previously unknown, pattern.

http://sciencebulletin.org/archives/2623.html

Makes me wonder if they were using spell check, or the new, shortend speak. By using constructed kernels, or images of words and concepts, it looks like machine learning retrieval or construction is already  being practiced here ?

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

This is part of a semi-monthly reading group on Eliezer Yudkowsky's ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.

Welcome to the Rationality reading group. This fortnight we discuss Part Z: The Craft and the Community (pp. 1651-1750). This post summarizes each article of the sequence, linking to the original LessWrong post where available.

Z. The Craft and the Community

312. Raising the Sanity Waterline - Behind every particular failure of social rationality is a larger and more general failure of social rationality; even if all religious content were deleted tomorrow from all human minds, the larger failures that permit religion would still be present. Religion may serve the function of an asphyxiated canary in a coal mine - getting rid of the canary doesn't get rid of the gas. Even a complete social victory for atheism would only be the beginning of the real work of rationalists. What could you teach people without ever explicitly mentioning religion, that would raise their general epistemic waterline to the point that religion went underwater?

313. A Sense That More Is Possible - The art of human rationality may have not been much developed because its practitioners lack a sense that vastly more is possible. The level of expertise that most rationalists strive to develop is not on a par with the skills of a professional mathematician - more like that of a strong casual amateur. Self-proclaimed "rationalists" don't seem to get huge amounts of personal mileage out of their craft, and no one sees a problem with this. Yet rationalists get less systematic training in a less systematic context than a first-dan black belt gets in hitting people.

314. Epistemic Viciousness - An essay by Gillian Russell on "Epistemic Viciousness in the Martial Arts" generalizes amazingly to possible and actual problems with building a community around rationality. Most notably the extreme dangers associated with "data poverty" - the difficulty of testing the skills in the real world. But also such factors as the sacredness of the dojo, the investment in teachings long-practiced, the difficulty of book learning that leads into the need to trust a teacher, deference to historical masters, and above all, living in data poverty while continuing to act as if the luxury of trust is possible.

315. Schools Proliferating Without Evidence - The branching schools of "psychotherapy", another domain in which experimental verification was weak (nonexistent, actually), show that an aspiring craft lives or dies by the degree to which it can be tested in the real world. In the absence of that testing, one becomes prestigious by inventing yet another school and having students, rather than excelling at any visible performance criterion. The field of hedonic psychology (happiness studies) began, to some extent, with the realization that you could measure happiness - that there was a family of measures that by golly did validate well against each other. The act of creating a new measurement creates new science; if it's a good measurement, you get good science.

316. Three Levels of Rationality Verification - How far the craft of rationality can be taken, depends largely on what methods can be invented for verifying it. Tests seem usefully stratifiable into reputational, experimental, andorganizational. A "reputational" test is some real-world problem that tests the ability of a teacher or a school (like running a hedge fund, say) - "keeping it real", but without being able to break down exactly what was responsible for success. An "experimental" test is one that can be run on each of a hundred students (such as a well-validated survey). An "organizational" test is one that can be used to preserve the integrity of organizations by validating individuals or small groups, even in the face of strong incentives to game the test. The strength of solution invented at each level will determine how far the craft of rationality can go in the real world.

317. Why Our Kind Can't Cooperate - The atheist/libertarian/technophile/sf-fan/early-adopter/programmer/etc crowd, aka "the nonconformist cluster", seems to be stunningly bad at coordinating group projects. There are a number of reasons for this, but one of them is that people are as reluctant to speak agreement out loud, as they are eager to voice disagreements - the exact opposite of the situation that obtains in more cohesive and powerful communities. This is not rational either! It is dangerous to be half a rationalist (in general), and this also applies to teaching only disagreement but not agreement, or only lonely defiance but not coordination. The pseudo-rationalist taboo against expressing strong feelings probably doesn't help either.

318. Tolerate Tolerance - One of the likely characteristics of someone who sets out to be a "rationalist" is a lower-than-usual tolerance for flawed thinking. This makes it very important to tolerate other people's tolerance - to avoid rejecting them because they tolerate people you wouldn't - since otherwise we must all have exactly the same standards of tolerance in order to work together, which is unlikely. Even if someone has a nice word to say about complete lunatics and crackpots - so long as they don't literally believe the same ideas themselves - try to be nice to them? Intolerance of tolerance corresponds to punishment of non-punishers, a very dangerous game-theoretic idiom that can lock completely arbitrary systems in place even when they benefit no one at all.

319. Your Price for Joining - The game-theoretical puzzle of the Ultimatum game has its reflection in a real-world dilemma: How much do you demand that an existing group adjust toward you, before you will adjust toward it? Our hunter-gatherer instincts will be tuned to groups of 40 with very minimal administrative demands and equal participation, meaning that we underestimate the inertia of larger and more specialized groups and demand too much before joining them. In other groups this resistance can be overcome by affective death spirals and conformity, but rationalists think themselves too good for this - with the result that people in the nonconformist cluster often set their joining prices way way way too high, like an 50-way split with each player demanding 20% of the money. Nonconformists need to move in the direction of joining groups more easily, even in the face of annoyances and apparent unresponsiveness. If an issue isn't worth personally fixing by however much effort it takes, it's not worth a refusal to contribute.

320. Can Humanism Match Religion's Output? - Anyone with a simple and obvious charitable project - responding with food and shelter to a tidal wave in Thailand, say - would be better off by far pleading with the Pope to mobilize the Catholics, rather than with Richard Dawkins to mobilize the atheists. For so long as this is true, any increase in atheism at the expense of Catholicism will be something of a hollow victory, regardless of all other benefits. Can no rationalist match the motivation that comes from the irrational fear of Hell? Or does the real story have more to do with the motivating power of physically meeting others who share your cause, and group norms of participating?

321. Church vs. Taskforce - Churches serve a role of providing community - but they aren't explicitly optimized for this, because their nominal role is different. If we desire community without church, can we go one better in the course of deleting religion? There's a great deal of work to be done in the world; rationalist communities might potentially organize themselves around good causes, while explicitly optimizing for community.

322. Rationality: Common Interest of Many Causes - Many causes benefit particularly from the spread of rationality - because it takes a little more rationality than usual to see their case, as a supporter, or even just a supportive bystander. Not just the obvious causes like atheism, but things like marijuana legalization. In the case of my own work this effect was strong enough that after years of bogging down I threw up my hands and explicitly recursed on creating rationalists. If such causes can come to terms with not individually capturing all the rationalists they create, then they can mutually benefit from mutual effort on creating rationalists. This cooperation may require learning to shut up about disagreements between such causes, and not fight over priorities, except in specialized venues clearly marked.

323. Helpless Individuals - When you consider that our grouping instincts are optimized for 50-person hunter-gatherer bands where everyone knows everyone else, it begins to seem miraculous that modern-day large institutions survive at all. And in fact, the vast majority of large modern-day institutions simply fail to exist in the first place. This is why funding of Science is largely through money thrown at Science rather than donations from individuals - research isn't a good emotional fit for the rare problems that individuals can manage to coordinate on. In fact very few things are, which is why e.g. 200 million adult Americans have such tremendous trouble supervising the 535 members of Congress. Modern humanity manages to put forth very little in the way of coordinated individual effort to serve our collective individual interests.

324. Money: The Unit of Caring - Omohundro's resource balance principle implies that the inside of any approximately rational system has a common currency of expected utilons. In our world, this common currency is called "money" and it is the unit of how much society cares about something - a brutal yet obvious point. Many people, seeing a good cause, would prefer to help it by donating a few volunteer hours. But this avoids the tremendous gains of comparative advantage, professional specialization, and economies of scale - the reason we're not still in caves, the only way anything ever gets done in this world, the tools grownups use when anyone really cares. Donating hours worked within a professional specialty and paying-customer priority, whether directly, or by donating the money earned to hire other professional specialists, is far more effective than volunteering unskilled hours.

325. Purchase Fuzzies and Utilons Separately - Wealthy philanthropists typically make the mistake of trying to purchase warm fuzzy feelings, status among friends, and actual utilitarian gains, simultaneously; this results in vague pushes along all three dimensions and a mediocre final result. It should be far more effective to spend some money/effort on buying altruistic fuzzies at maximum optimized efficiency (e.g. by helping people in person and seeing the results in person), buying status at maximum efficiency (e.g. by donating to something sexy that you can brag about, regardless of effectiveness), and spending most of your money on expected utilons (chosen through sheer cold-blooded shut-up-and-multiply calculation, without worrying about status or fuzzies).

326. Bystander Apathy - The bystander effect is when groups of people are less likely to take action than an individual. There are a few explanations for why this might be the case.

327. Collective Apathy and the Internet - The causes of bystander apathy are even worse on the Internet. There may be an opportunity here for a startup to deliberately try to avert bystander apathy in online group coordination.

328. Incremental Progress and the Valley - The optimality theorems for probability theory and decision theory, are for perfect probability theory and decision theory. There is no theorem that incremental changes toward the ideal, starting from a flawed initial form, must yield incremental progress at each step along the way. Since perfection is unattainable, why dare to try for improvement? But my limited experience with specialized applications suggests that given enough progress, one can achieve huge improvements over baseline - it just takes a lot of progress to get there.

329. Bayesians vs. Barbarians - Suppose that a country of rationalists is attacked by a country of Evil Barbarians who know nothing of probability theory or decision theory. There's a certain concept of "rationality" which says that the rationalists inevitably lose, because the Barbarians believe in a heavenly afterlife if they die in battle, while the rationalists would all individually prefer to stay out of harm's way. So the rationalist civilization is doomed; it is too elegant and civilized to fight the savage Barbarians... And then there's the idea that rationalists should be able to (a) solve group coordination problems, (b) care a lot about other people and (c) win...

330. Beware of Other-Optimizing - Aspiring rationalists often vastly overestimate their own ability to optimize other people's lives. They read nineteen webpages offering productivity advice that doesn't work for them... and then encounter the twentieth page, or invent a new method themselves, and wow, it really works - they've discovered the true method. Actually, they've just discovered the one method in twenty that works for them, and their confident advice is no better than randomly selecting one of the twenty blog posts. Other-Optimizing is exceptionally dangerous when you have power over the other person - for then you'll just believe that they aren't trying hard enough.

331. Practical Advice Backed by Deep Theories - Practical advice is genuinely much, much more useful when it's backed up by concrete experimental results, causal models that are actually true, or valid math that is validly interpreted. (Listed in increasing order of difficulty.) Stripping out the theories and giving the mere advice alone wouldn't have nearly the same impact or even the same message; and oddly enough, translating experiments and math into practical advice seems to be a rare niche activity relative to academia. If there's a distinctive LW style, this is it.

332. The Sin of Underconfidence - When subjects know about a bias or are warned about a bias, overcorrection is not unheard of as an experimental result. That's what makes a lot of cognitive subtasks so troublesome - you know you're biased but you're not sure how much, and if you keep tweaking you may overcorrect. The danger of underconfidence (overcorrecting for overconfidence) is that you pass up opportunities on which you could have been successful; not challenging difficult enough problems; losing forward momentum and adopting defensive postures; refusing to put the hypothesis of your inability to the test; losing enough hope of triumph to try hard enough to win. You should ask yourself "Does this way of thinking make me stronger, or weaker?"

333. Go Forth and Create the Art! - I've developed primarily the art of epistemic rationality, in particular, the arts required for advanced cognitive reductionism... arts like distinguishing fake explanations from real ones and avoiding affective death spirals. There is much else that needs developing to create a craft of rationality - fighting akrasia; coordinating groups; teaching, training, verification, and becoming a proper experimental science; developing better introductory literature... And yet it seems to me that there is a beginning barrier to surpass before you can start creating high-quality craft of rationality, having to do with virtually everyone who tries to think lofty thoughts going instantly astray, or indeed even realizing that a craft of rationality exists and that you ought to be studying cognitive science literature to create it. It's my hope that my writings, as partial as they are, will serve to surpass this initial barrier. The rest I leave to you.

This has been a collection of notes on the assigned sequence for this fortnight. The most important part of the reading group though is discussion, which is in the comments section. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

This is the end, beautiful friend!

on the 22 August 2015, I wrote an apps list of useful apps, in the comments were a number of suggestions which I immediately tried.  This is an update.  Original can be found at this link:

http://lesswrong.com/r/discussion/lw/mnm/a_list_of_apps_that_are_useful_to_me_and_other/

I rewrite the whole list below.  

But first - my recommended list in short:

• Get an external battery block (and own more than enough spare power cables)
• Wunderlist
• Ingress
• How are you feeling?
• Alarm clock plus
• Twilight
• Business calendar
• Clipper
• Rain alarm
• Data monitor
• Rescuetime
• Powercalc
• Es File viewer
• WheresmyDroid?
• Google Docs/sheets etc.
• (possibly pushbullet and DTG GTD but I have not had them for long enough)

New:

Timestamp Widget. - on clicking to open it - it logs a timestamp.  Can include notes too.

Wunderlist - Recommend it - for shared shopping lists, or any kind of list of things to do.  It's not perfect but it works.

T2 mood tracker - as a second backup to my other mood tracker.  This one takes more effort to do so I only enter the data every few days.  YMMV it might be useful to you.

HOVBX - an overlay for google hangouts that sits on top of the call buttons so you don't accidentally call people (useful for groups who butt-dial each other)

Fleksy - A different keyboard - it seems faster but I am used to swiftkey so I don't use this one.

Tagtime - useful to try.  reminds you hourly or so to tag what you are currently working on.  I used it for a while to help keep me on track.  I noticed I was significantly off track and eventually stopped using it because I felt bad about it.  I feel like I spend more time on-task now but because I want to.  This was a step in the journey of deciding to do that.

Alarm clock plus - it's the best alarm clock app.  I don't use alarms often but this one does everything.

Squats/Push ups/sit ups/pull ups - Rittr labs - good at a simple exercise routine.  Just tells you what to do.  designed to get you from zero to "up to N" of an exercise (250 or 100) so gives you instruction on how many to do each day.  Worth trying.  Didn't work for me, but for other reasons about my lifestyle.

Twilight - mentioned above, replaces night mode and does what f.lux with a PC (filters to be less blue at night)

World clock - started talking to people in different time zones and this was handy.

CPU-Z - lists out all the phone's sensors and tells you their outputs.  cool for looking at gyroscopes/accelerometers.

Coffee meets bagel - dating app.  One profile per day, accept/reject.  Has a different feel to tinder

Bumble - US only; Like Tinder but the girl has to message you first or the connection disappears.

Business Calendar - Best calendar I have found so far

Clipper - Clipboard app for holding the last 20 or so things you have copied.  Also for showing you what's currently on the "copy" 

Pixlr - photo editor.  It's a good one, don't use it often

Rain Alarm - Very good app.  Tells you if it's raining anywhere nearby.  Can be enough to tell you "I should walk home sooner" but also just interesting to have a bit more awareness of your environment.

Audio Scope - Cool science app for viewing the audio scope

Spectrum analyze - Cool science app for viewing the audio spectrum

Frequensee - Fun science app for viewing audio spectrum data

PitchLab lite - Neat for understanding pitch when singing or listening to musical notes.  Another science-visualisation app

Spectralview analyser - another spectrum analyser

Pulsepoint AED - Initiative to gather a public map of all AED's worldwide.  To help; get the app and check the details of nearby AED's

FBreader - Ebook reader.  Pretty good, can control brightness and font size.

KIK - Social app like whatsapp/viber etc.  Don't use it yet, got it on a recommendation.

Wildwilderness - Reporting app for if you see suspicious wildlife trade going on anywhere in the world.  Can report anonymously, any details help.  

DGT GTD - Newly suggested by LW, have not tried to use it yet

Pushbullet - Syncs phone notifications with your PC so you can access things via PC.

I have noticed I often wish "Damn I wish someone had made an app for that" and when I search for it I can't find it.  Then I outsource the search to facebook or other people; and they can usually say - yes, its called X.  Which I can put down to an inability to know how to search for an app on my part; more than anything else.

With that in mind; I wanted to solve the problem of finding apps for other people.

The following is a list of apps that I find useful (and use often) for productive reasons:

The environment

This list is long.  The most valuable ones are the top section that I use regularly.  

Other things to mention:

Internal storage - I have a large internal memory card because I knew I would need lots of space.  So I played the "out of sight out of mind game" and tried to give myself as much space as possible by buying a large internal card.  The future of phones is to not use a microSD card and just use internal storage.  I was taking 1000 photos a month, and since having storage troubles and my phone slowing down I don't take nearly even 1 photo a day.  I would like to change that and will probably make it a future bug of mine to solve.

Battery - I use anker external battery blocks to save myself the trouble of worrying about batteries.  If prepared I leave my house with 2 days of phone charge (of 100% use).  I used to count "wins" of days I beat my phone battery (stay awake longer than it) but they are few and far between.  Also I doubled my external battery power and it sits at two days not one (28000mA + 2*460ma spare phone batteries) This is still true but those batteries don't do what they used to.  Anker have excellent service and refunded the battery that did not stay strong.  I would recommend to all phone users to have a power block.  Phones just are not made with enough battery.

Phone - I have a Samsung S4 (android Running KitKat) because it has a few features I found useful that were not found in many other phones - Cheap, Removable battery, external storage card, replaceable case. I am now on lolipop, and have made use of the external antenna port for a particularly bad low-signal location.

Screen cover - I am using the one that came with the phone still Still

I carry a spare phone case, in the beginning I used to go through one each month; now I have a harder case than before it hasn't broken. I change phone case colours for aesthetics every few months.

I also have swapped out the plastic frame that holds the phone case on as these broke, it was a few dollars on ebay and I needed a teeny screwdriver but other than that it works great now!

MicroUSB cables - I went through a lot of effort to sort this out, it's still not sorted, but its "okay for now".  The advice I have - buy several good cables (read online reviews about it), test them wherever possible, and realise that they die.  Also carry a spare or two.  I have now spent far too much time on this problem.  I am at the end of my phone's life and the MicroUSB port is dying, I have replaced it with a new one which is also not great, and I now leave my phone plugged into it's microUSB cable.  I now use Anker brand cabled which are excellent, but my phone still kills one every few weeks.  The whole idea of the MicroUSB plug is awful.  They don't work very well at all.

Restart - I restart my phone probably most days when it gets slow.  It's got programming bugs, but this solution works for now.

The overlays

These sit on my screen all the time.

Data monitor - Gives an overview of bits per second upload or download. updated every second. ✓

CpuTemp - Gives an overlay of the current core temperature.  My phone is always hot, I run it hard with bluetooth, GPS and wifi blaring all the time.  I also have a lot of active apps. ✓

M̶i̶n̶d̶f̶u̶l̶n̶e̶s̶s̶ ̶b̶e̶l̶l̶ ̶-̶ ̶M̶y̶ ̶p̶h̶o̶n̶e̶ ̶m̶a̶k̶e̶s̶ ̶a̶ ̶c̶h̶i̶m̶e̶ ̶e̶v̶e̶r̶y̶ ̶h̶a̶l̶f̶ ̶h̶o̶u̶r̶ ̶t̶o̶ ̶r̶e̶m̶i̶n̶d̶ ̶m̶e̶ ̶t̶o̶ ̶c̶h̶e̶c̶k̶,̶ ̶"̶A̶m̶ ̶I̶ ̶d̶o̶i̶n̶g̶ ̶s̶o̶m̶e̶t̶h̶i̶n̶g̶ ̶o̶f̶ ̶h̶i̶g̶h̶-̶v̶a̶l̶u̶e̶ ̶r̶i̶g̶h̶t̶ ̶n̶o̶w̶?̶"̶ ̶i̶t̶ ̶s̶o̶m̶e̶t̶i̶m̶e̶s̶ ̶s̶t̶o̶p̶s̶ ̶m̶e̶ ̶f̶r̶o̶m̶ ̶d̶o̶i̶n̶g̶ ̶c̶r̶a̶p̶ ̶t̶h̶i̶n̶g̶s̶.̶ Wow that didn't last.  It was so annoying that I stopped using it.

Facebook chat heads - I often have them open, they have memory leaks and start slowing down my phone after a while, I close and reopen them when I care enough.✓ memory leaks improved but are still there.

The normals:

Facebook - communicate with people.  I do this a lot.✓

Inkpad - its a note-taking app, but not an exceptionally great one; open to a better suggestion.✓

Ingress - it makes me walk; it gave me friends; it put me in a community.  Downside is that it takes up more time than you want to give it.  It's a mobile GPS game.  Join the Resistance. Highly recommend

Maps (google maps) - I use this most days; mostly for traffic assistance to places that I know how to get to.✓

Camera - I take about 1000 photos a month.  Generic phone-app one. I take significantly less photos now, my phone slowed down so the activation energy for *open the camera* is higher.  I plan to try to fix this soon

Assistive light - Generic torch app (widget) I use this daily.✓

Hello - SMS app.  I don't like it but its marginally better than the native one.✓

S̶u̶n̶r̶i̶s̶e̶ ̶c̶a̶l̶e̶n̶d̶a̶r̶ ̶-̶ ̶I̶ ̶d̶o̶n̶'̶t̶ ̶l̶i̶k̶e̶ ̶t̶h̶e̶ ̶n̶a̶t̶i̶v̶e̶ ̶c̶a̶l̶e̶n̶d̶a̶r̶;̶ ̶I̶ ̶d̶o̶n̶'̶t̶ ̶l̶i̶k̶e̶ ̶t̶h̶i̶s̶ ̶o̶r̶ ̶a̶n̶y̶ ̶o̶t̶h̶e̶r̶ ̶c̶a̶l̶e̶n̶d̶a̶r̶.̶ ̶ ̶T̶h̶i̶s̶ ̶i̶s̶ ̶t̶h̶e̶ ̶l̶e̶a̶s̶t̶ ̶b̶a̶d̶ ̶o̶n̶e̶ ̶I̶ ̶h̶a̶v̶e̶ ̶f̶o̶u̶n̶d̶.̶ ̶ ̶I̶ ̶h̶a̶v̶e̶ ̶a̶n̶ ̶a̶p̶p̶ ̶c̶a̶l̶l̶e̶d̶ ̶"̶f̶a̶c̶e̶b̶o̶o̶k̶ ̶s̶y̶n̶c̶"̶ ̶w̶h̶i̶c̶h̶ ̶h̶e̶l̶p̶s̶ ̶w̶i̶t̶h̶ ̶e̶n̶t̶e̶r̶i̶n̶g̶ ̶i̶n̶ ̶a̶ ̶f̶r̶a̶c̶t̶i̶o̶n̶ ̶o̶f̶ ̶t̶h̶e̶ ̶e̶v̶e̶n̶t̶s̶ ̶i̶n̶ ̶m̶y̶ ̶l̶i̶f̶e̶.̶

Business Calendar - works better, has a better interface than Sunrise.

Phone, address book, chrome browser.✓  I use tab sync, and recommend it for all your chrome-enabled devices.

GPS logger - I have a log of my current gps location every 5 minutes.  If google tracks me I might as well track myself.  I don't use this data yet but its free for me to track; so if I can find a use for the historic data that will be a win. I don't make use of this data and can access my google data just fine so I might stop tracking this.

Quantified apps:

Fit - google fit; here for multiple redundancy✓

S Health - Samsung health - here for multiple redundancy✓

Fitbit - I wear a flex step tracker every day, and input my weight daily manually through this app✓

Basis - I wear a B1 watch, and track my sleep like a hawk.✓

Rescuetime - I track my hours on technology and wish it would give a better breakdown. (I also paid for their premium service)✓

Voice recorder - generic phone app; I record around 1-2 hours of things I do per week.  Would like to increase that. I now use this for one hour a month or less.

Narrative - I recently acquired a life-logging device called a narrative, and don't really know how to best use the data it gives.  But its a start. I tried using the device but it has poor battery life.  I also received negative feedback when wearing it in casual settings.  This increases the activation energy to using it.  I also can't seem to wear it at the right height and would regularly take photos of the tops of people's heads.  I would come home with a photo a minute for a day (and have the battery die on it a few times) and have one use-able photo in the lot.  significantly lower than I was expecting.

How are you feeling? - Mood tracking app - this one is broken but the best one I have found, it doesn't seem to open itself after a phone restart; so it won't remind you to enter in a current mood.  I use a widget so that I can enter in the mood quickly.  The best parts of this app are the way it lets you zoom out, and having a 10 point scale.  I used to write a quick sentence about what I was feeling, but that took too much time so I stopped doing it. Highly recommend I use this every day.

Stopwatch - "hybrid stopwatch" - about once a week I time something and my phone didn't have a native one.  This app is good at being a stopwatch.✓

Callinspector - tracks ingoing or outgoing calls and gives summaries of things like, who you most frequently call, how much data you use, etc.  can also set data limits. I dont do anything with this data so I think I will stop using it and save my phone's battery life.

Misc

Powercalc - the best calculator app I could find ✓

N̶i̶g̶h̶t̶ ̶m̶o̶d̶e̶ ̶-̶ ̶f̶o̶r̶ ̶s̶a̶v̶i̶n̶g̶ ̶b̶a̶t̶t̶e̶r̶ ̶(̶i̶t̶ ̶d̶i̶m̶s̶ ̶y̶o̶u̶r̶ ̶s̶c̶r̶e̶e̶n̶)̶,̶ ̶I̶ ̶d̶o̶n̶'̶t̶ ̶u̶s̶e̶ ̶t̶h̶i̶s̶ ̶o̶f̶t̶e̶n̶ ̶b̶u̶t̶ ̶i̶t̶ ̶i̶s̶ ̶g̶o̶o̶d̶ ̶a̶t̶ ̶w̶h̶a̶t̶ ̶i̶t̶ ̶d̶o̶e̶s̶.̶ ̶ ̶I̶ ̶w̶o̶u̶l̶d̶ ̶c̶o̶n̶s̶i̶d̶e̶r̶ ̶a̶n̶ ̶a̶p̶p̶ ̶t̶h̶a̶t̶ ̶d̶i̶m̶s̶ ̶t̶h̶e̶ ̶b̶l̶u̶e̶ ̶l̶i̶g̶h̶t̶ ̶e̶m̶i̶t̶t̶e̶d̶ ̶f̶r̶o̶m̶ ̶m̶y̶ ̶s̶c̶r̶e̶e̶n̶;̶ ̶h̶o̶w̶e̶v̶e̶r̶ ̶I̶ ̶d̶o̶n̶'̶t̶ ̶n̶o̶t̶i̶c̶e̶ ̶a̶n̶y̶ ̶n̶e̶g̶a̶t̶i̶v̶e̶ ̶s̶l̶e̶e̶p̶ ̶e̶f̶f̶e̶c̶t̶s̶ ̶s̶o̶ ̶I̶ ̶h̶a̶v̶e̶ ̶b̶e̶e̶n̶ ̶p̶u̶t̶t̶i̶n̶g̶ ̶o̶f̶f̶ ̶g̶e̶t̶t̶i̶n̶g̶ ̶a̶r̶o̶u̶n̶d̶ ̶t̶o̶ ̶i̶t̶.̶ ̶

Advanced signal status - about once a month I am in a place with low phone signal - this one makes me feel better about knowing more details of what that means.✓

Ebay - To be able to buy those $5 solutions to problems on the spot is probably worth more than $5 of "impulse purchases" that they might be classified as.✓

C̶a̶l̶ ̶-̶ ̶a̶n̶o̶t̶h̶e̶r̶ ̶c̶a̶l̶e̶n̶d̶a̶r̶ ̶a̶p̶p̶ ̶t̶h̶a̶t̶ ̶s̶o̶m̶e̶t̶i̶m̶e̶s̶ ̶c̶a̶t̶c̶h̶e̶s̶ ̶e̶v̶e̶n̶t̶s̶ ̶t̶h̶a̶t̶ ̶t̶h̶e̶ ̶f̶i̶r̶s̶t̶ ̶o̶n̶e̶ ̶m̶i̶s̶s̶e̶s̶.̶ Nope just using business calendar now.

ES file explorer - for searching the guts of my phone for files that are annoying to find.  Not as used or as useful as I thought it would be but still useful.✓

Maps.Me - I went on an exploring adventure to places without signal; so I needed an offline mapping system.  This map saved my life.✓ Have not used this since then, but I will not delete it.

Wikipedia - information lookup✓

Youtube - don't use it often, but its there.✓

How are you feeling? (again) - I have this in multiple places to make it as easy as possible for me to enter in this data✓

Play store - Makes it easy to find.✓

Gallery - I take a lot of photos, but this is the native gallery and I could use a better app.✓

Social

In no particular order;

F̶a̶c̶e̶b̶o̶o̶k̶ ̶g̶r̶o̶u̶p̶s̶ was so annoying I got rid of it, Yahoo Mail, Skype, Facebook Messenger chat heads, Whatsapp, meetup, google+, Hangouts, Slack, Viber, OKcupid, Gmail, Tinder, Chatango, CoffeeMeetsBagel, Signal.  Of which I use very little.

They do social things.  

I don't really use:  Viber, OKC, Gmail, Tinder, Chatango, CMB, Signal, whatsapp, G+.

I use: Slack, Facebook messenger, yahoo mail  every day.

Not used:

(ticks here mean they are still in this category and are not used)

Trello✓

Workflowy✓

pocketbook✓

snapchat Deleted.

AnkiDroid - Anki memoriser app for a phone. ✓

MyFitnessPal - looks like a really good app, have not used it ✓

Fitocracy - looked good✓

I got these apps for a reason; but don't use them.

Not on my front pages:

These I don't use as often; or have not moved to my front pages (skipping the ones I didn't install or don't use)

S memo - samsung note taking thing, I rarely use, but do use once a month or so.✓

Drive, Docs, Sheets - The google package.  Its terrible to interact with documents on your phone, but I still sometimes access things from my phone.✓Useful for viewing, not effective for editing.

bubble - I don't think I have ever used this Deleted

Compass pro - gives extra details about direction. I never use it.Deleted

(ingress apps) Glypher, Agentstats, integrated timer, cram, notify Don't use them, but still there

TripView (public transport app for my city) Deleted

Convertpad - converts numbers to other numbers. Sometimes quicker than a google search.✓

ABC Iview - National TV broadcasting channel app.  Every program on this channel is uploaded to this app, I have used it once to watch a documentary since I got the app. Deleted

AnkiDroid - I don't need to memorise information in the way it is intended to be used; so I don't use it. Cram is also a flashcard app but I don't use it. Not used

First aid - I know my first aid but I have it anyway for the marginal loss of 50mb of space. Still haven't used it once.

Triangle scanner - I can scan details from NFC chips sometimes. Still haven't used it once.

MX player - does videos better than native apps. Rarely used

Zarchiver - Iunno.  Does something.  Rarely used

Pandora - Never used Deleted

Soundcloud - used once every two months, some of my friends post music online.  Deleted - They have a web interface.

Barcode scanner - never used

Diskusage - Very useful.  Visualises where data is being taken up on your phone, helps when trying to free up space.✓

Swiftkey - Better than native keyboards.  Gives more freedom, I wanted a keyboard with black background and pale keys, swiftkey has it.✓

Google calendar - don't use it, but its there to try to use.✓

Sleepbot - doesn't seem to work with my phone, also I track with other methods, and I forget to turn it on; so its entirely not useful in my life for sleep tracking. Deleted

My service provider's app.

AdobeAcrobat - use often; not via the icon though. ✓

Wheresmydroid? - seems good to have; never used.  My phone is attached to me too well for me to lose it often.  I have it open most of the waking day maybe. ✓ I actually set this up and tested if it worked.  It doesn't work from install, needs an account (which I now have) make sure you actually have an account

Uber - I don't use ubers. Deleted

Terminal emulator, AIDE, PdDroid party, Processing Android, An editor for processing, processing reference, learn C++ - programming apps for my phone, I don't use them, and I don't program much. Deleted some to make space on my phone.

Airbnb - Have not used yet, done a few searches for estimating prices of things. Deleted - Web interface better.

Heart rate - measures your heart rate using the camera/flash.  Neat, not useful other than showing off to people how its possible to do. ✓

Basis - (B1 app), - has less info available than their new app. ✓

BPM counter - Neat if you care about what a "BPM" is for music.  Don't use often. ✓

Sketch guru - fun to play with, draws things. ✓

DJ studio 5 - I did a dj thing for a friend once, used my phone.  was good. ✓

Facebook calendar Sync - as the name says. ✓

Dual N-back - I Don't use it.  I don't think it has value giving properties. Deleted

Awesome calendar - I don't use but it comes with good reccomendations. Deleted Use Business Calendar now.

Battery monitor 3 - Makes a graph of temperature and frequency of the cores.  Useful to see a few times.  Eventually its a bell curve. ✓

urbanspoon - local food places app. ✓use google mostly now.

Gumtree - Australian Ebay (also ebay owns it now) ✓

Printer app to go with my printer ✓

Car Roadside assistance app to go with my insurance ✓

Virgin air entertainment app - you can use your phone while on the plane and download entertainment from their in-flight system. ✓

Two things now;

What am I missing? Was this useful?  Ask me to elaborate on any app and why I used it.  If I get time I will do that anyway. 

P.S. this took 1.5 hours to review and rewrite.

P.P.S - I was intending to make, keep and maintain a list of useful apps, that is not what this document is.  If there are enough suggestions that it's time to make and keep a list; I will do that.

My table of contents links to my other writings

This is part of a semi-monthly reading group on Eliezer Yudkowsky's ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.

Welcome to the Rationality reading group. This fortnight we discuss Part V: Value Theory (pp. 1359-1450). This post summarizes each article of the sequence, linking to the original LessWrong post where available.

V. Value Theory

264. Where Recursive Justification Hits Bottom - Ultimately, when you reflect on how your mind operates, and consider questions like "why does Occam's Razor work?" and "why do I expect the future to be like the past?", you have no other option but to use your own mind. There is no way to jump to an ideal state of pure emptiness and evaluate these claims without using your existing mind.

265. My Kind of Reflection - A few key differences between Eliezer Yudkowsky's ideas on reflection and the ideas of other philosophers.

266. No Universally Compelling Arguments - Because minds are physical processes, it is theoretically possible to specify a mind which draws any conclusion in response to any argument. There is no argument that will convince every possible mind.

267. Created Already in Motion - There is no computer program so persuasive that you can run it on a rock. A mind, in order to be a mind, needs some sort of dynamic rules of inference or action. A mind has to be created already in motion.

268. Sorting Pebbles into Correct Heaps - A parable about an imaginary society that has arbitrary, alien values.

269. 2-Place and 1-Place Words - It is possible to talk about "sexiness" as a property of an observer and a subject. It is also equally possible to talk about "sexiness" as a property of a subject, as long as each observer can have a different process to determine how sexy someone is. Failing to do either of these will cause you trouble.

270. What Would You Do Without Morality? - If your own theory of morality was disproved, and you were persuaded that there was no morality, that everything was permissible and nothing was forbidden, what would you do? Would you still tip cabdrivers?

271. Changing Your Metaethics - Discusses the various lines of retreat that have been set up in the discussion on metaethics.

272. Could Anything Be Right? - You do know quite a bit about morality. It's not perfect information, surely, or absolutely reliable, but you have someplace to start. If you didn't, you'd have a much harder time thinking about morality than you do.

273. Morality as Fixed Computation - A clarification about Yudkowsky's metaethics.

274. Magical Categories - We underestimate the complexity of our own unnatural categories. This doesn't work when you're trying to build a FAI.

275. The True Prisoner's Dilemma - The standard visualization for the Prisoner's Dilemma doesn't really work on humans. We can't pretend we're completely selfish.

276. Sympathetic Minds - Mirror neurons are neurons that fire both when performing an action oneself, and watching someone else perform the same action - for example, a neuron that fires when you raise your hand or watch someone else raise theirs. We predictively model other minds by putting ourselves in their shoes, which is empathy. But some of our desire to help relatives and friends, or be concerned with the feelings of allies, is expressed as sympathy, feeling what (we believe) they feel. Like "boredom", the human form of sympathy would not be expected to arise in an arbitrary expected-utility-maximizing AI. Most such agents would regard any agents in its environment as a special case of complex systems to be modeled or optimized; it would not feel what they feel.

277. High Challenge - Life should not always be made easier for the same reason that video games should not always be made easier. Think in terms of eliminating low-quality work to make way for high-quality work, rather than eliminating all challenge. One needs games that are fun to play and not just fun to win. Life's utility function is over 4D trajectories, not just 3D outcomes. Values can legitimately be over the subjective experience, the objective result, and the challenging process by which it is achieved - the traveller, the destination and the journey.

278. Serious Stories - Stories and lives are optimized according to rather different criteria. Advice on how to write fiction will tell you that "stories are about people's pain" and "every scene must end in disaster". I once assumed that it was not possible to write any story about a successful Singularity because the inhabitants would not be in any pain; but something about the final conclusion that the post-Singularity world would contain no stories worth telling seemed alarming. Stories in which nothing ever goes wrong, are painful to read; would a life of endless success have the same painful quality? If so, should we simply eliminate that revulsion via neural rewiring? Pleasure probably does retain its meaning in the absence of pain to contrast it; they are different neural systems. The present world has an imbalance between pain and pleasure; it is much easier to produce severe pain than correspondingly intense pleasure. One path would be to address the imbalance and create a world with more pleasures, and free of the more grindingly destructive and pointless sorts of pain. Another approach would be to eliminate pain entirely. I feel like I prefer the former approach, but I don't know if it can last in the long run.

279. Value is Fragile - An interesting universe, that would be incomprehensible to the universe today, is what the future looks like if things go right. There are a lot of things that humans value that if you did everything else right, when building an AI, but left out that one thing, the future would wind up looking dull, flat, pointless, or empty. Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals, will contain almost nothing of worth.

280. The Gift We Give to Tomorrow - How did love ever come into the universe? How did that happen, and how special was it, really?

This has been a collection of notes on the assigned sequence for this fortnight. The most important part of the reading group though is discussion, which is in the comments section. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

The next reading will cover Part W: Quantified Humanism (pp. 1453-1514) and Interlude: The Twelve Virtues of Rationality (pp. 1516-1521). The discussion will go live on Wednesday, 23 March 2016, right here on the discussion forum of LessWrong.

So I'm going through the sequences (in AI to Zombies) and I get to the bit about Asch's Conformity Experiment.

It's a good bit of writing, but I mostly pass by without thinking about it too much.  I've been taught about the experiment before, and while Eliezer's point of whether or not the subjects were behaving rationally is interesting, it kind of got swallowed up by his discussion of lonely dissent, which I thought was more engaging.

Later, after I'd passed the section on cult attractors and got into the section on letting go, a thought occurred to me, something I'd never actually thought before.

Eliezer notes:

Three-quarters of the subjects in Asch's experiment gave a "conforming" answer at least once.  A third of the subjects conformed more than half the time.

That answer is surprising.  It was surprising to me the first time I learned about the experiment, and I think it's surprising to just about everyone the first time they hear it.  Same thing with a lot of the psychology surrounding heuristics and biases, actually.  Forget the Inquisition - no one saw the Stanford Prison Experiment coming.

Here's the thought I had:  Why was that result so surprising to me?

I'm not an expert in history, but I know plenty of religious people.  I've learned about the USSR and China, about Nazi Germany and Jonestown.  I have plenty of available evidence of times where people went along with things they wouldn't have on their own.  And not all of them are negative.  I've gone to blood drives I probably wouldn't have if my friends weren't going as well.

When I thought about what my prediction would be, had I been asked what percentage of people I thought would dissent before being told, I think I would have guessed that more than 80% of subject would consistently dissent.  If not higher.

And yet that isn't what the experiment shows, and it isn't even what history shows.  For every dissenter in history, there have to be at least a few thousand conformers.  At least.  So why did I think dissent was the norm?

I notice that I am confused.

So I decide to think about it, and my brain immediately spits out: you're an American in an individualistic culture.  Hypothesis: you expect people to conform less because of the culture you live in/were raised in.  This begs the question: have their been cross-cultural studies done on Asch's Conformity Experiment?  Because if people in China conform more than people in America, then how much people conform probably has something to do with culture.

A little googling brings up a 1996 paper that does a meta-analysis on studies that repeated Asch's experiments, either with a different culture, or at a later date in time.  Their findings:

The results of this review can be summarized in three parts.

First, we investigated the impact of a number of potential moderator variables, focusing just on those studies conducted in the United States where we were able to investigate their relationship with conformity, free of any potential interactions with cultural variables. Consistent with previous research, conformity was significantly higher, (a) the larger the size of the majority, (b) the greater the proportion of female respondents, (c) when the majority did not consist of out-group members, and (d) the more ambiguous the stimulus. There was a nonsignificant tendency for conformity to be higher, the more consistent the majority. There was also an unexpected interaction effect: Conformity was higher in the Asch (1952b, 1956) paradigm (as was expected), but only for studies using Asch's (1956) stimulus materials; where other stimulus materials were used (but where the task was also judging which of the three comparison lines was equal to a standard), conformity was higher in the Crutchfield (1955) paradigm. Finally, although we had expected conformity to be lower when the participant's response was not made available to the majority, this variable did not have a significant effect.

The second area of interest was on changes in the level of conformity over time. Again the main focus was on the analysis just using studies conducted in the United States because it is the changing cultural climate of Western societies which has been thought by some to relate to changes in conformity. We found a negative relationship. Levels of conformity in general had steadily declined since Asch's studies in the early 1950s. We did not find any evidence for a curvilinear trend (as, e.g., Larsen, 1982, had hypothesized), and the direction was opposite to that predicted by Lamb and Alsifaki (1980).

The third and major area of interest was in the impact of cultural values on conformity, and specifically differences in individualism-collectivism. Analyses using measures of cultural values derived from Hofstede (1980, 1983), Schwartz (1994), and Trompenaars (1993) revealed significant relationships confirming the general hypothesis that conformity would be higher in collectivist cultures than in individualist cultures. That all three sets of measures gave similar results, despite the differences in the samples and instruments used, provides strong support for the hypothesis. Moreover, the impact of the cultural variables was greater than any other, including those moderator variables such as majority size typically identified as being important factors.

Cultural values, it would seem, are significant mediators of response in group pressure experiments.

So, while the paper isn't definitive, it (and the papers it draws from) show reasonable evidence that there is a cultural impact on how much people conform.

I thought about that for a little while, and then I realized that I hadn't actually answered my own question.

My confusion stems from the disparity between my prediction and reality.  I'm not wondering about the effect culture has on conformity (the territory), I'm wondering about the effect culture has on my prediction of conformity (the map).

In other words, do people born and raised in a culture with collectivist values (China, for example) or who actually do conform beyond the norm (people who are in a flying-saucer cult, or the people actually living in a compound) expect people to conform more than I did?  Is their map any different from mine?

Think about it - with all the different cult attractors, it probably never feels as though you are vastly conforming, even if you are in a cult.  The same can probably be said for any collectivist society.  Imagine growing up in the USSR - would you predict that people would conform with any higher percentage than someone born in 21st century America?  If you were raised in an extremely religious household, would you predict that people would conform as much as they do?  Less?  More?

How many times have I agreed with a majority even when I knew they probably weren't right, and never thought of it as "conformity"?  It took a long time for my belief in god to finally die, even when I could admit that I just believed that I believed.  And why did I keep believing (or keep trying to/saying that I believed)?

Because it's really hard to actually dissent.  And I wasn't even lonely.

So why was my map that wrong?

What background process or motivated reasoning or...whatever caused that disparity?

One thing that, I think, contributes, is that I was generalizing from fictional evidence.  Batman comes far more readily to my mind than Jonestown.  For that matter, Batman comes more readily to my mind than the millions of not-Batmans in Gotham city.  I was also probably not being moved by history enough.  For every Spartacus, there are at minimum hundreds of not-Spartuses, no matter what the not-Spartacuses say when asked.

But to predict that three-quarters of subjects would conform at least once seems to require a level of pessimism beyond even that.  After all, there were no secret police in Asch's experiment; no one had emptied their bank accounts because they thought the world was ending.

Perhaps I'm making a mistake by putting myself into the place of the subject of the experiment.  I think I'd dissent, but I would predict that most people think that, and most people conformed at least once.  I'm also a reasonably well-educated person, but that didn't seem to help the college students in the experiment.

Has any research been done on people's prediction of their own and other's conformity, particularly across cultures or in groups that are "known" for their conformity (communism, the very religious, etc.)?  Do people who are genuine dissenters predict that more people will dissent than people who genuinely conform?

I don't think this is a useless question.  If you're starting a business that offers a new solution to a problem where solutions already exist, are you overestimating how many people will dissent and buy your product?

Based on an earlier suggestion, here's an outreach thread where you can leave comments about any recent outreach that you have done to convey rationality-style ideas broadly. The goal of having this thread is to organize information about outreach and provide community support and recognition for raising the sanity waterline. Likewise, doing so can help inspire others to emulate some aspects of these good deeds through social proof and network effects.

For a moment lets assume there is some alien intelligent life on our galaxy which is older than us and that it have succeeded in creating super-intelligent self-modifying AI.

Then what set of values and/or goals it is plausible for it to have, given our current observations (I.e. that there is no evidence of it`s existence)?

Some examples:

It values non-interference with nature (some kind of hippie AI)

It values camouflage/stealth for it own defense/security purposes.

It just cares about exterminating their creators and nothing else.

Other thoughts?

TL;DR: Biorisks could result in extinction because of multipandemic in near future and their risks is the same order magnitude as risks of UFAI. A lot of biorisks exist, they are cheap and could happen soon.

It may be surprising that number of published research about risks of biological global catastrophe is much less than number of papers about risks of self-improving AI. (One of exception here is "Strategic terrorism” research parer by former chief technology officer of Microsoft.)

It can’t be explain by the fact that biorisks have smaller probability (it will not be known until Bostrom will write the book “Supervirus”). I mean we don’t know it until a lot of research will be done.

Also biorisks are closer in time than AI risks and because of it they shadow AI risks, lowering the probability that extinction will happen by means of UFAI, because it could happen before it by means of bioweapons (e.g. if UFAI risk is 0.9, but chances that we will die before its creation from bioweapons is 0.8, than actual AI risk is 0.18). So studying biorisks may be more urgent than AI risks. 

There is no technical problem to create new flu virus that could kill large part of human population. And the idea of multi pandemic - that it the possibility to release 100 different agents simultaneously - tells us that biorisk could have arbitrary high global lethality. Most of bad things from this map may be created in next 5-10 years, and no improbable insights are needed. Biorisks are also very cheap in production and small civic or personal biolab could  be used to create them.

May be research in estimation probability of human extinction by biorisks had been done secretly? I am sure that a lot of analysis of biorisks exist in secret. But this means that they do not exist in public and scientists from other domains of knowledge can’t independently verify them and incorporate into broader picture of risks. The secrecy here may be useful if it concerns concrete facts about how to crete a dangerous virus. (I was surprised by effectiveness with which Ebola epidemic was stopped after the decision to do so was made, so maybe I should not underestimate government knowledge on the topic).

I had concerns if I should publish this map. I am not a biologist and chances that I will find really dangerous information are small. But what if I inspire bioterrorists to create bioweapons? Anyway we have a lot of movies with such inspiration. 

So I self-censored one idea that may be too dangerous to publish and put black box instead. I also have a section of prevention methods in the lower part of the map. All ideas in the map may be found in wikipedia or other open sources.

The goal of this map is to show importance of risks connected with new kinds of biological weapons which could be created if all recent advances in bioscience will be used for bad. The map shows what we should be afraid off and try to control. So it is map of possible future development of the field of biorisks.

Not any biocatastrophe will result in extinction, it is in the fat tail of the distribution. But smaller catastrophes may delay other good things and wider our window of vulnerability. If protecting measures will be developed on the same speed as possible risks we are mostly safe. If total morality of bioscientists is high we are most likely safe too - no one will make dangerous experiments.

Timeline: Biorisks are growing at least exponentially with the speed of Moore law in biology. After AI will be created and used to for global government and control, biorisks will probably ended. This means that last years before AI creation will be most dangerous from the point of biorisks. 

The first part of the map presents biological organisms that could be genetically edited for global lethality and each box presents one scenario of a global catastrophe. While many boxes are similar to existing bioweapons, they are not the same as not much known bioweapons could result in large scale pandemic (except smallpox and flu). Most probable biorisks are outlined in red in the map. And the real one will be probably not from the map as the world bio is very large and I can’t cover it all.

The map is provided with links which are clickable in the pdf, which is here: http://immortality-roadmap.com/biorisk.pdf 

I have grown up in a family of meat-eaters and therefore have been eating meat all my life. I until recently I have never spent much time thinking about it. I justified my behaviour by saying that animal lives do not matter, because they are not self-conscious and animal pain does not matter, because they have no memory of pain and therefore, as soon as the actual pain is over it is like it has never happened.

In the recent weeks I have spent some time to properly think this through and form an informed believe about whether I can justify eating meat. I would like to hear your thoughts about my thought process and results, because this is a decision that I really don’t want to get wrong.

I have Identified 5 possible problems with meat consumption.

1. Meat requires us to kill animals. 
2.  Factory farmed animals are in a considerable amount of pain for most of their life.
3.  Meat productions requires much more space than producing plants, and therefore might contribute to the world hunger
4.  Some Studies claim that meat, especially if factory farmed, is unhealthy.
5. Meat production is bad for the environment (partly because of point 4, but also for other reasons)

I have decided to ignore problems 4-5 at the beginning, because admitting that they are true would impose weaker restrictions on me. If I come to the conclusion, that I don’t want to eat meat for reason 1, I could no longer eat any meat and reason 2 would forbid me to eat factory farmed meat, which would essentially bring my meat consumption down to something close to zero. 

Reasons 4 and 5 would limit my meat consumption far less, since I do lots of other things that are unhealthy (like eating candy and snacks) or harmful to the environment (like traveling by plane) and while I might come to the conclusion that I want to reduce my meat consumption for reasons 4-5, I expect to have many situations left, where eating meat gives me enough utility to still do it in spite of that reasons.

Reason 3 would also be important, but I am fairly sure, that the problem mostly lies with the lack of spending power in poorer countries, and that it will not lead to more food in Africa if I stop eating meat. For that reason I did not do further research on this.

So what I did was to think about problems 1 and 2 and decide to revisit 4 and 5 if I come to the conclusion that 1 and 2 still allow me to continue eating meat like I do now. 

Is it justifiable to kill animals?

It is clear to me that it is wrong to kill a Human being with a not significantly damaged brain. It is also clear that I have absolutely no problems with killing bacteria or other very simple living beings. Therefore there must exist some features besides the fact that they live that a human has and a bacterium has not, that divides living beings into things that I am willing to kill and things that I am not willing to kill.

The criterion that I used up to know was self-consciousness, which is very convenient because it puts the line between humans (and likely great apes as well) on one side, and basically everything I want to eat on the other side.

There are quite a few things that justify this criterion such as:

1. From a preference utilitarian Perspective, only a self-conscious being can have preferences for the future, therefore you can only violate the preferences of a self-conscious being by killing it. This would be a knock down argument under the premise that preference utilitarism (and not for example normal utilitarism) is the ethical principle to go with 
2. Although I am no expert in this field I believe that it is relatively easy to build a virtual being (for example in a computer game) or with a bit more effort even a robot, that behaves in the way that leads current researchers come to the conclusion that animals have some kind Of Utility. I count the fact that it is easy to build such a thing as evidence, that animals might function in a similar way and I would not have a problem with “hurting” this virtual thing. Therefor if Animals work this way I have no problem with hurting them.
3. This explanation from Eliezer: https://m.facebook.com/yudkowsky/posts/10152588738904228  which I will come back to when I talk about pain, but which is relevant here as well. (Might to some degree be similar to my  point 2)  

There are however other Arguments against it. 

1. Some animals do things that are far more complex than reacting to pain and simple pleasures such as forming relationships for life or mourning if a group member dies. Those things require a more developed brain and are features that most people would see as characteristic for Humans. Since the fact that we kill animals but not humans must come from differences between them, the similar both are, the less likely it is that treating them differently is justified. 
2. From a certain utilitarian perspective (Namely the one that cares about utility of existing beings but not about none existing beings it would be wrong to kill animals with positive utility. And since if animals can have utility it would obviously be wrong to breed them and make their life miserable so that they have negative utility, this would mean that we could not kill animals

I find the arguments against killing animals to be far weaker, since I do not follow the particular form of utilitarism that supports them and since I cannot really explain why the features I named under 1 should forbid me to kill animals. In addition to that I count the fact that Peter Singer, who is against all killing of animals and is arguably a pretty clever person has found no better way to justify his statement, that one should not kill animals at all, than the idea that this will lead us to continue to objectify them and ignore their pain. Since Singer has found no better reason and he probably spent a lot of time doing it, it is likely that there is none.

Although I am fairly confident, that killing Animals is in line with my ethical believes I still see some trouble. If I am wrong on this this might be an incredible harmful decision, since it will lead to the death of many animals (probably hundreds of them, if I don’t reduce my meat consumption for other reasons). Therefore I have to be incredibly confident that I have not overlooked something in order to continue to eat meat. And I have limited time and probably a strong motivation to come to the conclusion that meat eating is okay, which clouds my judgement. I feel that I need more evidence. As far as I know there are lots of meat eaters here and some of them will have thought about this. Why are you so confident that animal life’s do not matter? Is it that I overlooked major arguments or is the self-consciousness just a more of a knock down argument than I think?

Animals and Pain

It is relatively well established that animals show reactions that one could associate with pain and they have a nerve system that allows pain. Singer has proclaimed that in his 1975 book Animal Liberation for mammals and birds and cited research on it, and as far as I know no one has really corrected him on that. I also found papers that claim the same for fish and lobsters and I have not found any counterevidence. So the question that remains is, do animals get negative utility from pain, and do they have utility functions at all.

Eliezer Argues in this post https://m.facebook.com/yudkowsky/posts/10152588738904228 that they don’t have utility. I can understand his model, but I could also imagine that an animal mind works in other ways. I am no expert in evolutionary biology, but as far as I know, the mainstream opinion among scientists right now is that animals have pain.

There is for Example the Cambridge declaration of conciousness (http://fcmconference.org/img/CambridgeDeclarationOnConsciousness.pdf). It might have a different understanding of the word consciousness compared to the one which I think is most popular among the lesswrong community (Consciousness as being aware of its own existence), but it clearly states that animals have affective states and therefor utility. If animals can suffer pain, than factory farming is incredibly wrong. I would therefore have to be very certain (surely above 99% confidence) of the fact that they don’t or I cannot justify to eat factory farmed meat. The question is: How can I be so sure if a significant amount of experts are of a different opinion. Does anyone have any actual research on the topic that explains the reasons why animals do not have utility in more detail than Eliezer did? Basically I would need something that not only explains why this is a plausible hypothesis but something that explain why they could not possibly have evolved in a way that they feel pain. So basically, why a pig that feels pain makes no sense from an evolutionary perspective.

If my current believes don’t shift anymore I will stop eating factory farmed meat, but not stop to eat any meat at all. I would be happy about any additional evidence, or about oppinions on the conclusions I draw from my evidence.

[Cross-posted from FB]

I've got an economic question that I'm not sure how to answer.

I've been thinking about trends in AI development, and trying to get a better idea of what we should expect progress to look like going forward.

One important question is: how much do existing AI systems help with research and the development of new, more capable AI systems?

The obvious answer is, "not much." But I think of AI systems as being on a continuum from calculators on up. Surely AI researchers sometimes have to do arithmetic and other tasks that they already outsource to computers. I expect that going forward, the share of tasks that AI researchers outsource to computers will (gradually) increase. And I'd like to be able to draw a trend line. (If there's some point in the future when we can expect most of the work of AI R&D to be automated, that would be very interesting to know about!)

So I'd like to be able to measure the share of AI R&D done by computers vs humans. I'm not sure of the best way to measure this. You could try to come up with a list of tasks that AI researchers perform and just count, but you might run into trouble as the list of tasks to changes over time (e.g. suppose at some point designing an AI system requires solving a bunch of integrals, and that with some later AI architecture this is no longer necessary).

What seems more promising is to abstract over the specific tasks that computers vs human researchers perform and use some aggregate measure, such as the total amount of energy consumed by the computers or the human brains, or the share of an R&D budget spent on computing infrastructure and operation vs human labor. Intuitively, if most of the resources are going towards computation, one might conclude that computers are doing most of the work.

Unfortunately I don't think that intuition is correct. Suppose AI researchers use computers to perform task X at cost C_x1, and some technological improvement enables X to be performed more cheaply at cost C_x2. Then, all else equal, the share of resources going towards computers will decrease, even though their share of tasks has stayed the same.

On the other hand, suppose there's some task Y that the researchers themselves perform at cost H_y, and some technological improvement enables task Y to be performed more cheaply at cost C_y. After the team outsources Y to computers the share of resources going towards computers has gone up. So it seems like it could go either way -- in some cases technological improvements will lead to the share of resources spent on computers going down and in some cases it will lead to the share of resources spent on computers going up.

So here's the econ part -- is there some standard economic analysis I can use here? If both machines and human labor are used in some process, and the machines are becoming both more cost effective and more capable, is there anything I can say about how the expected share of resources going to pay for the machines changes over time?

This will be of interest mainly to EA-friendly LWs, and is cross-posted on the EA Forum, The Life You Can Save, and Intentional Insights

The Life You Can Save has an excellent tool to help people easily visualize and quantify the impact of their giving: the Impact Calculator. It enables people to put in any amount of money they want, then click on a charity, and see how much of an impact their money can have. It's a really easy way to promote effective giving to non-EAs, but even EAs who didn't see it before can benefit. I certainly did, when I first played around with it. So I wrote a blog post, copy-pasted below, for The Life You Can Save and for Intentional Insights, to help people learn about the Impact Calculator. If you like the blog, please share this link to The Life You Can Save blog, as opposed to this post. Any feedback on the blog post itself is welcomed!

__________________________________________________________________________

How a Calculator Helped Me Multiply My Giving

It feels great to see hope light up in the eyes of a beggar in the street as you stop to look at them when others pass them by without a glance. Their faces widen in a smile as you reach into your pocket and take out your wallet. "Thank you so much" is such a heartwarming phrase to hear from them as you pull out five bucks and put the money in the hat in front of them. You walk away with your heart beaming as you imagine them getting a nice warm meal at McDonalds due to your generosity.

In risk modeling, there is a well-known distinction between aleatory and epistemic uncertainty, which is sometimes referred to, or thought of, as irreducible versus reducible uncertainty. Epistemic uncertainty exists in our map; as Eliezer put it, “The Bayesian says, ‘Uncertainty exists in the map, not in the territory.’” Aleatory uncertainty, however, exists in the territory. (Well, at least according to our map that uses quantum mechanics, according to Bells Theorem – like, say, the time at which a radioactive atom decays.) This is what people call quantum uncertainty, indeterminism, true randomness, or recently (and somewhat confusingly to myself) ontological randomness – referring to the fact that our ontology allows randomness, not that the ontology itself is in any way random. It may be better, in Lesswrong terms, to think of uncertainty versus randomness – while being aware that the wider world refers to both as uncertainty. But does the distinction matter?

To clarify a key point, many facts are treated as random, such as dice rolls, are actually mostly uncertain – in that with enough physics modeling and inputs, we could predict them. On the other hand, in chaotic systems, there is the possibility that the “true” quantum randomness can propagate upwards into macro-level uncertainty. For example, a sphere of highly refined and shaped uranium that is *exactly* at the critical mass will set off a nuclear chain reaction, or not, based on the quantum physics of whether the neutrons from one of the first set of decays sets off a chain reaction – after enough of them decay, it will be reduced beyond the critical mass, and become increasingly unlikely to set off a nuclear chain reaction. Of course, the question of whether the nuclear sphere is above or below the critical mass (given its geometry, etc.) can be a difficult to measure uncertainty, but it’s not aleatory – though some part of the question of whether it kills the guy trying to measure whether it’s just above or just below the critical mass will be random – so maybe it’s not worth finding out. And that brings me to the key point.

In a large class of risk problems, there are factors treated as aleatory – but they may be epistemic, just at a level where finding the “true” factors and outcomes is prohibitively expensive. Potentially, the timing of an earthquake that would happen at some point in the future could be determined exactly via a simulation of the relevant data. Why is it considered aleatory by most risk analysts? Well, doing it might require a destructive, currently technologically impossible deconstruction of the entire earth – making the earthquake irrelevant. We would start with measurement of the position, density, and stress of each relatively macroscopic structure, and the perform a very large physics simulation of the earth as it had existed beforehand. (We have lots of silicon from deconstructing the earth, so I’ll just assume we can now build a big enough computer to simulate this.) Of course, this is not worthwhile – but doing so would potentially show that the actual aleatory uncertainty involved is negligible. Or it could show that we need to model the macroscopically chaotic system to such a high fidelity that microscopic, fundamentally indeterminate factors actually matter – and it was truly aleatory uncertainty. (So we have epistemic uncertainty about whether it’s aleatory; if our map was of high enough fidelity, and was computable, we would know.)

It turns out that most of the time, for the types of problems being discussed, this distinction is irrelevant. If we know that the value of information to determine whether something is aleatory or epistemic is negative, we can treat the uncertainty as randomness. (And usually, we can figure this out via a quick order of magnitude calculation; Value of Perfect information is estimated to be worth $100 to figure out which side the dice lands on in this game, and building and testing / validating any model for predicting it would take me at least 10 hours, my time is worth at least $25/hour, it’s negative.) But sometimes, slightly improved models, and slightly better data, are feasible – and then worth checking whether there is some epistemic uncertainty that we can pay to reduce. In fact, for earthquakes, we’re doing that – we have monitoring systems that can give several minutes of warning, and geological models that can predict to some degree of accuracy the relative likelihood of different sized quakes.

So, in conclusion; most uncertainty is lack of resolution in our map, which we can call epistemic uncertainty. This is true even if lots of people call it “truly random” or irreducibly uncertain – or if they are fancy, aleatory uncertainty. Some of what we assume is uncertainty is really randomness. But lots of the epistemic uncertainty can be safely treated as aleatory randomness, and value of information is what actually makes a difference. And knowing the terminology used elsewhere can be helpful.

I was just reading an article online, and one of the comments mentioned a political issue (the legality of corporate contributions to political campaigns). One of the responses what a comment saying "Not until we abandon this mentality, we the victims are the majority, we can take back this country, all we need to do is open our eyes and stand up." When I saw this comment, I agreed with the sentiment - but nevertheless, I shrugged and moved on. Sure, it is an issue that I strongly believe in, and an issue on which I thought most people would agree with me - but nevertheless, there was nothing I could do about it. Sure, if everyone who agreed on this took a stand (or at least wrote a letter to their congressional representative) we could probably do something about it together - but I could only control my own actions, and in acting alone I'd only be wasting my time.

That got me thinking. This isn't the first time I've come across these sorts of issues. At its heart, this is a coordination problem - lots of people want to do something, but it doesn't make sense for any individual to act unless many others do as well. We don't have a way to solve these sorts of problems, which is quite unfortunate. Except... why can't we have such a system?

Right now, I'm imagining a website where you get to create "causes" and also add your name to them along with a number specifying how many other supporters you'd need to see before you would be willing to take (a pre-specified) action towards the cause. What are the reasons that something like this wouldn't work?

I fact, we do have several websites that work sort-of like this already. Kickstarter is one. The White House Petitions system is another. The first of these has been a wild success; the second, less so (as far as I understand it). So there is clearly some merit to the idea, but also some major setbacks. 

What do people think of this?

If you’ve been planning to get around to maybe thinking about Effective Altruism, now is a great time to get consider making a commitment. As part of Giving What We Can's pledge campaign, people are signing the Giving What We Can pledge - to donate 10% of their future income to the charities they believe will do the most good in the world. It is based on the belief that we can make a real difference by thoroughly assessing evidence and contributing some of our resources to address the most pressing global concerns. The pledge is not legally binding, but is a public declaration of a lasting commitment to the cause. For anyone not ready to make the full commitment to taking the pledge, people are also signing up to 'Try Giving' as part of the campaign - where you commit to donate an amount for a finite time.

Last year in a similar event over 80 people took the pledge, which resulted in almost $19,000,000 being pledged to effective charities. To give you an idea of what this could achieve, a recent GiveWell estimate suggests that, if donated today to the Against Malaria Foundation, this amount could be expected to buy and distribute about 3.5 million bednets and avert the loss of almost 6700 lives (though there is much uncertainty around these figures).

If you think the campaign is a good idea and you'd like more people to hear about it, it would be a great help if you invited anyone you think would be interested in the event; also if you supported the campaign on Thunderclap. If you'd like to help out even more, then join our pledge event organisation Facebook group.

Any questions about the pledge, the campaign, or anything related are more than welcome.
 
About Giving What We Can: GWWC is a meta-charity which researches and evaluates charities on the basis of the impact they have, and also a community with GWWC chapters across the world. It is part of the Centre for Effective Altruism and was co-founded by a LessWronger.

This is part of a semi-monthly reading group on Eliezer Yudkowsky's ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.

Welcome to the Rationality reading group. This fortnight we discuss The World: An Introduction (pp. 834-839) and Part O: Lawful Truth (pp. 843-883). This post summarizes each article of the sequence, linking to the original LessWrong post where available.

O. Lawful Truth

The World: An Introduction

181. Universal Fire - You can't change just one thing in the world and expect the rest to continue working as before.

182. Universal Law - In our everyday lives, we are accustomed to rules with exceptions, but the basic laws of the universe apply everywhere without exception. Apparent violations exist only in our models, not in reality.

183. Is Reality Ugly? - There are three reasons why a world governed by math can still seem messy. First, we may not actually know the math. Secondly, even if we do know all of the math, we may not have enough computing power to do the full calculation. And finally, even if we did know all the math, and we could compute it, we still don't know where in the mathematical system we are living.

184. Beautiful Probability - Bayesians expect probability theory, and rationality itself, to be math. Self-consistent, neat, even beautiful. This is why Bayesians think that Cox's theorems are so important.

185. Outside the Laboratory - Those who understand the map/territory distinction will integrate their knowledge, as they see the evidence that reality is a single unified process.

186. The Second Law of Thermodynamics, and Engines of Cognition - To form accurate beliefs about something, you really do have to observe it. It's a very physical, very real process: any rational mind does "work" in the thermodynamic sense, not just the sense of mental effort. Engines of cognition are not so different from heat engines, though they manipulate entropy in a more subtle form than burning gasoline. So unless you can tell me which specific step in your argument violates the laws of physics by giving you true knowledge of the unseen, don't expect me to believe that a big, elaborate clever argument can do it either.

187. Perpetual Motion Beliefs - People learn under the traditional school regimen that the teacher tells you certain things, and you must believe them and recite them back; but if a mere student suggests a belief, you do not have to obey it. They map the domain of belief onto the domain of authority, and think that a certain belief is like an order that must be obeyed, but a probabilistic belief is like a mere suggestion. And when half-trained or tenth-trained rationalists abandon their art and try to believe without evidence just this once, they often build vast edifices of justification, confusing themselves just enough to conceal the magical steps. It can be quite a pain to nail down where the magic occurs - their structure of argument tends to morph and squirm away as you interrogate them. But there's always some step where a tiny probability turns into a large one - where they try to believe without evidence - where they step into the unknown, thinking, "No one can prove me wrong".

188. Searching for Bayes-Structure - If a mind is arriving at true beliefs, and we assume that the second law of thermodynamics has not been violated, that mind must be doing something at least vaguely Bayesian - at least one process with a sort-of Bayesian structure somewhere - or it couldn't possibly work.

This has been a collection of notes on the assigned sequence for this fortnight. The most important part of the reading group though is discussion, which is in the comments section. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

The next reading will cover Part P: Reductionism 101 (pp. 887-935). The discussion will go live on Wednesday, 16 December 2015, right here on the discussion forum of LessWrong.

Most discussion of uploading talks of uploading from the inside out: simply, a biological person undergoes a disruptive procedure which digitises their mind. The digital mind then continues the person’s timeline as a digital existence, with all that entails.

The thing that stands out here is the disruptive nature of the process from biological to digital being. It is not only a huge step to undergo such a transformation, but few things in reality operate in such binary terms. More commonly, things happen gradually.

Being an entrepreneur and also having a keen interest in the future, I both respect audacious visions, and study how they come to be realised. Very rarely does progress come from someone investing a bunch of resources in a black-box process that ends in a world-changing breakthrough. Much more commonly, massive innovations are realised through a process of iteration and exploration, fueled by a need that motivates people to solve thousands of problems, big and small. Massive trends interact with other innovations to open up opportunities that when exploited cause a further acceleration of innovation. Every successful startup and technology, from Facebook to Tesla and from mobile phones to modern medicine can be understood in these terms.

With this lens in mind, how might uploading be realised? This is one potential timeline, barring AI explosion or existential catastrophy.

It is perhaps useful to explore the notion of “above/below the API”. A slew of companies have formed, often called “Uber for X” or “AirBnB for Y”, solving needs we have, through a computer system, such as a laptop or a mobile phone app. The app might issue a call to a server via an API, and that server may delegate the task to some other system, often powered by other humans. The original issuer of the command then gets their need covered, minimising direct contact with other humans, the traditional way of having our needs covered. It is crucial to understand that API-mediated interactions win because they are superior to their traditional alternative. Once they were possible, it was only natural for them to proliferate. As an example, compare the experience of taking a taxi with using Uber.

And so computer systems are inserted between human-to-human interactions. This post is composed on a computer, through which I will publish it in a digital location, where it might be seen by others. If I am to hear their response to it, it will also be mediated by APIs. Whenever a successful new API is launched, fortunes are made and lost. An entire industry, venture capital, exists to fund efforts to bring new APIs into existence, each new API making life easier for its users than what came before, and adding additional API layers.

As APIs flood interpersonal space, humans gain superpowers. Presence is less and less important, and a person anywhere in the connected world can communicate and effect change anywhere else. And with APIs comes control of personal space and time. Personal safety increases both by decreasing random physical contact and by always being connected to others who can send help if something goes wrong. The demand for connectivity and computation is driving networking everywhere, and the cost of hardware to fall through the floor.

Given the trends that are in motion, what’s next? Well, if computer-mediated experience is increasing, it might grow to the point where every interaction a human has with the world around them will be mediated by computers. If this sounds absurd, think of noise-cancelling headphones. Many of us now use them not to listen to music, but to block the sound from our environment. Or consider augmented reality. If the visual field, the data pipeline of the brain, can be used to provide critical, or entertaining, context about the physical environment, who would want to forego it? Consider biofeedback: if it’s easy to know at all times what is happening within our bodies and prevent things from going wrong, who wouldn’t want to? It’s not a question of whether these needs exist, but of when technology will be able to cover them.

Once most interaction is API-mediated, the digital world switches from opt-in to opt-out. It’s not a matter of turning the laptop on, but of turning it off for a while, perhaps to enjoy a walk in nature, or for a repair. But wouldn’t you want to bring your augmented reality goggles that can tell you the story of each tree, and ensure you’re not exposed to any pathogens as you wander in the biological jungle? As new generations grow up in a computer-mediated world, fewer and fewer excursions into the offline will happen. Technology, after all, is what was invented after you were born. Few of us consider hunting and gathering our food or living in caves to be a romantic return to the past. When we take a step backward, perhaps to signal virtue, like foregoing vaccination or buying locally grown food, we make sure our move will not deprive us of the benefits of the modern world.

Somewhere around the time when APIs close the loop around us or even before then, the human body will begin to be modified. Artificial limbs that are either plainly superior to their biological counterparts, or better adapted to that world will make sense, and brain-computer interfaces (whether direct or via the existing senses) will become ever more permanent. As our bodies are replaced with mechanical parts, the brain will come next. Perhaps certain simple parts will be easy to replace with more durable, better performing ones. Intelligence enhancement will finally be possible by adding processing power natural selection alone could never have evolved. Gradually, step by small step, the last critical biological components will be removed, as a final cutting of the cord with the physical world.

Humans will have digitised themselves, not by inventing a machine that takes flesh as input and outputs ones and zeroes, not by cyberpunk pioneers jumping into an empty digital world to populate it. We will have done it by making incremental choices, each one a sound rational decision that was in hindsight inevitable, incorporating inventions that made sense, and in the end it will be unclear when the critical step was made. We will have uploaded ourselves simply in the course of everyday life.

The First Step

The first step on the path to truth is superstition.  We all start there, and should acknowledge that we start there.

Superstition is, contrary to our immediate feelings about the word, the first stage of understanding.  Superstition is the attribution of unrelated events to a common (generally unknown or unspecified) cause - it could be called pattern recognition. The "supernatural" component generally included in the definition is superfluous, because supernatural merely refers to that which isn't part of nature - which means reality -, which is an elaborate way of saying something whose relationship to nature is not yet understood, or else nonexistent.  If we discovered that ghosts are real, and identified an explanation - overlapping entities in a many-worlds universe, say - they'd cease to be supernatural and merely be natural.

Just as the supernatural refers to unexplained or imaginary phenomena, superstition refers to unexplained or imaginary relationships, without the necessity of cause.  If you designed an AI in a game which, after five rounds of being killed whenever it went into rooms with green-colored walls, started avoiding rooms with green-colored walls, you've developed a good AI.  It is engaging in superstition, it has developed an incorrect understanding of the issue.  But it hasn't gone down the wrong path - there is no wrong path in understanding, there is only the mistake of stopping.  Superstition, like all belief, is only useful if you're willing to discard it.

The Next Step

Incorrect understanding is the first - and necessary - step to correct understanding.  It is, indeed, every step towards correct understanding.  Correct understanding is a path, not an achievement, and it is pursued, not by arriving at the correct conclusion in the first place, but by testing your ideas and discarding those which are incorrect.

No matter how much intelligent you are, you cannot skip the "incorrect understanding" step of knowledge, because that is every step of knowledge.  You must come up with wrong ideas in order to get at the right ones - which will always be one step further.  You must test your ideas.  And again, the only mistake is stopping, in assuming that you have it right now.

Intelligence is never your bottleneck.  The ability to think faster isn't necessarily the ability to arrive at the right answer faster, because the right answer requires many wrong ones, and more importantly, identifying which answers are indeed wrong, which is the slow part of the process.

Better answers are arrived at by the process of invalidating wrong answers.

The Winding Path

The process of becoming Less Wrong is the process of being, in the first place, wrong.  It is the state of realizing that you're almost certainly incorrect about everything - but working on getting incrementally closer to an unachievable "correct".  It is a state of anti-hubris, and requires a delicate balance between the idea that one can be closer to the truth, and the idea that one cannot actually achieve it.

The art of rationality is the art of walking this narrow path.  If ever you think you have the truth - discard that hubris, for three steps from here you'll see it for superstition, and if you cannot see that, you cannot progress, and there your search for truth will end.  That is the path of the faithful.

But worse, the path is not merely narrow, but winding, with frequent dead ends requiring frequent backtracking.  If ever you think you're closer to the truth - discard that hubris, for it may inhibit you from leaving a dead end, and there your search for truth will end.  That is the path of the crank.

The path of rationality is winding and directionless.  It may head towards beauty, then towards ugliness; towards simplicity, then complexity.  The correct direction isn't the aesthetic one; those who head towards beauty may create great art, but do not find truth.  Those who head towards simplicity might open new mathematical doors and find great and useful things inside - but they don't find truth, either.  Truth is its own path, found only by discarding what is wrong.  It passes through simplicity, it passes through ugliness; it passes through complexity, and also beauty.  It doesn't belong to any one of these things.

The path of rationality is a path without destination.

Written as an experiment in the aesthetic of Less Wrong.  I'd appreciate feedback into the aesthetic interpretation of Less Wrong, rather than the sense of deep wisdom emanating from it (unless the deep wisdom damages the aesthetic).

I've made very extensive notes, along with my assessment, of Daniel Kahneman's Thinking, Fast and Slow, and have passed it around to aspiring rationalist friends who found my notes very useful. So I though I would share these with the Less Wrong community by creating a Less Wrong Wiki article with these notes. Feel free to optimize the article based on your own notes as well. Hope this proves as helpful to you as it did to those others whom I shared my notes with.

What are your intuitions about this? It has direct implications for whether the Simulation Argument is sound.

Imagine two rooms, A and B. Between times t1 and t2, 100 trillion people sojourn in room A while 100 billion sojourn in room B. At any given moment, though, exactly 1 person occupies room A while 1,000 people occupy room B. At t2, you find yourself in a room, but you don't know which one. If you have to place a bet on which room it is (at t2), what do you say? Do you consider the time-slice or the history of room occupants? How do you place your bet?

If you bet that you're in room B, then the Simulation Argument may be flawed: there could be a fourth disjunct that Bostrom misses, namely that we become a posthuman civilization that runs a huge number of simulations yet we don't have reason for believing that we're stimulants.

Thoughts?

(Happy to get feedback on this! It draws from and expounds ideas in this article: http://jetpress.org/v26.2/torres.htm)

Consider a seemingly simple question: if the means were available, who exactly would destroy the world? There is surprisingly little discussion of this question within the nascent field of existential risk studies. But it’s an absolutely crucial issue: what sort of agent would either intentionally or accidentally cause an existential catastrophe?

The first step forward is to distinguish between two senses of an existential risk. Nick Bostrom originally defined the term as: “One where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.” It follows that there are two distinct scenarios, one endurable and the other terminal, that could realize an existential risk. We can call the former an extinction risk and the latter a stagnation risk. The importance of this distinction with respect to both advanced technologies and destructive agents has been previously underappreciated.

So, the question asked above is actually two questions in disguise. Let’s consider each in turn.

Terror: Extinction Risks

First, the categories of agents who might intentionally cause an extinction catastrophe are fewer and smaller than one might think. They include:

(1) Idiosyncratic actors. These are malicious agents who are motivated by idiosyncratic beliefs and/or desires. There are instances of deranged individuals who have simply wanted to kill as many people as possible and then die, such as some school shooters. Idiosyncratic actors are especially worrisome because this category could have a large number of members (token agents). Indeed, the psychologist Martha Stout estimates that about 4 percent of the human population suffers from sociopathy, resulting in about 296 million sociopaths. While not all sociopaths are violent, a disproportionate number of criminals and dictators have (or very likely have) had the condition.

(2) Future ecoterrorists. As the effects of climate change and biodiversity loss (resulting in the sixth mass extinction) become increasingly conspicuous, and as destructive technologies become more powerful, some terrorism scholars have speculated that ecoterrorists could become a major agential risk in the future. The fact is that the climate is changing and the biosphere is wilting, and human activity is almost entirely responsible. It follows that some radical environmentalists in the future could attempt to use technology to cause human extinction, thereby “solving” the environmental crisis. So, we have some reason to believe that this category could become populated with a growing number of token agents in the coming decades.

(3) Negative utilitarians. Those who hold this view believe that the ultimate aim of moral conduct is to minimize misery, or “disutility.” Although some negative utilitarians like David Pearce see existential risks as highly undesirable, others would welcome annihilation because it would entail the elimination of suffering. It follows that if a “strong” negative utilitarian had a button in front of her that, if pressed, would cause human extinction (say, without causing pain), she would very likely press it. Indeed, on her view, doing this would be the morally right action. Fortunately, this version of negative utilitarianism is not a position that many non-academics tend to hold, and even among academic philosophers it is not especially widespread.

(4) Extraterrestrials. Perhaps we are not alone in the universe. Even if the probability of life arising on an Earth-analog is low, the vast number of exoplanets suggests that the probability of life arising somewhere may be quite high. If an alien species were advanced enough to traverse the cosmos and reach Earth, it would very likely have the technological means to destroy humanity. As Stephen Hawking once remarked, “If aliens visit us, the outcome would be much as when Columbus landed in America, which didn’t turn out well for the Native Americans.”

(5) Superintelligence. The reason Homo sapiens is the dominant species on our planet is due almost entirely to our intelligence. It follows that if something were to exceed our intelligence, our fate would become inextricably bound up with its will. This is worrisome because recent research shows that even slight misalignments between our values and those motivating a superintelligence could have existentially catastrophic consequences. But figuring out how to upload human values into a machine poses formidable problems — not to mention the issue of figuring out what our values are in the first place.

Making matters worse, a superintelligence could process information at about 1 million times faster than our brains, meaning that a minute of time for us would equal approximately 2 years in time for the superintelligence. This would immediately give the superintelligence a profound strategic advantage over us. And if it were able to modify its own code, it could potentially bring about an exponential intelligence explosion, resulting in a mind that’s many orders of magnitude smarter than any human. Thus, we may have only one chance to get everything just right: there’s no turning back once an intelligence explosion is ignited.

A superintelligence could cause human extinction for a number of reasons. For example, we might simply be in its way. Few humans worry much if an ant genocide results from building a new house or road. Or the superintelligence could destroy humanity because we happen to be made out of something it could use for other purposes: atoms. Since a superintelligence need not resemble human intelligence in any way — thus, scholars tell us to resist the dual urges of anthropomorphizing and anthropopathizing — it could be motivated by goals that appear to us as utterly irrational, bizarre, or completely inexplicable.

Terror: Stagnation Risks

Now consider the agents who might intentionally try to bring about a scenario that would result in a stagnation catastrophe. This list subsumes most of the list above in that it includes idiosyncratic actors, future ecoterrorists, and superintelligence, but it probably excludes negative utilitarians, since stagnation (as understood above) would likely induce more suffering than the status quo today. The case of extraterrestrials is unclear, given that we can infer almost nothing about an interstellar civilization except that it would be technologically sophisticated.

For example, an idiosyncratic actor could harbor not a death wish for humanity, but a “destruction wish” for civilization. Thus, she or he could strive to destroy civilization without necessarily causing the annihilation of Homo sapiens. Similarly, a future ecoterrorist could hope for humanity to return to the hunter-gatherer lifestyle. This is precisely what motivated Ted Kaczynski: he didn’t want everyone to die, but he did want our technological civilization to crumble. And finally, a superintelligence whose values are misaligned with ours could modify Earth in such a way that our lineage persists, but our prospects for future development are permanently compromised. Other stagnation scenarios could involve the following categories:

(6) Apocalyptic terrorists. History is overflowing with groups that not only believed the world was about to end, but saw themselves as active participants in an apocalyptic narrative that’s unfolding in realtime. Many of these groups have been driven by the conviction that “the world must be destroyed to be saved,” although some have turned their activism inward and advocated mass suicide.

Interestingly, no notable historical group has combined both the genocidal and suicidal urges. This is why apocalypticists pose a greater stagnation terror risk than extinction risk: indeed, many see their group’s survival beyond Armageddon as integral to the end-times, or eschatological, beliefs they accept. There are almost certainly less than about 2 million active apocalyptic believers in the world today, although emerging environmental, demographic, and societal conditions could cause this number to significantly increase in the future, as I’ve outlined in detail elsewhere (see Section 5 of this paper).

(7) States. Like terrorists motivated by political rather than transcendent goals, states tend to place a high value on their continued survival. It follows that states are unlikely to intentionally cause a human extinction event. But rogue states could induce a stagnation catastrophe. For example, if North Korea were to overcome the world’s superpowers through a sudden preemptive attack and implement a one-world government, the result could be an irreversible decline in our quality of life.

So, there are numerous categories of agents that could attempt to bring about an existential catastrophe. And there appear to be fewer agent types who would specifically try to cause human extinction than to merely dismantle civilization.

Error: Extinction and Stagnation Risks

There are some reasons, though, for thinking that error (rather than terror) could constitute the most significant threat in the future. First, almost every agent capable of causing intentional harm would also be capable of causing accidental harm, whether this results in extinction or stagnation. For example, an apocalyptic cult that wants to bring about Armageddon by releasing a deadly biological agent in a major city could, while preparing for this terrorist act, inadvertently contaminate its environment, leading to a global pandemic.

The same goes for idiosyncratic agents, ecoterrorists, negative utilitarians, states, and perhaps even extraterrestrials. (Indeed, the large disease burden of Europeans was a primary reason Native American populations were decimated. By analogy, perhaps an extraterrestrial destroys humanity by introducing a new type of pathogen that quickly wipes us out.) The case of superintelligence is unclear, since the relationship between intelligence and error-proneness has not been adequately studied.

Second, if powerful future technologies become widely accessible, then virtually everyone could become a potential cause of existential catastrophe, even those with absolutely no inclination toward violence. To illustrate the point, imagine a perfectly peaceful world in which not a single individual has malicious intentions. Further imagine that everyone has access to a doomsday button on her or his phone; if pushed, this button would cause an existential catastrophe. Even under ideal societal conditions (everyone is perfectly “moral”), how long could we expect to survive before someone’s finger slips and the doomsday button gets pressed?

Statistically speaking, a world populated by only 1 billion people would almost certainly self-destruct within a 10-year period if the probability of any individual accidentally pressing a doomsday button were a mere 0.00001 percent per decade. Or, alternatively: if only 500 people in the world were to gain access to a doomsday button, and if each of these individuals had a 1 percent chance of accidentally pushing the button per decade, humanity would have a meager 0.6 percent chance of surviving beyond 10 years. Thus, even if the likelihood of mistakes is infinitesimally small, planetary doom will be virtually guaranteed for sufficiently large populations.

The Two Worlds Thought Experiment

The good news is that a focus on agential risks, as I’ve called them, and not just the technological tools that agents might use to cause a catastrophe, suggests additional ways to mitigate existential risk. Consider the following thought-experiment: a possible world A contains thousands of advanced weapons that, if in the wrong hands, could cause the population of A to go extinct. In contrast, a possible world B contains only a single advanced “weapon of total destruction” (WTD). Which world is more dangerous? The answer is obviously world A.

But it would be foolishly premature to end the analysis here. Imagine further that A is populated by compassionate, peace-loving individuals, whereas B is overrun by war-mongering psychopaths. Now which world appears more likely to experience an existential catastrophe? The correct answer is, I would argue, world B.

In other words: agents matter as much as, or perhaps even more than, WTDs. One simply can’t evaluate the degree of risk in a situation without taking into account the various agents who could become coupled to potentially destructive artifacts. And this leads to the crucial point: as soon as agents enter the picture, we have another variable that could be manipulated through targeted interventions to reduce the overall probability of an existential catastrophe.

The options here are numerous and growing. One possibility would involve using “moral bioenhancement” techniques to reduce the threat of terror, given that acts of terror are immoral. But a morally enhanced individual might not be less likely to make a mistake. Thus, we could attempt to use cognitive enhancements to lower the probability of catastrophic errors, on the (tentative) assumption that greater intelligence correlates with fewer blunders.

Furthermore, implementing stricter regulations on CO2 emissions could decrease the probability of extreme ecoterrorism and/or apocalyptic terrorism, since environmental degradation is a “trigger” for both.

Another possibility, most relevant to idiosyncratic agents, is to reduce the prevalence of bullying (including cyberbullying). This is motivated by studies showing that many school shooters have been bullied, and that without this stimulus such individuals would have been less likely to carry out violent rampages. Advanced mind-reading or surveillance technologies could also enable law enforcement to identify perpetrators before mass casualty crimes are committed.

As for superintelligence, efforts to solve the “control problem” and create a friendly AI are of primary concern among many many researchers today. If successful, a friendly AI could itself constitute a powerful mitigation strategy for virtually all the categories listed above.

(Note: these strategies should be explicitly distinguished from proposals that target the relevant tools rather than agents. For example, Bostrom’s idea of “differential technological development” aims to neutralize the bad uses of technology by strategically ordering the development of different kinds of technology. Similarly, the idea of police “blue goo” to counter “grey goo” is a technology-based strategy. Space colonization is also a tool intervention because it would effectively reduce the power (or capacity) of technologies to affect the entire human or posthuman population.)

Agent-Tool Couplings

Devising novel interventions and understanding how to maximize the efficacy of known strategies requires a careful look at the unique properties of the agents mentioned above. Without an understanding of such properties, this important task will be otiose. We should also prioritize different agential risks based on the likely membership (token agents) of each category. For example, the number of idiosyncratic agents might exceed the number of ecoterrorists in the future, since ecoterrorism is focused on a single issue, whereas idiosyncratic agents could be motivated by a wide range of potential grievances.[1] We should also take seriously the formidable threat posed by error, which could be nontrivially greater than that posed by terror, as the back-of-the-envelope calculations above show.

Such considerations, in combination with technology-based risk mitigation strategies, could lead to a comprehensive, systematic framework for strategically intervening on both sides of the agent-tool coupling. But this will require the field of existential risk studies to become less technocentric than it currently is.

[1] Although, on the other hand, the stimulus of environmental degradation would be experienced by virtually everyone in society, whereas the stimuli that motivate idiosyncratic agents might be situationally unique. It’s precisely issues like these that deserve further scholarly research.

This is great, by using small interlocking magnetic fields, you can keep the water in a higher vibrational state, allowing a "super-cooling" without getting crystallization and cell rupture

Subzero 12-hour Nonfreezing Cryopreservation of Porcine Heart in a Variable Magnetic Field

"invented a special refrigerator, termed as the Cells Alive System (CAS; ABI Co. Ltd., Chiba, Japan). Through the application of a combination of multiple weak energy sources, this refrigerator generates a special variable magnetic field that causes water molecules to oscillate, thus inhibiting crystallization during ice formation18 (Figure 1). Because the entire material is frozen without the movement of water molecules, cells can be maintained intact and free of membranous damage. This refrigerator has the ability to achieve a nonfreezing state even below the solidifying point."

http://mobile.journals.lww.com/transplantationdirect/_layouts/15/oaks.journals.mobile/articleviewer.aspx?year=2015&issue=10000&article=00005#ath

This is the monthly thread for posting media of various types that you've found that you enjoy. Post what you're reading, listening to, watching, and your opinion of it. Post recommendations to blogs. Post whatever media you feel like discussing! To see previous recommendations, check out the older threads.

Rules:

• Please avoid downvoting recommendations just because you don't personally like the recommended material; remember that liking is a two-place word. If you can point out a specific flaw in a person's recommendation, consider posting a comment to that effect.
• If you want to post something that (you know) has been recommended before, but have another recommendation to add, please link to the original, so that the reader has both recommendations.
• Please post only under one of the already created subthreads, and never directly under the parent media thread.
• Use the "Other Media" thread if you believe the piece of media you want to discuss doesn't fit under any of the established categories.
• Use the "Meta" thread if you want to discuss about the monthly media thread itself (e.g. to propose adding/removing/splitting/merging subthreads, or to discuss the type of content properly belonging to each subthread) or for any other question or issue you may have about the thread or the rules.

Hi all,

Mostly lurker, I very rarely post, mostly  just read the excellent posts here.

I'm a Filipino, which means I am a citizen of the Republic of the Philippines.  My annual salary, before taxes, is about $20,000 (USA dollars).  I work at an IC development company (12 years at this company), developing the logic parts of LCD display drivers.  My understanding is that the median US salary for this kind of job is about $80,000 -> $100,000 a year.  This is a fucking worthless third world country, so the government eats up about ~30% of my salary and converts it to lousy service, rich government officials, bad roadworks, long commute times, and a (tiny) chance of being falsely accused of involvement in the drug trade and shot without trial.  Thus my take-home pay amounts to about $15,000 a year.  China is also murmuring vague threats about war because of the South China Sea (which the local intelligentsia insist on calling the West Philippine Sea); as we all know, the best way to survive a war is not be in one.

This has lead to my deep dissatisfaction with my current job.

I'm also a programmer as a hobby, and have been programming for 23 years (I started at 10 years old on Atari LOGO; I know a bunch of languages from low-level X86 assembly to C to C++ to ECMAScript to Haskell, and am co-author of SRFI-105 and SRFI-110).  My understanding is that a USA programmer would *start* at the $20,000-a-year level (?), and that someone with experience can probably get twice that, and a senior one can get $100,000/year.

As we all know, once a third world citizen starts having first world skill level, he starts demanding first world renumeration also.

I've been offered a senior software developer job at a software company, offering approximately $22,000/year; because of various attempts at tax reform it offers a flat 15% income tax, so I can expect about $18,000/year take home pay.  I've turned it down with a heavy heart, because seriously, $22,000/year at 15% tax for a senior software developer?

Leaving my current job is something I've been planning on doing, and I intend to do so early next year.  The increasing stress (constant overtime, management responsibilities (I'm a tech geek with passable social skills, and exercising my social skills drains me), 1.5-hour commutes) and the low renumeration makes me want to consider my alternate options.

My options are:

1.  Get myself to the USA, Europe, or other first-world country somehow, and look for a job there.  High risk, high reward, much higher probability of surviving to the singularity (can get cryonics there, can't get it here).  Complications: I have a family: a wife, a 4-year-old daughter, and a son on the way.  My wife wants to be near me, so it's difficult to live for long apart.  I have no work visa for any first-world country.  I'm from a third-world country that is sometimes put on terrorist watch lists, and prejudice is always high in first-world countries.

2.  Do freelance programming work.  Closer to free market ideal, so presumably I can get nearer to the USA levels of renumeration.  Lets me stay with my family.  Complications: I need to handle a lot of the human resources work myself (healthcare provider, social security, tax computations, time and task management - the last is something I do now in my current job position, but I dislike it).

3.  Become a landowning farmer.  My paternal grandparents have quite a few parcels of land (some of which have been transferred to my father, who is willing to pass it on to me), admittedly somewhere in the boondocks of the provinces of this country, but as any Georgian knows, landowners can sit in a corner staring at the sky, blocking the occasional land reform bill, and earn money.  Complications: I have no idea about farming.  I'd actually love to advocate a land value tax, which would undercut my position as a landowner.

For now, my basic current plan is some combination of #2 and #3 above: go sit in a corner of our clan's land and do freelance programming work.  This keeps me with my family, may reduce my level of stress, may increase my renumeration to nearer the USA levels.

My current job has a retirement pay, and since I've worked for 12 years, I've already triggered it, and they'll give me about $16,000 or so when I leave.  This seems reasonably comfortable to live on (note that this is what I take home in a year, and I've supported a family on that, remember this is a lousy third-world country).

Is my basic plan sound?  I'm trying to become more optimal, which seems to me to point me away from my current job and towards either #1 or #2, with #3 as a fallback.  I'd love to get cryonics and will start to convince my wife of its sensibility if I had a chance to actually get it, but that will require me either leaving the country (option #1 above) or running a cryonics company in a third-world country myself.

--

I got introduced to Less Wrong when I first read on Reddit about some weirdo who was betting he could pretend he was a computer in a box and convince someone to let him out of the box, and started lurking on Overcoming Bias.  When that weirdo moved over to Less Wrong, I followed and lurked there also.  So here I am ^^.  I'm probably very atypical even for Less Wrong; I highly suspect I am the only Filipino here (I'll have to check the diaspora survey results in detail).

Looking back, my big mistake was being arrogant and thinking "meh, I already know programming, so I should go for a challenge, why don't I take up electronics engineering instead because I don't know about it" back when I was choosing a college course.  Now I'm an IC developer.  Two of my cousins (who I can beat the pants off in a programming task) went with software engineering and pull in more money than I do.  Still, maybe I can correct that, even if it's over a decade late.  I really need to apply more of what I learn on Less Wrong.

Some years ago I applied for a CFAR class, but couldn't afford it, sigh.  Even today it's a few month's worth of salary for me.  So I guess I'll just have to settle for Less Wrong and Rationality from AI to Zombies.

Time start: 20:48:35

I

The feelings of wonder, awe, amazement. It's a very human experience, and it is processed in the brain as a type of pleasure. If fact, if we look at the number of "5 photos you wouldn't believe" and similar clickbait on the Internet, it functions as a mildly addictive drug.

If I proposed that there is something wrong with those feelings, I would soon be drowned in voices of critique, pointing out that I'm suggesting we all become straw Vulcans, and that there is nothing wrong with subjective pleasure obtained cheaply and at no harm to anyone else.

I do not disagree with that. However, caution is required here, if one cares about epistemic purity of belief. Let's look at why.

II

Stories are supposed to be more memorable. Do you like stories? I'm sure you do. So consider a character, let's call him Jim.

Jim is very interested in technology and computers, and he is checking news sites every day when he comes to work in the morning. Also, Jim has read a number of articles on LessWrong, including the one about noticing confusion.

He cares about improving his thinking, so when he first read about the idea of noticing confusion on a 5 second level, he thought he wants to apply it in his life. He had a few successes, and while it's not perfect, he feels he is on the right track to notice having wrong models of the world more often.

A few days later, he opens his favorite news feed at work, and there he sees the following headline:

"AlphaGo wins 4-1 against Lee Sedol"

He goes on to read the article, and finds himself quite elated after he learns the details. 'It's amazing that this happened so soon! And most experts apparently thought it would happen in more than a decade, hah! Marvelous!'

Jim feels pride and wonder at the achievement of Google DeepMind engineers... and it is his human right to feel it, I guess.

But is Jim forgetting something?

III

Yes, I know that you know. Jim is feeling amazed, but... has he forgotten the lesson about noticing confusion?

There is a significant obstacle to Jim applying his "noticing confusion" in the situation described above: his internal experience has very little to do with feelings of confusion.

His world in this moment is dominated with awe, admiration etc., and those feelings are pleasant. It is not at all obvious that this inner experience corresponds to a innacurate model of the world he had before.

Even worse - improving his model's predictive power would result in less pleasant experiences of wonder and amazement in the future! (Or would it?) So if Jim decides to update, he is basically robbing himself of the pleasures of life, that are rightfully his. (Or is he?)

Time end: 21:09:50

(Speedwriting stats: 23 wpm, 128 cpm, previous: 30/167, 33/183)

Volunteer/Intern Position: Media Engagement on Global Catastrophic Risk

http://gcrinstitute.org/volunteerintern-position-media-engagement-on-global-catastrophic-risk/

The Global Catastrophic Risk Institute (GCRI) seeks a volunteer/intern to contribute on the topic of media engagement on global catastrophic risk, which is the risk of events that could harm or destroy global human civilization. The work would include two parts: (1) analysis of existing media coverage of global catastrophic risk and (2) formulation of strategy for media engagement by GCRI and our colleagues. The intern may also have opportunities to get involved in other aspects of GCRI.

All aspects of global catastrophic risk would be covered. Emphasis would be placed on GCRI’s areas of focus, including nuclear war and artificial intelligence. Additional emphasis could be placed on topics of personal interest to the intern, potentially including (but not limited to) climate change, other global environmental threats, pandemics, biotechnology risks, asteroid collision, etc.

The ideal candidate is a student or early-career professional seeking a career at the intersection of global catastrophic risk and the media. Career directions could include journalism, public relations, advertising, or academic research in related social science disciplines. Candidates seeking other career directions would also be considered, especially if they see value in media experience. However, we have a strong preference for candidates intending a career on global catastrophic risk.

The position is unpaid. The intern would receive opportunities for professional development, networking, and publication. GCRI is keen to see the intern benefit professionally from this position and will work with the intern to ensure that this happens. This is not a menial labor activity, but instead is one that offers many opportunities for enrichment.

A commitment of at least 10 hours per month is expected. Preference will be given to candidates able to make a larger time commitment. The position will begin during August-September 2016. The position will run for three months and may be extended pending satisfactory performance.

The position has no geographic constraint. The intern can work from anywhere in the world. GCRI has some preference for candidates from American time zones, but we regularly work with people from around the world. GCRI cannot provide any relocation assistance.

Candidates from underrepresented demographic groups are especially encouraged to apply.

Applications will be considered on an ongoing basis until 30 September, 2016.

To apply, please send the following to Robert de Neufville (robert [at] gcrinstitute.org):

* A cover letter introducing yourself and explaining your interest in the position. Please include a description of your intended career direction and how it would benefit from media experience on global catastrophic risk. Please also describe the time commitment you would be able to make.

* A resume or curriculum vitae.

* A writing sample (optional).

I just thought I'd clarify the difference between learning values and learning knowledge. There are some more complex posts about the specific problems with learning values, but here I'll just clarify why there is a problem with learning values in the first place.

Consider the term "chocolate bar". Defining that concept crisply would be extremely difficult. But nevertheless it's a useful concept. An AI that interacted with humanity would probably learn that concept to a sufficient degree of detail. Sufficient to know what we meant when we asked it for "chocolate bars". Learning knowledge tends to be accurate.

Contrast this with the situation where the AI is programmed to "create chocolate bars", but with the definition of "chocolate bar" left underspecified, for it to learn. Now it is motivated by something else than accuracy. Before, knowing exactly what a "chocolate bar" was would have been solely to its advantage. But now it must act on its definition, so it has cause to modify the definition, to make these "chocolate bars" easier to create. This is basically the same as Goodhart's law - by making a definition part of a target, it will no longer remain an impartial definition.

What will likely happen is that the AI will have a concept of "chocolate bar", that it created itself, especially for ease of accomplishing its goals ("a chocolate bar is any collection of more than one atom, in any combinations"), and a second concept, "Schocolate bar" that it will use to internally designate genuine chocolate bars (which will still be useful for it to do). When we programmed it to "create chocolate bars, here's an incomplete definition D", what we really did was program it to find the easiest thing to create that is compatible with D, and designate them "chocolate bars".

This is the general counter to arguments like "if the AI is so smart, why would it do stuff we didn't mean?" and "why don't we just make it understand natural language and give it instructions in English?"

Content warning: a couple LWers apparently think that the concept of ego depletion—also known as willpower depletion—is a memetic hazard, though I find it helpful. Also, the material presented here won't fit everyone's experiences.

What happens if we assume that the idea of ego depletion is basically correct, and try to draw an analogy between thermodynamics and willpower?

Figure 1. Thermodynamics Picture

You probably remember seeing something like the above diagram in a chemistry class. The diagram shows how unstable, or how high in energy, the states that a material can pass through in a chemical reaction are. Here's what the abbreviations mean:

• SM is the starting material.
• TS1 and TS2 are the two transition states, which must be passed through to go from SM to EM1 or EM2.
• EM1 and EM2 are the two possible end materials.

The valleys of both curves represent configurations a material may occupy at the start or end of a chemical reaction. Lower energy valleys are more stable. However, higher peaks can only be reliably crossed if energy is available from e.g. the temperature being sufficiently high.

The main takeaway from Figure 1 is that reactions which produce the most stable end materials, like ending material 2, from a given set of starting materials aren't always the reactions which are easiest to make happen.

Figure 2. Willpower Picture

We can draw a similar diagram to illustrate how much stress we lose while completing a relaxing activity. Here's what the abbreviations used in Figure 2 mean:

• SM is your starting mood.
• TS is your state of topmost stress, which depends on which activity you choose.
• EM1 and EM2 are your two possible ending moods.

Above, the valley on the left represents how stressed you are before starting one of two possible relaxing activities. The peak in the middle represents how stressed you'll be when attempting to get the activity underway, and the valley on the right represents how stressed you'll be once you're done.

For the sake of simplification, let's say that stress is the opposite of willpower, such that losing stress means you gain willpower, and vice versa. For many people, there's a point at which it's very hard to take on additional stress or use more willpower, such that getting started on an activity that would normally get you to ending mood 2 from an already stressed starting mood is very hard. 

In chemistry, if you want to make end material 2 instead of end material 1, you have to make sure that you have some way of getting over the big peak at transition state 2—such as by making sure the temperature is high enough. In real life, it's also good to have a plan for getting over the big peak at the point of topmost stress. Spending time or attention figuring what your ending mood 2-producing activities are may also be worthwhile.

Some leisure activities, like browsing the front page of reddit, are ending mood 1-producing activities; they're easy to start, but not very rewarding. Examples of what qualifies as an ending mood 2-producing activity vary between people—but reading books, writing, hiking, meditating, or making games or art qualify as ending mood 2-producing activities for some.

At a minimum, making sure that you end up in a high willpower, low stress ending mood requires paying attention to your ability to handle stress and conserve willpower. Sometimes this implies that taking a break before you really need to means that you'll get more out of your break. Sometimes it means that you should monitor how many spoons and forks you have. In general, though, preferring ending mood 2-producing activities over ending mood 1-producing activities will give you the best results in the long run.

The best-case scenario is that you find a way to automatically turn impulses to do ending mood 1-producing activities into impulses to do ending mood 2-producing activities, such as with the trigger action plan [open Reddit -> move hands into position to do a 5-minute meditation].

“Identity” here refers to the question “will my copy be me, and if yes, on which conditions?” It results in several paradoxes which I will not repeat here, hoping that they are known to the reader.

Identity is one of the most complex problems, like safe AI or aging. It only appears be simple. It is complex because it has to answer the question: “Who is who?” in the universe, that is to create a trajectory in the space of all possible minds, connecting identical or continuous observer-moments. But such a trajectory would be of the same complexity as all space of possible minds, and that is very complex.

There have been several attempts to dismiss the complexity of the identity problem, like open individualism (I am everybody) or zero-individualism (I exist only now). But they do not prevent the existence of “practical identity” which I use when planning my tomorrow or when I am afraid of future pain.

The identity problem is also very important. If we (or AI) arrive at an incorrect solution, we will end up being replaced by p-zombies or just copies-which-are-not-me during a “great uploading”. It will be a very subtle end of the world.

The identity problem is also equivalent to the immortality problem. if I am able to describe “what is me”, I would know what I need to save forever. This has practical importance now, as I am collecting data for my digital immortality (I even created a startup about it and the map will be my main contribution to it. If I solve the identity problem I will be able to sell the solution as a service http://motherboard.vice.com/read/this-transhumanist-records-everything-around-him-so-his-mind-will-live-forever)

So we need to know how much and what kind of information I should preserve in order to be resurrected by future AI. What information is enough to create a copy of me? And is information enough at all?

Moreover, the identity problem (IP) may be equivalent to the benevolent AI problem, because the first problem is, in a nutshell, “What is me” and the second is “What is good for me”. Regardless, the IP requires a solution of consciousness problem, and AI problem (that is solving the nature of intelligence) are somewhat similar topics.

I wrote 100+ pages trying to solve the IP, and became lost in the ocean of ideas. So I decided to use something like the AIXI method of problem solving: I will list all possible solutions, even the most crazy ones, and then assess them.

The following map is connected with several other maps: the map of p-zombies, the plan of future research into the identity problem, and the map of copies. http://lesswrong.com/lw/nsz/the_map_of_pzombies/

The map is based on idea that each definition of identity is also a definition of Self, and it is also strongly connected with one philosophical world view (for example, dualism). Each definition of identity answers a question “what is identical to what”. Each definition also provides its own answers to the copy problem as well as to its own definition of death - which is just the end of identity – and also presents its own idea of how to reach immortality.

So on the horizontal axis we have classes of solutions:

“Self" definition - corresponding identity definition - philosophical reality theory - criteria and question of identity - death and immortality definitions.

On the vertical axis are presented various theories of Self and identity from the most popular on the upper level to the less popular described below:

1) The group of theories which claim that a copy is not original, because some kind of non informational identity substrate exists. Different substrates: same atoms, qualia, soul or - most popular - continuity of consciousness. All of them require that the physicalism will be false. But some instruments for preserving identity could be built. For example we could preserve the same atoms or preserve the continuity of consciousness of some process like the fire of a candle. But no valid arguments exist for any of these theories. In Parfit’s terms it is a numerical identity (being the same person). It answers the question “What I will experience in the next moment of time"

2) The group of theories which claim that a copy is original, if it is informationally the same. This is the main question about the required amount of information for the identity. Some theories obviously require too much information, like the positions of all atoms in the body to be the same, and other theories obviously do not require enough information, like the DNA and the name.

3) The group of theories which see identity as a social phenomenon. My identity is defined by my location and by the ability of others to recognise me as me.

4) The group of theories which connect my identity with my ability to make plans for future actions. Identity is a meaningful is part of a decision theory.

5)  Indirect definitions of self. This a group of theories which define something with which self is strongly connected, but which is not self. It is a biological brain, space-time continuity, atoms, cells or complexity. In this situation we say that we don’t know what constitutes identity but we could know with what it is directly connected and could preserve it.

6) Identity as a sum of all its attributes, including name, documents, and recognition by other people. It is close to Leibniz’s definition of identity. Basically, it is a duck test: if it looks like a duck, swims like a duck, and quacks like a duck, then it is probably a duck. 

7) Human identity is something very different to identity of other things or possible minds, as humans have evolved to have an idea of identity, self-image, the ability to distinguish their own identity and the identity of others, and to predict its identity. So it is a complex adaptation which consists of many parts, and even if some parts are missed, they could be restored using other parts. 

There also a problem of legal identity and responsibility. 

8)  Self-determination. “Self” controls identity, creating its own criteria of identity and declaring its nature. The main idea here is that the conscious mind can redefine its identity in the most useful way. It also includes the idea that self and identity evolve during differing stages of personal human evolution. 

9) Identity is meaningless. The popularity of this subset of ideas is growing. Zero-identity and  open identity both belong to this subset. The main contra-argument here is that if we cut the idea of identity, future planning will be impossible and we will have to return to some kind of identity through the back door. The idea of identity comes also with the idea of the values of individuality. If we are replaceable like ants in an anthill, there are no identity problems. There is also no problem with murder.

The following is a series of even less popular theories of identity, some of them I just constructed ad hoc.

10)  Self is a subset of all thinking beings. We could see a space of all possible minds as divided into subsets, and call them separate personalities.

11)  Non-binary definitions of identity.

The idea that me or not-me identity solutions are too simple and result in all logical problems. if we define identity continuously, as a digit of the interval (0,1), we will get rid of some paradoxes and thus be able to calculate the identity level of similarity or time until the given next stage could be used as such a measure. Even a complex digit can be used if we include informational and continuous identity (in a Parfit meaning).

12) Negative definitions of identity: we could try to say what is not me.

13) Identity as overlapping observer-moments.

14) Identity as a field of indexical uncertainty, that is a group of observers to which I belong, but can’t know which one I am.

15) Conservative approach to identity. As we don’t know what identity is we should try to save as much as possible, and risk our identity only if it is the only means of survival. That means no copy/paste transportation to Mars for pleasure, but yes if it is the only chance to survive (this is my own position).

16)  Identity as individuality, i.e. uniqueness. If individuality doesn’t exist or doesn’t have any value, identity is not important.

17) Identity as a result of the ability to distinguish different people. Identity here is a property of perception.

18) Mathematical identity. Identity may be presented as a number sequence, where each number describes a full state of mind. Useful toy model.

19) Infinite identity. The main idea here is that any mind has the non-zero probability of becoming any other mind after a series of transformations. So only one identity exists in all the space of all possible minds, but the expected time for me to become a given person is dramatically different in the case of future me (1 day) and a random person (10 to the power of 100 years). This theory also needs a special version of quantum immortality which resets “memories” of a dying being to zero, resulting in something like reincarnation, or an infinitely repeating universe in the style of Nietzsche's eternal recurrence.  

20) Identity in a multilevel simulation. As we probably live in a simulation, there is a chance that it is multiplayer game in which one gamer has several avatars and can constantly have experiences through all of them. It is like one eye through several people.

21) Splitting identity. This is an idea that future identity could split into several (or infinitely many) streams. If we live in a quantum multiverse we split every second without any (perceived) problems. We are also adapted to have several future copies if we think about “me-tomorrow” and “me-the-day-after-tomorrow”.

This list shows only groups of identity definitions, many more smaller ideas are included in the map.

The only rational choice I see is a conservative approach, acknowledging that we don’t know the nature of identity and trying to save as much as possible of each situation in order to preserve identity.

The pdf: http://immortality-roadmap.com/identityeng8.pdf

If it's worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should start on Monday, and end on Sunday.

4. Unflag the two options "Notify me of new top level comments on this article" and "Make this post available under..." before submitting.

Rationality quotes are self-explanatory.  Irrationality quotes often need some context and explication, so they would break the flow in Rationality Quotes.

I always had the informal impression that the optimal policies were deterministic (choosing the best option, rather than some mix of options). Of course, this is not the case when facing other agents, but I had the impression this would hold when facing the environment rather that other players.

But stochastic policies can also be needed if the environment is partially observable, at least if the policy is Markov (memoryless). Consider the following POMDP (partially observable Markov decision process):

There are two states, 1a and 1b, and the agent cannot tell which one they're in. Action A in state 1a and B in state 1b, gives a reward of -R and keeps the agent in the same place. Action B in state 1a and A in state 1b, gives a reward of R and moves the agent to the other state.

The returns for the two deterministic policies - A and B - are -R every turn except maybe for the first. While the return for the stochastic policy of 0.5A + 0.5B is 0 per turn.

Of course, if the agent can observe the reward, the environment is no longer partially observable (though we can imagine the reward is delayed until later). And the general policy of "alternate A and B" is more effective that the 0.5A + 0.5B policy. Still, that stochastic policy is the best of the memoryless policies available in this POMDP.

Time: 22:56:47

I

This is going to be an exercise in speed writing a LW post.

Not writing posts at all seems to be worse than writing poorly edited posts.

It is currently hard for me to do anything that even resembles actual speed writing: even as I type this sentence, I have a very hard to resist urge to check it for grammar mistakes and make small corrections/improvements before I've even finished typing.

But to reduce the burden of writing, I predict it is going to be highly useful to develop the ability of actually writing a post as fast as I can type, without going back.

If this proves to have acceptable results, you can expect more regular posts from me in the future.

And possibly, if I develop the habit of writing regularly, I'll finally get to describing some of the topics on which I have (what I believe are) original and sizable clusters of knowledge, which is not easily available somewhere else.

But for now, just some thoughts on a very particular aspect of modelling how human brains think about a very particular thing.

This thing is immense suffering.

Time: 23:03:18

(Still slow!)

II

You might have heard this or similar from someone, possibly more than once in your life:

"you have no idea how I feel!"

or

"you can't even imagine how I feel!"

For me, this kind of phrase has always had the ring of a challenge. I have a potent imagination, and non-negligible experience in the affairs of humans. Therefore, I am certainly able to imagine how you feel, am I not?

Not so fast.

(Note added later: as Gram_Stone mentions, these kinds of statements tend to be used in epistemically unsound arguments, and as such can be presumed to be suspicious; however here, I am more concerned with the fact of the matter of how imagination works.)

Let's back up a little bit and recount some simple observations about imagining numbers.

You might be able to imagine and hold the image of five, six, nine, or even sixteen apples in your mind.

If I tell you to imagine something more complex, like pointed arrows arranged in a circle, you might be able to imagine four, or six, or maybe even eight of them.

If your brain is constructed differently from mine, you might easily go higher with the numbers.

But at some fairly small number, your mental machinery simply no longer has the capacity to imagine more shapes.

III

However, if I tell you that "you can't even imagine 35 apples!" it is obviously not an insult or a challenge, and what is more:

"imagining 35 apples" is NOT EQUAL to "comprehending in every detail what 35 apples are"

I.e. depending on how good your knowledge of natural numbers is, that is to say, if you passed the first class of primary school, you can analyse the situation of "35 apples" in every possible way, and imagine it partially - but not all of it at the same time.

Directly imagining apples is very similar to actually experiencing apples in your life, but it has a severe limitation.

You can experience 35 apples in your life, but you can't imagine all of them at once even if you saw them 3 seconds ago.

Meta: I think I'm getting better at not stopping when I write.

Time: 23:13:00

IV

But, you ask, what is the point of writing all this obvious stuff about apples?

Well, if you move to more emotionally charged topics, like someone's emotions, it is much harder to think about the situation in a clear way.

And if you have a clear model of how your brain processes this information, you might be able to respond in a more effective way.

In particular, you might be saved from feeling guilty or inadequate about not being able to imagine someone's feelings or suffering.

It is a simple fact about your brain that it has a limited capability to imagine emotion.

And especially with suffering, the amount of suffering you are able to experience IS OF A COMPLETELY DIFFERENT ORDER OF MAGNITUDE than the amount you are able to imagine, even with the best intentions and knowledge.

However, can you comprehend it?

V

From this model, it is also immediately obvious that the same thing happens when you think about your own suffering in the past.

We know generally that humans can't remember their emotions very well, and their memories don't correlate very well with reported experience-in-the-moment.

Based on my personal experience, I'll tentatively make some bolder claims.

If you have suffered a tremendous amount, and then enough time has passed to "get over it", your brain is not only unable to imagine how much you have suffered in the past:

it is also unable to comprehend the amount of suffering.

Yes, even if it's your own suffering.

And what is more, I propose that the exact mechanism of "getting over something" is more or less EQUIVALENT to losing the ability to comprehend that suffering.

The same would (I expect) hold in case of getting better after severe PTSD etc.

VI

So in this sense, a person telling you "you cannot even imagine how I feel" is right also with a less literal interpretation of their statement.

If you are a mentally healthy individual, not suffering any major traumas etc., I suggest your brain literally has a defense mechanism (that protects your precious mental health) that makes it impossible for you to not only imagine, but also fully comprehend the amounts of suffering you are being told about.

Time: 23:28:04

Publish!

TL;DR: Many models of the future exist. Several are relevant. Hyperbolic model is strongest, but too strange.

Our need: correct model of the future
Different people: different models = no communication.

Assumptions:
Model of the future = main driving force of historical process + graphic of changes
Model of the future determines global risks

The map: lists all main future models.
Structure: from fast growth – to slow growth models.
Pfd: http://immortality-roadmap.com/futuremodelseng.pdf