Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: lukeprog 23 July 2014 05:01:40AM 4 points [-]

Good work!

Not sure if you're planning to make further clarifications to the visualizations and the post, but one suggestion would be to introduce a new arrow (or arrows) showing that multipolar scenarios may very well resolve into a unipolar outcome after not much time (decades or centuries). This provides one major justification for the book's focus on singleton scenarios, another justification being that singleton scenarios are easier to analyze.

Comment author: Eliezer_Yudkowsky 19 June 2014 11:42:31PM 7 points [-]

So there's a nice analogy to MIRI's work, where we're trying to figure out what an AGI would look like if it was built from the ground up to get the strongest safety guarantees possible for such an autonomous and capable system.

Except we're not; we're trying to get adequate guarantees which is much harder.

The main image reason I object to "safe AI" is the image it implies of, "Oh, well, AIs might be dangerous because, you know, AIs are naturally dangerous for some mysterious reason, so instead you have to build a class of AIs that can never harm people because they have the First Law of Robotics, and then we're safe."

Which is just not at all what the technical research program is about.

Which isn't at all what the bigger picture looks like. The vast majority of self-improving agents have utility functions indifferent to your existence; they do not hate you, nor do they love you, and you are made of atoms they can use for something else. If you don't want that to happen you need to build, from the ground up, an AI that has something so close to your normalized / idealized utility function as to avert all perverse instantiation pathways.

There isn't a small class of "threat" pathways that you patch, or a conscience module that you install, and then you're left with an AI that's like the previous AI but safe, like a safe paperclip maximizer that doesn't harm humans. That's not what's happening here.

It sounds like you're nervous about some unspecified kind of bad behavior from AIs, like someone nervous in an unspecified way about, oh, say, genetically modified foods, and then you want "safe foods" instead, or you want to slap some kind of wacky crackpot behavior-limiter on the AI so it can never threaten you in this mysterious way you worry about.

Which brings us to the other image problem: you're using a technophobic codeword, "safe".

Imagine somebody advocating for "safe nuclear power plants, instead of the nuclear plants we have now".

If you're from a power plant company the anti-nuclear advocates are like, "Nice try, but we know that no matter what kind of clever valve you're putting on the plant, it's not really safe." Even the pro-nuclear people would quietly grit their teeth and swallow their words, because they know, but cannot say, that this safety is not perfect. I can't imagine Bruce Schneier getting behind any cryptographic initiative that was called "safe computing"; everyone in the field knows better, and in that field they're allowed to say so.

If you're not from a power plant company---which we're not, in the metaphor---if you look more like some kind of person making a bunch of noise about social interests, then the pro-nuclear types who see the entire global warming problem as being caused by anti-nuclear idiots giving us all these coal-burning plants, think that you're trying to call your thing "safe" to make our on-the-whole good modern nuclear power plants sound "unsafe" by contrast, and that you'll never be satisfied until everything is being done your way.

Most of our supporters come from technophilic backgrounds. The fundamental image that a technophile has of a technophobe / neo-Luddite is that when a technophobe talks about "safety" their real agenda is to demand unreasonable levels of safety, to keep raising the bar until the technology is driven to near-extinction, all in the name of "safety". They're aware of how they lost the fight for nukes. They're aware that "You're endangering the children!" is a memetic superweapon, and they regard anyone who resorts to "You're endangering the children!" as a defector against their standards of epistemic hygiene. You know how so many people think that MIRI is arguing that we ought to do these crazy expensive measures because if there's even a chance that AI is dangerous, we ought to do these things? even though I've repeatedly repudiated that kind of reasoning at every possible juncture? It's because they've been primed to expect attack with a particular memetic superweapon.

When you say "Safe AI", that's what a technophile thinks you're preparing to do---preparing to demand expensive, unnecessary measures and assert your own status over real scientists, using a "You're endangering the children!" argument that requires unlimited spending on tiny risks. They've seen it over, and over, and over again; they've seen it with GMOs and nuclear weapons and the FDA regulating drug development out of existence.

"Safety" is a word used by their enemies that means "You must spend infinite money on infinitesimal risks." Again, this is the fight they've seen the forces of science and sanity lose, over and over again.

Take that phenomenon, combined with the fact that what we want is not remotely like a conscience module slapped onto exogenously originating magical threat-risks from otherwise okay AIs, combined with people knowing perfectly well that your innovations do not make AI truly perfectly safe. Then "safe AI" does not sound like a good name to me. Talking about how we want the "best possible" "guarantee" is worse.

"Friendly AI" is there to just not sound like anything, more or less, and if we want to replace it with a more technical-sounding term, it should perhaps also not sound like anything. Maybe we can go back to Greek or Latin roots.

Failing that, "high-assurance AI" at least sounds more like what we actually do than "safe AI". It doesn't convey the concept that low-assurance AIs automatically kill you with probability ~1, but at least you're not using a codeword that people know from anti-GMO campaigns, and at least the corresponding research process someone visualizes sounds a bit more like what we actually do (having to design things from scratch to support certain guarantees, rather than slapping a safety module onto something that already exists).

Comment author: lukeprog 10 July 2014 05:53:24PM *  3 points [-]

After thinking and talking about it more, I still think "AGI safety" is the best term I've got so far. Or, "AI safety," in contexts where we don't mind being less specific, and are speaking to an audience that doesn't know what "AGI" means.

Basically, (1) I think your objections to "safe AGI" mostly don't hold for "AGI safety," and (2) I think the audience you seem most concerned about (technophiles) isn't the right audience to be most concerned about.

Maybe Schneier wouldn't get behind something called "safe computing" or "secure computing," but he happily works in a field called "computer security." The latter phrasing suggests the idea that we can get some degree of security (or safety) even though we can never make systems 100% safe or secure. Scientists don't object to people working on "computer security," and I haven't seen technophiles object to it either. Heck, many of them work in computer security. "X security" and "X safety" don't imply to anyone I know that "you must spend infinite money on infinitesimal risks." It just implies you're trying to provide some reasonable level of safety and security, and people like that. Technophiles want their autonomous car to be reasonably safe just like everyone else does.

I think your worry that "safety" implies there's a small class of threat pathways that need to be patched, rather than implying that an AGI needs to be designed from the ground up to stably optimize for your idealized values, is more of a concern. But it's a small concern. A term like "Friendly AI" is a non-starter for many smart and/or influential people, whereas "AGI safety" serves as a rung in Wittgeinstein's ladder from which you can go on to explain that the challenge of AGI safety is not to patch a small class of threat pathways but instead to build a system from the ground to ensure desirable behavior.

(Here again, the analogy to other safety-critical autonomous systems is strong. Such systems are often, like FAI, built from the ground up for safety and/or security precisely because in such autonomous systems there isn't a small class of threat pathways. Instead, almost all possible designs you might come up with don't do what you intended in some system states or environments. See e.g. my interviews with Michael Fisher and Benjamin Pierce. But that's not something even most computer scientists will know anything about — it's an approach to AI safety work that would have to be explained after they've already got a foot on the "AGI safety" rung of the expository ladder.)

Moreover, you seem to be most worried about how our terminology will play to the technophile audience. But playing well to technophiles isn't MIRI's current or likely future bottleneck. Attracting brilliant researchers is. If we can attract brilliant researchers, funding (from technophiles and others) won't be so hard. But it's hard to attract brilliant researchers with a whimsical home-brewed term like "Friendly AI" (especially when it's paired with other red flags like a shockingly-arrogant-for-academia tone and an apparent lack of familiarity with related work, but that's a different issue).

As Toby reports, it's also hard to get the ear of policy-makers with a term like "Friendly AI," but I know you are less interested in reaching policy-makers than I am.

Anyway, naming things is hard, and I certainly don't fault you (or was it Bostrom?) for picking "Friendly AI" back in the day, but from our current vantage point we can see better alternatives. Even LWers think so, and I'd expect them to be more sympathetic to "Friendly AI" than anyone else.

Comment author: lukeprog 30 June 2014 05:51:48PM *  0 points [-]

In addition to the links provided by others, I also recommend looking at Paul's new report: "Non-omniscience, probabilistic inference, and metamathematics."

Edit: Oops, this was already linked from this comment.

Comment author: eli_sennesh 25 June 2014 08:54:58AM 2 points [-]

When did you start talking to formal verification researchers?

Because, you know, I totally have not wanted to do a PhD under any of these guys, or anything like that...

Comment author: lukeprog 25 June 2014 03:28:52PM 5 points [-]

When did you start talking to formal verification researchers?

Many months ago. Subscribe to the MIRI blog! :)

Comment author: lukeprog 24 June 2014 11:55:32PM 0 points [-]

From Osnos' Age of Ambition:

I lived in China for eight years, and I watched this age of ambition take shape. Above all, it is a time of plenty— the crest of a transformation one hundred times the scale, and ten times the speed, of the first Industrial Revolution, which created modern Britain. The Chinese people no longer want for food— the average citizen eats six times as much meat as in 1976— but this is a ravenous era of a different kind, a period when people have awoken with a hunger for new sensations, ideas, and respect. China is the world’s largest consumer of energy, movies, beer, and platinum; it is building more high-speed railroads and airports than the rest of the world combined.

And:

In the event that these censorship efforts failed, the Party was testing a weapon of last resort: the OFF switch. On July 5, 2009, members of China’s Muslim Uighur minority in the far western city of Urumqi protested police handling of a brawl between Hans and Uighurs. The protests turned violent, and nearly two hundred people died, most of them Han, who had been targeted for their ethnicity. Revenge attacks on Uighur neighborhoods followed, and in an effort to prevent people from communicating and organizing, the government abruptly disabled text messages, cut long-distance phone lines, and shut off Internet access almost entirely. The digital blackout lasted ten months, and the economic effects were dramatic: exports from Xinjiang, the Uighur autonomous region, plummeted more than 44 percent. But the Party was willing to accept immense economic damage to smother what it considered a political threat. In the event of a broader crisis someday, China probably has too many channels in and out to impose so complete a blackout on a national scale, but even a limited version would have a profound effect.

And:

On October 8, 2010, ten months after Liu Xiaobo was convicted, the Nobel Committee awarded him the Peace Prize “for his long and non-violent struggle for fundamental human rights.” He was the first Chinese citizen to receive the award, not counting the Dalai Lama, who had lived for decades in exile. The award to Liu Xiaobo drove Chinese leaders into a rage; the government denounced Liu’s award as a “desecration” of Alfred Nobel’s legacy. For years, China had coveted a Nobel Prize as a validation of the nation’s progress and a measure of the world’s acceptance. The obsession with the prize was so intense that scholars had named it the “Nobel complex,” and each fall they debated China’s odds of winning it, like sports fans in a pennant race. There was once a television debate called “How Far Are We from a Nobel Prize?”

When the award was announced, most Chinese people had never heard of Liu, so the state media made the first impression; it splashed an article across the country reporting that he earned his living “bad-mouthing his own country.” The profile was a classic of the form: it described him as a collector of fine wines and porcelain, and it portrayed him telling fellow prisoners, “I’m not like you. I don’t lack for money. Foreigners pay me every year, even when I’m in prison.” Liu “spared no effort in working for Western anti-China forces” and, in doing so, “crossed the line of freedom of speech into crime.”

For activists, the news of the award was staggering. “Many broke down in tears, even uncontrollable sobbing,” one said later. In Beijing, bloggers, lawyers, and scholars gathered in the back of a restaurant to celebrate, but police arrived and detained twenty of them. When the announcement was made, Han Han, on his blog, toyed with censors and readers; he posted nothing but a pair of quotation marks enclosing an empty space. The post drew one and a half million hits and more than 28,000 comments.

Comment author: lukeprog 25 June 2014 12:05:16AM 0 points [-]

More (#2) from Osnos' Age of Ambition:

In all, authorities executed at least fourteen yuan billionaires in the span of eight years, on charges ranging from pyramid schemes to murder for hire. (Yuan Baojing, a former stockbroker who made three billion yuan before his fortieth birthday, was convicted of arranging the killing of a man who tried to blackmail him.) The annual rich list was nicknamed the “death list.”

And:

As the Party’s monopoly on information gave way, so did its moral credibility. For people such as the philosophy student Tang Jie, the pursuit of truth did not satisfy their skepticism; it led them to deeper questions about who they wanted to be and whom they wanted to believe. In the summer of 2012, people noticed that another search word had been blocked. The anniversary of the Tiananmen Square demonstrations had just passed, and people had been discussing it, in code, by calling it “the truth”— zhenxiang. The censors picked up on this, and when people searched Weibo for anything further, they began receiving a warning: “In accordance with relevant laws, regulations, and policies, search results for ‘the truth’ have not been displayed.”

Comment author: lukeprog 24 June 2014 11:55:32PM 0 points [-]

From Osnos' Age of Ambition:

I lived in China for eight years, and I watched this age of ambition take shape. Above all, it is a time of plenty— the crest of a transformation one hundred times the scale, and ten times the speed, of the first Industrial Revolution, which created modern Britain. The Chinese people no longer want for food— the average citizen eats six times as much meat as in 1976— but this is a ravenous era of a different kind, a period when people have awoken with a hunger for new sensations, ideas, and respect. China is the world’s largest consumer of energy, movies, beer, and platinum; it is building more high-speed railroads and airports than the rest of the world combined.

And:

In the event that these censorship efforts failed, the Party was testing a weapon of last resort: the OFF switch. On July 5, 2009, members of China’s Muslim Uighur minority in the far western city of Urumqi protested police handling of a brawl between Hans and Uighurs. The protests turned violent, and nearly two hundred people died, most of them Han, who had been targeted for their ethnicity. Revenge attacks on Uighur neighborhoods followed, and in an effort to prevent people from communicating and organizing, the government abruptly disabled text messages, cut long-distance phone lines, and shut off Internet access almost entirely. The digital blackout lasted ten months, and the economic effects were dramatic: exports from Xinjiang, the Uighur autonomous region, plummeted more than 44 percent. But the Party was willing to accept immense economic damage to smother what it considered a political threat. In the event of a broader crisis someday, China probably has too many channels in and out to impose so complete a blackout on a national scale, but even a limited version would have a profound effect.

And:

On October 8, 2010, ten months after Liu Xiaobo was convicted, the Nobel Committee awarded him the Peace Prize “for his long and non-violent struggle for fundamental human rights.” He was the first Chinese citizen to receive the award, not counting the Dalai Lama, who had lived for decades in exile. The award to Liu Xiaobo drove Chinese leaders into a rage; the government denounced Liu’s award as a “desecration” of Alfred Nobel’s legacy. For years, China had coveted a Nobel Prize as a validation of the nation’s progress and a measure of the world’s acceptance. The obsession with the prize was so intense that scholars had named it the “Nobel complex,” and each fall they debated China’s odds of winning it, like sports fans in a pennant race. There was once a television debate called “How Far Are We from a Nobel Prize?”

When the award was announced, most Chinese people had never heard of Liu, so the state media made the first impression; it splashed an article across the country reporting that he earned his living “bad-mouthing his own country.” The profile was a classic of the form: it described him as a collector of fine wines and porcelain, and it portrayed him telling fellow prisoners, “I’m not like you. I don’t lack for money. Foreigners pay me every year, even when I’m in prison.” Liu “spared no effort in working for Western anti-China forces” and, in doing so, “crossed the line of freedom of speech into crime.”

For activists, the news of the award was staggering. “Many broke down in tears, even uncontrollable sobbing,” one said later. In Beijing, bloggers, lawyers, and scholars gathered in the back of a restaurant to celebrate, but police arrived and detained twenty of them. When the announcement was made, Han Han, on his blog, toyed with censors and readers; he posted nothing but a pair of quotation marks enclosing an empty space. The post drew one and a half million hits and more than 28,000 comments.

Comment author: lukeprog 25 June 2014 12:02:03AM 0 points [-]

More (#1) from Osnos' Age of Ambition:

One of the most common rackets was illegal subcontracting. A single contract could be divvied up and sold for kickbacks, then sold again and again, until it reached the bottom of a food chain of labor, where the workers were cheap and unskilled. Railway ministry jobs were bought and sold: $ 4,500 to be a train attendant, $ 15,000 to be a supervisor. In November 2011 a former cook with no engineering experience was found to be building a high-speed railway bridge using a crew of unskilled migrant laborers who substituted crushed stones for cement in the bridge’s foundation. In railway circles, the practice of substituting cheap materials for real ones was common enough to rate its own expression: touliang huanzhu—“ robbing the beams to put in the pillars.”

With so many kickbacks changing hands, it wasn’t surprising that parts of the railway went wildly over budget. A station in Guangzhou slated to be built for $ 316 million ended up costing seven times that. The ministry was so large that bureaucrats would create fictional departments and run up expenses for them. A five-minute promotional video that went largely unseen cost nearly $ 3 million. The video led investigators to the ministry’s deputy propaganda chief, a woman whose home contained $ 1.5 million in cash and the deeds to nine houses.

Reporters who tried to expose the corruption in the railway world ran into dead ends. Two years before the crash, a journalist named Chen Jieren posted an article about problems in the ministry entitled, “Five Reasons That Liu Zhijun Should Take Blame and Resign,” but the piece was deleted from every major Web portal. Chen was later told that Liu oversaw a slush fund used for buying the loyalty of editors at major media and websites. Other government agencies also had serious financial problems— out of fifty, auditors found problems with forty-nine— but the scale of cash available in the railway world was in a class by itself. Liao Ran, an Asia specialist at Transparency International, told the International Herald Tribune that China’s high-speed railway was shaping up to be “the biggest single financial scandal not just in China, but perhaps in the world.”

And:

In February 2011, five months before the train crash, the Party finally moved on Liu Zhijun. According to Wang Mengshu, investigators concluded that Liu was preparing to use his illegal gains to bribe his way onto the Party Central Committee and, eventually, the Politburo. “He told Ding Shumiao, ‘Put together four hundred million for me. I’m going to need to spread some money around,’” Wang told me. Four hundred million yuan is about sixty-four million dollars. Liu managed to assemble nearly thirteen million yuan before he was stopped, Wang said. “The central government was worried that if he really succeeded in giving out four hundred million in bribes he would essentially have bought a government position. That’s why he was arrested.”

Liu was expelled from the Party the following May, for “severe violations of discipline” and “primary leadership responsibilities for the serious corruption problem within the railway system.” An account in the state press alleged that Liu took a 4 percent kickback on railway deals; another said he netted $ 152 million in bribes. He was the highest-ranking official to be arrested for corruption in five years. But it was Liu’s private life that caught people by surprise. The ministry accused him of “sexual misconduct,” and the Hong Kong newspaper Ming Pao reported that he had eighteen mistresses. His friend Ding was said to have helped him line up actresses from a television show in which she invested. Chinese officials are routinely discovered indulging in multiple sins of the flesh, prompting President Hu Jintao to give a speech a few years ago warning comrades against the “many temptations of power, wealth, and beautiful women.” But the image of a gallivanting Great Leap Liu, and the sheer logistics of keeping eighteen mistresses, made him into a punch line. When I asked Liu’s colleague if the mistress story was true, he replied, “What is your definition of a mistress?”

By the time the libidinous Liu was deposed, at least eight other senior officials had been removed and placed under investigation, including Zhang, Liu’s bombastic aide. Local media reported that Zhang, on an annual salary of less than five thousand dollars, had acquired a luxury home near Los Angeles, stirring speculation that he had been preparing to join the growing exodus of officials who were taking their fortunes abroad. In recent years, corrupt cadres who sent their families overseas had become known in Chinese as “naked officials.” In 2011 the central bank posted to the Web an internal report estimating that, since 1990, eighteen thousand corrupt officials had fled the country, having stolen $ 120 billion— a sum large enough to buy Disney or Amazon. (The report was promptly removed.)

And:

in China, people were more inclined to quote a very different statistic: in forty-seven years of service, high-speed trains in Japan had recorded just one fatality, a passenger caught in a closing door. It was becoming clear that parts of the new China had been built too fast for their own good. Three years had been set aside for construction of one of the longest bridges in North China, but it was finished in eighteen months, and nine months later, in August 2012, it collapsed, killing three people and injuring five. Local officials blamed overloaded trucks, though it was the sixth bridge collapse in a single year.

And:

After years of not daring to measure the Gini coefficient, in January 2013 the government finally published a figure, 0.47, but many specialists dismissed it; the economist Xu Xiaonian called it “a fairy tale.” (An independent calculation put the figure at 0.61, higher than the level in Zimbabwe.) Yet, for all the talk about income, it was becoming clear that people cared most of all about the gap in opportunity. When the Harvard sociologist Martin Whyte polled the Chinese public in 2009, he discovered that people had a surprisingly high tolerance for the rise of the plutocracy. What they resented were the obstacles that prevented them from joining it: weak courts, abuses of power, a lack of recourse. Two scholars, Yinqiang Zhang and Tor Eriksson, tracked the paths of Chinese families from 1989 to 2006 and found a “high degree of inequality of opportunity.” They wrote, “The basic idea behind the market reforms was that by enabling some citizens to become rich this would in turn help the rest to become rich as well. Our analysis shows that at least so far there are few traces of the reforms leveling the playing field.” They found that in other developing countries, parents’ education was the most decisive factor in determining how much a child would earn someday. But in China, the decisive factor was “parental connections.” A separate study of parents and children in Chinese cities found “a strikingly low level of intergenerational mobility.” Writing in 2010, the authors ranked “urban China among the least socially mobile places in the world.”

Comment author: lukeprog 31 October 2013 10:27:12PM *  6 points [-]

Lately I've been listening to audiobooks (at 2x speed) in my down time, especially ones that seem likely to have passages relevant to the question of how well policy-makers will deal with AGI, basically continuing this project but only doing the "collection" stage, not the "analysis" stage.

I'll post quotes from the audiobooks I listen to as replies to this comment.

Comment author: lukeprog 24 June 2014 11:55:32PM 0 points [-]

From Osnos' Age of Ambition:

I lived in China for eight years, and I watched this age of ambition take shape. Above all, it is a time of plenty— the crest of a transformation one hundred times the scale, and ten times the speed, of the first Industrial Revolution, which created modern Britain. The Chinese people no longer want for food— the average citizen eats six times as much meat as in 1976— but this is a ravenous era of a different kind, a period when people have awoken with a hunger for new sensations, ideas, and respect. China is the world’s largest consumer of energy, movies, beer, and platinum; it is building more high-speed railroads and airports than the rest of the world combined.

And:

In the event that these censorship efforts failed, the Party was testing a weapon of last resort: the OFF switch. On July 5, 2009, members of China’s Muslim Uighur minority in the far western city of Urumqi protested police handling of a brawl between Hans and Uighurs. The protests turned violent, and nearly two hundred people died, most of them Han, who had been targeted for their ethnicity. Revenge attacks on Uighur neighborhoods followed, and in an effort to prevent people from communicating and organizing, the government abruptly disabled text messages, cut long-distance phone lines, and shut off Internet access almost entirely. The digital blackout lasted ten months, and the economic effects were dramatic: exports from Xinjiang, the Uighur autonomous region, plummeted more than 44 percent. But the Party was willing to accept immense economic damage to smother what it considered a political threat. In the event of a broader crisis someday, China probably has too many channels in and out to impose so complete a blackout on a national scale, but even a limited version would have a profound effect.

And:

On October 8, 2010, ten months after Liu Xiaobo was convicted, the Nobel Committee awarded him the Peace Prize “for his long and non-violent struggle for fundamental human rights.” He was the first Chinese citizen to receive the award, not counting the Dalai Lama, who had lived for decades in exile. The award to Liu Xiaobo drove Chinese leaders into a rage; the government denounced Liu’s award as a “desecration” of Alfred Nobel’s legacy. For years, China had coveted a Nobel Prize as a validation of the nation’s progress and a measure of the world’s acceptance. The obsession with the prize was so intense that scholars had named it the “Nobel complex,” and each fall they debated China’s odds of winning it, like sports fans in a pennant race. There was once a television debate called “How Far Are We from a Nobel Prize?”

When the award was announced, most Chinese people had never heard of Liu, so the state media made the first impression; it splashed an article across the country reporting that he earned his living “bad-mouthing his own country.” The profile was a classic of the form: it described him as a collector of fine wines and porcelain, and it portrayed him telling fellow prisoners, “I’m not like you. I don’t lack for money. Foreigners pay me every year, even when I’m in prison.” Liu “spared no effort in working for Western anti-China forces” and, in doing so, “crossed the line of freedom of speech into crime.”

For activists, the news of the award was staggering. “Many broke down in tears, even uncontrollable sobbing,” one said later. In Beijing, bloggers, lawyers, and scholars gathered in the back of a restaurant to celebrate, but police arrived and detained twenty of them. When the announcement was made, Han Han, on his blog, toyed with censors and readers; he posted nothing but a pair of quotation marks enclosing an empty space. The post drew one and a half million hits and more than 28,000 comments.

Comment author: lukeprog 21 June 2014 09:40:22PM 2 points [-]

From Ariely's The Honest Truth about Dishonesty:

A few years ago I received a letter from a woman named Rhonda who attended the University of California at Berkeley. She told me about a problem she’d had in her house and how a little ethical reminder helped her solve it.

She was living near campus with several other people—none of whom knew one another. When the cleaning people came each weekend, they left several rolls of toilet paper in each of the two bathrooms. However, by Monday all the toilet paper would be gone. It was a classic tragedy-of-the-commons situation: because some people hoarded the toilet paper and took more than their fair share, the public resource was destroyed for everyone else.

After reading about the Ten Commandments experiment on my blog, Rhonda put a note in one of the bathrooms asking people not to remove toilet paper, as it was a shared commodity. To her great satisfaction, one roll reappeared in a few hours, and another the next day. In the other note-free bathroom, however, there was no toilet paper until the following weekend, when the cleaning people returned.

Comment author: lukeprog 24 June 2014 11:48:54PM 0 points [-]

More (#2) from Ariely's The Honest Truth about Dishonesty:

On one particular flight, I was flipping through a magazine and discovered a MENSA quiz (questions that are supposed to measure intelligence). Since I am rather competitive, I naturally had to try it. The directions said that the answers were in the back of the magazine. After I answered the first question, I flipped to the back to see if I was correct, and lo and behold, I was. But as I continued with the quiz, I also noticed that as I was checking the answer to the question I just finished solving, my eyes strayed just a bit to the next answer. Having glanced at the answer to the next question, I found the next problem to be much easier. At the end of the quiz, I was able to correctly solve most of the questions, which made it easier for me to believe that I was some sort of genius. But then I had to wonder whether my score was that high because I was supersmart or because I had seen the answers out of the corner of my eye (my inclination was, of course, to attribute it to my own intelligence).

The same basic process can take place in any test in which the answers are available on another page or are written upside down, as they often are in magazines and SAT study guides. We often use the answers when we practice taking tests to convince ourselves that we’re smart or, if we get an answer wrong, that we’ve made a silly mistake that we would never make during a real exam. Either way, we come away with an inflated idea of how bright we actually are—and that’s something we’re generally happy to accept.

And:

During one unbearably long operation on my hands, the doctors inserted long needles from the tips of my fingers through the joints in order to hold my fingers straight so that the skin could heal properly. At the top of each needle they placed a cork so that I couldn’t unintentionally scratch myself or poke my eyes. After a couple of months of living with this unearthly contraption, I found that it would be removed in the clinic—not under anesthesia. That worried me a lot, because I imagined that the pain would be pretty awful. But the nurses said, “Oh, don’t worry. This is a simple procedure and it’s not even painful.” For the next few weeks I felt much less worried about the procedure.

When the time came to withdraw the needles, one nurse held my elbow and the other slowly pulled out each needle with pliers. Of course, the pain was excruciating and lasted for days—very much in contrast to how they described the procedure. Still, in hindsight, I was very glad they had lied to me. If they had told me the truth about what to expect, I would have spent the weeks before the extraction anticipating the procedure in misery, dread, and stress—which in turn might have compromised my much-needed immune system. So in the end, I came to believe that there are certain circumstances in which white lies are justified.

Comment author: lukeprog 21 June 2014 09:40:22PM 2 points [-]

From Ariely's The Honest Truth about Dishonesty:

A few years ago I received a letter from a woman named Rhonda who attended the University of California at Berkeley. She told me about a problem she’d had in her house and how a little ethical reminder helped her solve it.

She was living near campus with several other people—none of whom knew one another. When the cleaning people came each weekend, they left several rolls of toilet paper in each of the two bathrooms. However, by Monday all the toilet paper would be gone. It was a classic tragedy-of-the-commons situation: because some people hoarded the toilet paper and took more than their fair share, the public resource was destroyed for everyone else.

After reading about the Ten Commandments experiment on my blog, Rhonda put a note in one of the bathrooms asking people not to remove toilet paper, as it was a shared commodity. To her great satisfaction, one roll reappeared in a few hours, and another the next day. In the other note-free bathroom, however, there was no toilet paper until the following weekend, when the cleaning people returned.

Comment author: lukeprog 24 June 2014 11:42:05PM 1 point [-]

More (#1) from Ariely's The Honest Truth about Dishonesty:

Armed with our evidence that when people sign their names to some kind of pledge, it puts them into a more honest disposition (at least temporarily), we approached the IRS, thinking that Uncle Sam would be glad to hear of ways to boost tax revenues. The interaction with the IRS went something like this:

ME: By the time taxpayers finish entering all the data onto the form, it is too late. The cheating is done and over with, and no one will say, “Oh, I need to sign this thing, let me go back and give honest answers.” You see? If people sign before they enter any data onto the form, they cheat less. What you need is a signature at the top of the form, and this will remind everyone that they are supposed to be telling the truth.

IRS: Yes, that’s interesting. But it would be illegal to ask people to sign at the top of the form. The signature needs to verify the accuracy of the information provided.

ME: How about asking people to sign twice? Once at the top and once at the bottom? That way, the top signature will act as a pledge—reminding people of their patriotism, moral fiber, mother, the flag, homemade apple pie—and the signature at the bottom would be for verification.

IRS: Well, that would be confusing.

ME: Have you looked at the tax code or the tax forms recently?

IRS: [No reaction.]

ME: How about this? What if the first item on the tax form asked if the taxpayer would like to donate twenty-five dollars to a task force to fight corruption? Regardless of the particular answer, the question will force people to contemplate their standing on honesty and its importance for society! And if the taxpayer donates money to this task force, they not only state an opinion, but they also put some money behind their decision, and now they might be even more likely to follow their own example.

IRS: [Stony silence.]

And:

Over the course of many years of teaching, I’ve noticed that there typically seems to be a rash of deaths among students’ relatives at the end of the semester, and it happens mostly in the week before final exams and before papers are due. In an average semester, about 10 percent of my students come to me asking for an extension because someone has died—usually a grandmother. Of course I find it very sad and am always ready to sympathize with my students and give them more time to complete their assignments. But the question remains: what is it about the weeks before finals that is so dangerous to students’ relatives?

Most professors encounter the same puzzling phenomenon, and I’ll guess that we have come to suspect some kind of causal relationship between exams and sudden deaths among grandmothers. In fact, one intrepid researcher has successfully proven it. After collecting data over several years, Mike Adams (a professor of biology at Eastern Connecticut State University) has shown that grandmothers are ten times more likely to die before a midterm and nineteen times more likely to die before a final exam. Moreover, grandmothers of students who aren’t doing so well in class are at even higher risk—students who are failing are fifty times more likely to lose a grandmother compared with non-failing students.

In a paper exploring this sad connection, Adams speculates that the phenomenon is due to intrafamilial dynamics, which is to say, students’ grandmothers care so much about their grandchildren that they worry themselves to death over the outcome of exams. This would indeed explain why fatalities occur more frequently as the stakes rise, especially in cases where a student’s academic future is in peril. With this finding in mind, it is rather clear that from a public policy perspective, grandmothers—particularly those of failing students—should be closely monitored for signs of ill health during the weeks before and during finals. Another recommendation is that their grandchildren, again particularly the ones who are not doing well in class, should not tell their grandmothers anything about the timing of the exams or how they are performing in class.

Though it is likely that intrafamilial dynamics cause this tragic turn of events, there is another possible explanation for the plague that seems to strike grandmothers twice a year. It may have something to do with students’ lack of preparation and their subsequent scramble to buy more time than with any real threat to the safety of those dear old women. If that is the case, we might want to ask why it is that students become so susceptible to “losing” their grandmothers (in e-mails to professors) at semesters’ end.

Perhaps at the end of the semester, the students become so depleted by the months of studying and burning the candle at both ends that they lose some of their morality and in the process also show disregard for their grandmothers’ lives. If the concentration it takes to remember a longer digit can send people running for chocolate cake, it’s not hard to imagine how dealing with months of cumulative material from several classes might lead students to fake a dead grandmother in order to ease the pressure (not that that’s an excuse for lying to one’s professors).

Comment author: paulfchristiano 23 June 2014 07:48:43AM *  6 points [-]

I don't think the linked PCP thing is a great example. Yes, the first time someone seriously writes an algorithm to do X it typically represents a big speedup on X. The prediction of the "progress is continuous" hypothesis is that the first time someone writes an algorithm to do X, it won't be very economically important---otherwise someone would have done it sooner---and this example conforms to that trend pretty well.

The other issue seems closer to relevant; mathematical problems do go from being "unsolved" to "solved" with comparatively little warning. I think this is largely because they are small enough problems that they are 1 man jobs (which would not be plausible if anyone really cared about the outcome), but that may not be the whole story and at any rate something is going on here.

In the PCP case, the relevantly similar outcome would be the situation where theoretical work on interactive proofs turned out to be useful right out of the box. I'm not aware of any historical cases where this has happened, but I could be missing some, and I don't really understand why it would happen as rarely as it does. It would be nice to understand this possibility better.

As for "people can't tell the difference between watson and being close to broadly human-level AI,” I think this is unlikely. At the very least the broader intellectual community is going to have little trouble distinguishing between watson and economically disruptive AI, so this is only plausible if we get a discontinuous jump. But even assuming a jump, the AI community is not all that impressed by watson and I expect this is an important channel by which significant developments would affect expectations.

Comment author: lukeprog 24 June 2014 11:24:17PM 0 points [-]

Just for the record, I wasn't proposing the PCP thing as a counterexample to your model of "economically important progress is continuous."

View more: Next