Articles Tagged ‘economics’ - Less Wrong

New report: Intelligence Explosion Microeconomics

Tue, 30 Apr 2013 09:14:58 +1000

Submitted by Eliezer_Yudkowsky • 43 votes • 225 comments

Summary: Intelligence Explosion Microeconomics (pdf) is 40,000 words taking some initial steps toward tackling the key quantitative issue in the intelligence explosion, "reinvestable returns on cognitive investments": what kind of returns can you get from an investment in cognition, can you reinvest it to make yourself even smarter, and does this process die out or blow up? This can be thought of as the compact and hopefully more coherent successor to the AI Foom Debate of a few years back.

(Sample idea you haven't heard before: The increase in hominid brain size over evolutionary time should be interpreted as evidence about increasing marginal fitness returns on brain size, presumably due to improved brain wiring algorithms; not as direct evidence about an intelligence scaling factor from brain size.)

I hope that the open problems posed therein inspire further work by economists or economically literate modelers, interested specifically in the intelligence explosion qua cognitive intelligence rather than non-cognitive 'technological acceleration'. MIRI has an intended-to-be-small-and-technical mailing list for such discussion. In case it's not clear from context, I (Yudkowsky) am the author of the paper.

Abstract:

I. J. Good's thesis of the 'intelligence explosion' is that a sufficiently advanced machine intelligence could build a smarter version of itself, which could in turn build an even smarter version of itself, and that this process could continue enough to vastly exceed human intelligence. As Sandberg (2010) correctly notes, there are several attempts to lay down return-on-investment formulas intended to represent sharp speedups in economic or technological growth, but very little attempt has been made to deal formally with I. J. Good's intelligence explosion thesis as such.

I identify the key issue as returns on cognitive reinvestment - the ability to invest more computing power, faster computers, or improved cognitive algorithms to yield cognitive labor which produces larger brains, faster brains, or better mind designs. There are many phenomena in the world which have been argued as evidentially relevant to this question, from the observed course of hominid evolution, to Moore's Law, to the competence over time of machine chess-playing systems, and many more. I go into some depth on the sort of debates which then arise on how to interpret such evidence. I propose that the next step forward in analyzing positions on the intelligence explosion would be to formalize return-on-investment curves, so that each stance can say formally which possible microfoundations they hold to be falsified by historical observations already made. More generally, I pose multiple open questions of 'returns on cognitive reinvestment' or 'intelligence explosion microeconomics'. Although such questions have received little attention thus far, they seem highly relevant to policy choices affecting the outcomes for Earth-originating intelligent life.

The dedicated mailing list will be small and restricted to technical discussants.

This topic was originally intended to be a sequence in Open Problems in Friendly AI, but further work produced something compacted beyond where it could be easily broken up into subposts.

Outline of contents:

1: Introduces the basic questions and the key quantitative issue of sustained reinvestable returns on cognitive investments.

2: Discusses the basic language for talking about the intelligence explosion, and argues that we should pursue this project by looking for underlying microfoundations, not by pursuing analogies to allegedly similar historical events.

3: Goes into detail on what I see as the main arguments for a fast intelligence explosion, constituting the bulk of the paper with the following subsections:

3.1: What the fossil record actually tells us about returns on brain size, given that most of the difference between Homo sapiens and Australopithecus was probably improved software.
3.2: How to divide credit for the human-chimpanzee performance gap between "humans are individually smarter than chimpanzees" and "the hominid transition involved a one-time qualitative gain from being able to accumulate knowledge".
3.3: How returns on speed (serial causal depth) contrast with returns from parallelism; how faster thought seems to contrast with more thought. Whether sensing and manipulating technologies are likely to present a bottleneck for faster thinkers, or how large of a bottleneck.
3.4: How human populations seem to scale in problem-solving power; some reasons to believe that we scale inefficiently enough for it to be puzzling. Garry Kasparov's chess match vs. The World, which Kasparov won.
3.5: Some inefficiencies that might cumulate in an estimate of humanity's net computational efficiency on a cognitive problem.
3.6: What the anthropological record actually tells us about cognitive returns on cumulative selection pressure, given that selection pressures were probably increasing over the course of hominid history. How the observed history would be expected to look different, if there were in fact diminishing returns on cognition.
3.7: How to relate the curves for evolutionary difficulty, human-engineering difficulty, and AI-engineering difficulty, considering that they are almost certainly different.
3.8: Correcting for anthropic bias in trying to estimate the intrinsic 'difficulty 'of hominid-level intelligence just from observing that intelligence evolved here on Earth.
3.9: The question of whether to expect a 'local' (one-project) FOOM or 'global' (whole economy) FOOM and how returns on cognitive reinvestment interact with that.
3.10: The great open uncertainty about the minimal conditions for starting a FOOM; why I. J. Good's postulate of starting from 'ultraintelligence' is probably much too strong (sufficient, but very far above what is necessary).
3.11: The enhanced probability of unknown unknowns in the scenario, since a smarter-than-human intelligence will selectively seek out and exploit flaws or gaps in our current knowledge.

4: A tentative methodology for formalizing theories of the intelligence explosion - a project of formalizing possible microfoundations and explicitly stating their alleged relation to historical experience, such that some possibilities can allegedly be falsified.

5: Which open sub-questions seem both high-value and possibly answerable.

6: Formally poses the Open Problem and mentions what it would take for MIRI itself to directly fund further work in this field.

225 comments

Is Sunk Cost Fallacy a Fallacy?

Sat, 04 Feb 2012 15:33:49 +1100

Submitted by gwern • 16 votes • 52 comments

I just finished the first draft of my essay, "Are Sunk Costs Fallacies?"; there is still material I need to go through, but the bulk of the material is now there. The formatting is too gnarly to post here, so I ask everyone's forgiveness in clicking through.

To summarize:

sunk costs are probably issues in big organizations
- but maybe not ones that can be helped
sunk costs are not issues in animals
they appear to be in children & adults
- but many apparent problems can be explained as part of a learning strategy
there are few clear indications sunk costs are genuine problems
much of what we call 'sunk cost' looks like simple carelessness & thoughtlessness

(If any of that seems unlikely or absurd to you, click through. I've worked very hard to provide multiple citations where possible, and fulltext for practically everything.)

I started this a while ago; but Luke/SIAI paid for much of the work, and that motivation plus academic library access made this essay more comprehensive than it would have been and finished months in advance.

52 comments

Prediction is hard, especially of medicine

Sat, 24 Dec 2011 07:34:13 +1100

Submitted by gwern • 46 votes • 46 comments

Summary: medical progress has been much slower than even recently predicted.

In the February and March 1988 issues of Cryonics, Mike Darwin (Wikipedia/LessWrong) and Steve Harris published a two-part article “The Future of Medicine” attempting to forecast the medical state of the art for 2008. Darwin has republished it on the New_Cryonet email list.

Darwin is a pretty savvy forecaster (who you will remember correctly predicting in 1981 in “The High Cost of Cryonics”/part 2 ALCOR’s recent troubles with grandfathering), so given my standing interests in tracking predictions, I read it with great interest; but they still blew most of them, and not the ones we would prefer them to’ve.

The full essay is ~10k words, so I will excerpt roughly half of it below; feel free to skip to the reactions section and other links.

1 The Future of Medicine

1.1 Part 1

What we hope we are especially good at as cryonicists is predicting the future — particularly the future of medicine. After all, our lives depend upon it. Because that’s what cryonics is about — tomorrow’s medicine today. In order for cryonics to seem reasonable, in order for it to be reasonable, it is necessary to have some idea, at least in broad outline, of where medicine is going and of where it ultimately can go. I think that the cryonicists’ record on this point in a broad sense has been very good.

…One thing which is rarely seen in cryonics publications is an attempt to see the shape of things to come in the near or intermediate future. Oddly enough, that’s a far more difficult and dangerous undertaking than predicting ultimates. Nor is this a problem confined to cryonics or the future of medicine. Sadi Carnot (the founder of thermodynamics) could tell you all about the “perfect heat engine,” but would have no doubt had trouble giving you hard numbers on how well heat engines would be made to perform over the 20 years or so following publication of his work….When I look over predictions made in the 1950’s or the 1960’s about the future of medicine and/or technology, I always chuckle about just how far afield these guys were. A good example is a list of predictions made by Herman Kahn which was summarized in CRYONICS REPORTS in August of 1967 (volume 2, #8). They are reproduced as Table 1 below. Read ’em and weep — or laugh if you will!

Table 1. Less Likely But Important Possibilities, from: The Next 33 Years: A Framework For Speculation, by Herman Kahn and Anthony J. Weiner (1967) [predictions for 2000 AD]

“True” artificial intelligence

Practical use of sustained fusion to produce neutrons

Artificial growth of new limbs and organs

Room temperature superconductors

Major use of rockets for transportation (either terrestrial or extraterrestrial)

Effective chemical or biological treatment for most mental illnesses

Almost complete control of marginal changes of heredity

Suspended animation (for years or centuries)

Practical materials with nearly “theoretical limit” strengths

Conversion of mammals (humans?) to fluid breathers

Direct input into human memory banks

…

…My personal perspective is one of being a hard-core cryonicist who was involved in clinical medicine for the better part of a decade. My biases about predicting the future could probably be summarized as follows: I have a lot of sympathy for the incrementalist view of progress - particularly in the highly regulated area of medicine. It’s regulated because it directly and powerfully touches people’s well-being and because it is not a very fault-tolerant area — mistakes are costly and since people like being alive (at least in the short run) they get edgy if an error separates them from their actuarial expectations.

I thus believe that any predictions about the future of medicine have to include what I call the “space program factor” (SPF). By this I mean simply that progress in the space program would have proceeded far, far faster (and thus approximated more closely what was theoretically possible) if it were not a high-visibility project with lots of political and social overtones which make it fault-intolerant — if you could burn up as many astronauts as you do test pilots every month, it would cost a lot less to get where you’re going. First-shot fail-safe engineering is costly. Medicine suffers from the same kinds of problems — witness the FDA as both the solution and the problem.

1.1.1 Diagnostics

I foresee a veritable explosion of diagnostic techniques and procedures. A large number of illnesses which are poorly understood today will be well-characterized the next decade and will be easy to diagnose very early in their development or even before they develop because they will be found to have direct or indirect genetic causes. Fairly predictive tests for Alzheimer’s disease, schizophrenia, depression, some malignancies, heart disease, and most of the rest of the major killers and disablers will probably be in place by 2000 to 2010. Many if not most of these ailments will be assessable in terms of a very sophisticated genetic risk profile which it will be possible to generate in infancy or childhood (or in utero). A wide range of genetic probes for illness-generating genes should be available by the end of the century.

A side-note: genetic associations have been a very fertile field for John Ioannidis, and a big study just blew away a bunch of SNP-IQ correlations.

Real-time diagnosis will also be revolutionized by the turn of the century. The next 10 to 15 years will see increasing miniaturization of sensors and chemistry packages. Tissue probes or biosensors which can measure a wide array of biological and biochemical factors will be packaged in very small, very stable devices which hold calibration over prolonged periods of time (weeks to months to years) and which can easily be inserted into the patient’s body or tissues. For example, I foresee multi-sensor units mounted on very small needle or catheter tips which can be inserted intravenously, intracranially, intra-cerebrally, subcutaneously, and so on.

These sensors will be able to give real-time measurements of blood gases, pH, electrolytes, enzyme levels, and a host of other biochemical parameters that now involve costly, time-consuming, and/or impossible “laboratory studies” requiring withdrawal of a sample and processing. Real-time biosensors will revolutionize acute care of critically ill patients.

…The first generation of these devices should be in the marketplace somewhere between 1990 and 1995. More sophisticated instruments capable of a wider array of measurements will quickly follow. These sensors will also have a profound impact in acute stabilization of patients in a field setting. It will be possible for paramedical personnel to quickly and effectively insert such instruments in an acutely ill patient — a victim of cardiac arrest or trauma, and immediately and globally assess that patient’s condition, relaying that information to an expert (more on who that expert will be later).

…Diagnostic imaging should rapidly come down to a battle between ultrasound and MRI (NMR; (nuclear) magnetic resonance imaging). Because ultrasound units owe their size and weight almost entirely to the computer that processes the information, the size and effectiveness of these units will change on the same rapid exponential curve as the size and power of computers. MRI is a technology which has some other physical limitations, but by the year 2000, even MRI units will be far smaller, less costly, and capable of far, far better results. Bedside units or “on floor” units (i.e., units in the ICU or CCU) may be available for repeated assessment of the patient’s condition. MRI and its grandchildren and cousins should in particular be expected to undergo considerable refinement. Metabolic MRI will also be in wider use, allowing for real-time evaluation of the metabolic and working state of patient’s hearts, brains and other organs. By 2000 to 2010 the cost and size of these units may be drastically reduced and they may be in field use for acute metabolic and structural evaluation of patients with trauma or in cardiac arrest.

I recently learned that, besides the usual blame for increasing medical costs, some categories of doctors have been strenuously urged to reduce MRI use as actively harmful.

By the late 1990’s there should be an answer to this problem in the development of the Portable Doctor or Expert Medical Device (EMD). The EMD will be both a diagnostician and therapist integrated into one unit. In an emergency medical setting (either in an ambulance or in an ICU or CCU) this powerful computer will be directly coupled to a wide array of both simple and complex medical assessment devices….EMDs will be a very hot item. Initially (i.e., the 1990’s) they will be confined to ambulances and the ICU, CCU, and specialty areas of the hospital, such as radiology and cardiology labs. But there will be powerful incentives for wider application of these devices. As computing capacity drops in cost and increases radically in sophistication (i.e., parallel processors, neural networks, truly massive memories, and so on) expert medical (and other) systems will see increasing application. There will be devices on the market such as a “Home Doctor” diagnostic program, which will basically be an internal medicine physician in a can.

…After 2000, many people will probably have a small sensor array permanently implanted and coupled to telemetry equipment which can be activated to call for help or alert the person that trouble is brewing. People with a known risk of sudden health problems will be the first to use these kinds of devices. With the development of smaller and cheaper telemetry equipment (directly linked to large-antenna satellites), separate telemetry arrangements will disappear. Implantable, computer-controlled defibrillators are already a reality; analogous devices to deliver drugs in case of cardiac or brain infarct (stroke) will eventually become reality.

1.1.2 Resuscitation

Expect a shift back to open-chest heart massage and away from closed-chest massage in medical and perhaps even paramedical settings. Closed-chest CPR will be realized to be ineffective at maintaining cerebral viability and will be replaced by far more effective open chest methods. In paramedical (i.e., field) settings the emphasis will be on very rapid defibrillation — or actually “leaving the patient alone” until circulation can be effectively restored and medications given to inhibit reperfusion injury. Closed chest CPR and restarting circulation by laymen “in the field” will be realized to be doing more harm than good and there may well be a move away from field CPR, with laymen being instructed to leave the patient without circulation until it can be restarted adequately and under controlled conditions.

By the late 1990’s, extended use of CPR will be a thing of the past and major metropolitan areas will have “death reversal units” (DRUs) in emergency rooms and perhaps even in larger paramedical units. The DRUs will employ rapid femoral cut-downs and blood-pump/oxygenator supported resuscitation to recover people who have suffered extended periods of ischemia (in the 30 minute to 1 hour range). CPR will be realized very often to be ineffective at recovering patients who are profoundly ischemic and the advent of pharmacologic intervention allowing for cerebral resuscitation will provide tremendous pressure for emergency rooms to develop the capability to very rapidly put an ischemic patient on bypass and completely and adequately support his circulatory and respiratory needs until his brain can recover and/or his heart can be repaired and restarted. An intermediate scenario would be the development of small, flexible impeller pumps that can be collapsed and passed through a large bore percutaneous catheter through the femoral artery and into the abdominal aorta. Such a pump (acting much like the propeller on an outboard boat motor) could then be used to supplement CPR, perhaps providing 2–3 liters per minute of cardiac output.

…Another effect of drugs like the lazaroids and calcium channel blockers will be the more effective treatment of acute injuries to a wide range of tissues such as the spinal cord and brain. Much of the damage that occurs to these tissues is free radical related and can be inhibited by use of these drugs…Intervention into secondary inflammation will be most important in the brain and spinal cord. Deployment of these techniques will result in the salvage of many spinal cords that would be considered irreversibly injured by today’s medicine. There will be far, far fewer paraplegics. However, expect an increase in the number of permanently brain-injured patients and in the number of patients with “subtle” forms of cerebral injury resembling mild stroke or the cognitive or mood disorders seen in diseases like multiple sclerosis or acute head injury. These disease states will result because people with brain trauma who would have died acutely from secondary free radical mediated injury (cerebral edema and so on) will be saved with lazaroids and other cerebral rescue techniques.

1.1.3 Antibiotics

The next twenty years should see many powerful new antibiotics engineered directly from knowledge of the structure of the relevant microbial enzyme which it is desired to inhibit. Not only will these antibiotics be more powerful, but because they do not exist in nature, strain resistance will not so easily develop toward them as it has for the antibiotics of today.

In addition, the next generation of antibiotics will include many which have been designed for effect against viruses, an area where medicine is presently largely powerless.

The pharmaceutical industry and antibiotics have been a case-study in stagnation, failure, and diminishing marginal returns. There is only one, highly experimental, anti-viral that I have heard of. In a followup email, Darwin responded to someone else pointing out DRACO:

Finally, while Geoff cites this putative advance in antiviral drug therapy, the fact is that my prediction about a plethora of new and highly effective targeted molecular antimicrobials by 2008 was WRONG. In fact, antibiotic research is all but dead, and there are virtually no fundamentally new antibiotics in the drug pipeline. This should scare the crap out of all us, because we are rapidly approaching complete antibiotic resistance with a number of common and highly lethal bugs, including staph (MRSA), streptococcus, E. coli, pseudomonas and candida. It is only a matter of months to a few years, at most, before completely antibiotic resistance staph and streptococcus emerge. Pharmaceutical companies have a large negative incentive for developing new antimicrobials. At the cost of over a billion dollars a new drug (regulatory) and the high risk of withdrawal of the drug within 5 years (2 out of 3), as well as the near certainty of punishing litigation for adverse effects, antibiotics are not merely uneconomical to develop, they are fiscal suicide. Only drugs that will be chronically used by very large numbers of patients are now worth developing.

(This agrees with my own general impressions, which I didn't feel competent to baldly state.)

1.1.4 Immunology and cancer

…Monoclonal and synthetic antibodies carrying toxins or regulatory molecules will be used to turn off or destroy the fraction of immune cells which initially respond and proliferate when a transplant is carried out. More widespread transplantation of tissues will be undertaken, including transplantation of limbs and scalp. Xenografts will be used increasingly in the mid to late 1990’s and it will not be uncommon for people to have pancreatic tissue from bovine or porcine sources and perhaps hearts, lungs, and livers from other animals. Expect the first workable transplants to be from great apes (chimps, gorillas, orangutans), with porcine and bovine grafts coming later.

Immunology and immunotherapy will also be revolutionized by a far more complete understanding of the immune system resulting from the AIDS epidemic and basic research in the immunology of diseases such as multiple sclerosis and aging. The ability to rapidly and cheaply synthesize bioregulatory molecules will open up a wide array of therapeutic possibilities. Expect effective treatments for most autoimmune diseases (lupus, multiple sclerosis, myasthenia gravis, and so on) by the mid to late 1990’s. The mid to late 1990’s should also see the wider application of immunorestoratives for use with the aged and ill. Cancer therapy will improve considerably as a result of these advances as well as a result of selective targeting techniques. By the early to mid–1990s the first generations of monoclonal antibodies linked to chemotherapeutic agents or powerful natural toxins will be used against a few cancers.

1.1.5 Atherosclerosis

Atherosclerosis will undergo a very marked but nevertheless gradual reduction in frequency and severity of occurrence as physicians slowly become educated about what is already known and begin to use existing therapeutic modalities more aggressively. By the mid to late 1990’s it will be more widely understood that atherosclerosis can be reversed, and there will be wider use of drugs such as lovastatin to reduce serum cholesterol, coupled with sound dietary advice. However, even well into the late 1990’s and perhaps beyond, atherosclerotic disease (heart attack, stroke, ischemic limb disease, and so on) will continue to be a serious source of morbidity and mortality. By the late 1990’s, 2nd and 3rd generation therapies will be coming on-line which will be able to reverse atherosclerotic disease and more directly inhibit it

1.2 Part 2

1.2.1 Anesthesia

Expect “modular” anesthesia by the 1990’s to the early 2000’s. The development of potent anxieolytics (anxiety removers) which do not depress consciousness and the development of total pain inhibitors will allow for complicated surgical procedures on conscious patients. Expect to see major thoracic and limb surgery on high risk patients (i.e., patients unable to tolerate anesthesia) using such agents.Major abdominal surgery requiring deep muscle relaxation will continue to require skeletal muscle paralysis and general anesthesia. However, expect new drugs in the market place in the late 1990’s which induce unconsciousness without respiratory or cardiac depression.

Surgical and post surgical mortality will decrease sharply due to such anesthetics and the use of real-time physiological and biochemical monitoring during and after surgery using biosensors.

1.2.2 Surgery

…Catheters, laparascopes, and thorascopes with sensors, operating tools, and an impressive array of capabilities will be increasingly used. Abdominal surgery will shift more and more towards the use of the fiberoptic laparascope, endoscope, and laser as miniaturization of tools occurs and disease is diagnosed earlier. Early diagnosis will create the need for less drastic procedures.

Fine-tuned repair of heart valves and blood vessels, and examination and biopsy of suspected abdominal and retroperitoneal lesions will be early candidates for application of this technology.

…In contrast to therapeutic surgery, the frequency of cosmetic surgery will probably increase dramatically as techniques are refined and prosthetics improve in quality and drop in cost. As people live longer, and stay productive longer as well, they will increasingly turn to medicine to maintain not only their health but their appearance. Cosmetic surgery will experience a boom until such time as the fundamental mechanisms underlying the aging process can be brought under control.

1.2.3 Geriatrics

Advances will be slow here, but significant. Expect increasing understanding and application of trophic factors and bioregulatory compounds. Early candidates for rejuvenation will be the immune system and other stem cell systems or systems with higher cell turnover. By the early decades of 2000, significant rejuvenation and geroprophylaxis of skin, bone, immune, and other “high turn- over” tissues will be possible as the natural regulatory molecules which control these systems are understood and applied.

…By the early years of the 21st century the first generation of compounds effective at “rejuvenating” (i.e., restoring some degree of normal maintenance and repair to existing brain cells) the central nervous system will be available. These drugs will work by turning on protein synthesis and stimulating natural repair mechanisms.

However, pathologies of the brain and other non-dividing tissues (renal, cardiac, and musculoskeletal system) will continue to be major sources of morbidity and mortality over the next two decades. As atherosclerosis and immune-related disorders are dealt with more effectively, expect an increasing shift of morbidity and mortality to central nervous system-related causes. Beyond 2000 this may be treated to a limited extent with fetal transplant

We all know how well this has worked out. More troubling is that in some respects, we appear further from any solutions or treatments than before; while resveratrol did well in a recent human trial, the sirtuin research that seemed so promising has been battered by null results and failures to replicate. And anti-aging drugs have their own methodological difficulties; from the followup email:

Antiaging drugs are unlikely to be free of adverse effects. In fact, it seems very likely that they will be burdened with many adverse effects and that they will even kill a minority of people who use them. The common perception is that antiaging drugs will make people super fit, healthier and more resistant to disease. And yet, in calorie restriction and effective antiaging drug studies there is emerging evidence that slowing aging comes at the cost of interfering with fundamental processes that make organisms fitter for both reproduction and for surviving in a hostile environment.

Consider the putative antaging drug rapamycin. It seems likely that rapamycin interferes with senesence by affecting the PI3-kinase and TOR: PIKTORing cell growth pathways. This almost certainly means that in some individuals there will serious and even lethal side effects - cancer being one of them. [Persons with a history of promiscuity, and thus a heavy burden of chronic viral infection, and those with certain "unfavorable" genotypes will likely be at very high risk.] But, beyond cancer, interfering with these fundamental and deeply evolutionarily conserved pathways is likely to cause a range of adverse effects that negatively (and possibly irreversibly) impact normal body functions, such as energy level, cognition, sexual performance, and so on.. While some people are now using rapamycin as an antiaging drug...it is virtually inconceivable that any major pharmaceutical company anywhere in the world would (or will) market such a drug for "normal" aging. This is important to understand because it gives us basic insight into what will almost certainly be a major barrier to the development and marketing of antiaging drugs: they will necessarily be used by large numbers of people over the course of many decades (and thus millions of drug/person years) and they are incredibly unlikely to be free of adverse, and sometimes even lethal side effects.

1.2.4 Psychiatry & Behavior

Diagnosis by brain scanning (metabolic MRI) and chemical analysis of cerebrospinal fluids will be commonplace in 20 years. As neuroregulatory compounds are better understood and as the biochemistry underlying mental disorders is elucidated there will be more effective treatments. Expect 2nd and 3rd generation drugs and combinations thereof for treatment of depression and psychosis by the late 1990’s. There will probably be several very effective therapeutic agents for compulsive disorders in the marketplace by the early to mid 1990’s.

From the previously quoted followup email:

Similarly, psychiatric drugs (which are typically chronically used) are no longer economical to develop and market because of the litigation costs associated with them. Widespread chronic use of any drug means that the likelihood of adverse conditions that were impossible to detect in the testing phase of the drug development process are almost certain to emerge. Statistics rule in drug development, and a Phase III study that lasts a year and enrolls 5,000 patients is simply not adequately powered to predict what will happen when 5 million patients take a drug for 20 years! The only way to get that data is to do that study. And therein lies a powerful caution about antiaging drugs. These drugs will likely need to be taken starting in young adulthood, or in middle age, at latest, and they will need to be taken for a lifetime. Indeed, if they are effective, for a longer lifetime than any but a few super-centenarians has previously lived.

1.2.5 Implants & Prosthetics

Early spectacular applications will be small vessel prostheses (wide use by the early to mid 1990’s) for use in traumatized and atherosclerotic limbs and organs and venous prostheses (mid to late 1990’s) for use in treating traumatic injuries and deep vein incompetence (which results in varicosities, chronic pain, and edema-related skin changes in the leg, often leading to non-healing ulcers or limb loss). Another application of non-thrombogenic surfaces will be a practical artificial heart and more widespread use of extracorporeal support for infants, trauma and cardiac arrest victims, and others where anticoagulation provides a major barrier to the use of artificial circulation.

…Good synthetic bone and skin should be available by the late 1990’s to early 2000’s. Good red cell and plasma substitutes (synthetic blood) should be seen increasing in clinical use throughout the early 1990’s and in frequent use by the late 1990’s to early 2000’s.

There will be steady improvement in other synthetic materials such as hip, knee, and other joints, as well as in other less dramatic materials such as connective tissue replacements. Expect a slow replacement of prosthetic approaches to therapy as natural repair and regeneration processes are better understood and utilized. Expect to see synthetic connective tissue products for tendon repair which contain bioregulatory molecules (BRMs) that stimulate tendon regeneration. Artificial tendons made of both synthetic and/or natural materials will come into use in the late 1980’s to early 1990’s. In short, expect stunning advances in tissue replacement technology for all tissues that have primarily structural function and which are not complicated chemical processing plants, such as the liver or kidneys, or mechanically active such as the heart. In addition to connective tissue and bone, a candidate for early (late 1980’s to early 1990’s) replacement is the cornea. Expect evolution in biocompatible materials to allow for replacement of the cornea with an appropriate plastic, much like the lens of the eye is already replaced with polymer inserts.

1.2.6 Hemodialysis

Advances in hemodialysis will also be very incremental. There may be a gradual shift to peritoneal dialysis (PD) if good drugs to block glucosylation of proteins and inhibit cholesterol deposition are available. The major problem with PD today is that it raises blood sugars to astronomical levels, causing diabetic-like side effects. Inhibition of these side effects may lead to renewed application of this modality.

Direct changes in dialysis are likely to be along the lines of better membrane materials which allow for transport of wastes not currently removable by conventional dialysis and nonthrombogenic surfaces which will reduce the need for anticoagulation. The use of BRMs such as erythropoetin to treat anemia and bone growth factors to treat dialysis bone disease will help to improve the quality and quantity of patient’s lives on dialysis.

Perhaps the biggest advance in this area will be advances in immunology and infectious disease treatment. The ability to administer BRMs to stimulate immune function and improve general health should act to extend dialysis patients’ lives considerably.

…Of course, the biggest improvement in the life expectancy and health of dialysis patients will probably come in the form of the increasing use of transplantation and its application to a wider age range of patients with better long term results.

The most striking revolution in prosthetics will probably occur in dentistry. Expect a whole family of new materials to enter the dental operatory. A workable vaccine against streptococcus mutans should be available by the mid to late 1990’s, greatly reducing the incidence of tooth decay by eliminating the major class of mouth organisms that cause it. Similar advances in prevention and in treatment of gum disease can be expected as well, although probably not as soon. Repairing dental defects will also be revolutionized by the introduction of good, tough, and reliable polymers which will replace metallic amalgams. By the late 1990’s to early 2000’s biocompatible ceramics and coated polymers will be available that will allow for workable single tooth and multitooth gum-implanted prostheses.

1.2.7 Organ Preservation

Ever since the work of people like Mazur, Fahy, and Pegg was published, it has become pretty clear what the constraints are on long term viable cryopreservation of organs: don’t form any significant amount of ice; it injures mechanically and it injures chemically. The problem is that water loves to turn into ice when it’s cooled below 0øC. To circumvent this, a lot of very drastic changes have to be made in the system. Whenever you attempt to make a drastic change in a complicated, interdependent living system — like replacing half the water in it with industrial chemicals — you are in for trouble. The trouble will come in the form of a very tight or narrow window for success: everything will have to be “just right.”

This is where current vitrification technology is now. The existence of such a tight window means that vitrification of large masses will be a technological tour-de-force requiring very sophisticated computer controlled perfusion equipment and exotic and very costly high pressure chambers. Quality control and reliable storage and rewarming of organs will be very costly and difficult.

The future holds the possibility of developing better solute systems which vitrify more easily and which are less toxic (have a wider window for success). It is difficult to predict the pace of advance in this area since it will be arrived at by a mixture of empirical methods and theoretical insights. A big determining factor will be luck. Will the NIH and the Red Cross continue to fund such efforts? And, more to the point, will technological advances in other areas of organ preservation obviate the need for them? If we were betting men, we’d put our dollars on the latter rather than on the former. Major advances in organ preservation (as opposed to cell and tissue preservation) over the next decade will probably be in three areas: 1) Extended hypothermic storage of organs in the 2 to 3 weeks range; 2) Extended normothermic or room temperature storage of organs in the weeks to months range and; 3) mixtures of the above two modalities which yield similar available time courses of storage.

…The next 5 to 10 years should also see major advances in our understanding of the effects of deep hypothermia on the tissues and organs of non-hibernating mammals. These advances should be readily translatable into better flush and perfusion storage techniques for organs. A good understanding of lipid metabolism and mechanisms of cell swelling in deep hypothermia may allow for preservation of organs in the 2øC to 10øC temperature range for periods of several months — thus definitively ending the need for long term solid state preservation of transplantable organs.

1.2.8 Other Approaches to Organ Preservation

One possibility for a major advance over the next two decades is room temperature or hypothermic preservation of organs or organisms using metabolic inhibitors. There have been tantalizing clues in the examination of a wide variety of estivators (animals which go into states of profoundly reduced metabolism at normal temperatures, such as the African lungfish, which can shut off metabolism at temperatures in the range of 30øC to 40øC) that anti-metabolite compounds exist which may be able to induce states of profoundly reduced metabolism at ambient (i.e., 70øF) temperatures.

1.2.9 Genetic therapy

Expect very gradual application of this technology. Early candidates for gene replacement will be in storage diseases such as Lesch-Nyhan, Tay-Sachs, and other “single enzyme missing” disorders. Later applications will include treatments for hypercholesterolemia, some forms of hypertension, and other congenital missing enzyme syndromes. Very late applications (2000 or later) may be in the treatment of a wide range of mental illnesses and cancers.

1.2.10 Prevention

The principal lesson is the lesson of the impact of calorie restriction on overall health, well-being, and lifespan. The basic message here is “you are what you eat.” In terms of treating atherosclerotic disease, the role of prevention is already clear. By reducing fat intake and decreasing serum cholesterol to below 150 mg/dl, most atherosclerotic disease can be avoided. Similarly, basic changes in nutrition such as trace element and vitamin supplementation can greatly reduce the number of late onset malignancies. Eliminating smoking will also be a major factor in achieving this end….Calorie restriction achieved by means of education and therapeutic agents seems the next big area of preventics to be explored by medicine. Expect the development of truly effective anorectics for treatment of gross obesity and eating disorders by the late 1980’s and then secondary use of these for treatment of mild obesity and weight control in the normal middle aged. Products with reduced calories employing fat substitutes such as sucrose polyester should also be entering the marketplace in the early 1990’s and these will help to reduce the calorie load further.

1.2.11 The Downside

A little information is a dangerous thing, and sometimes a lot of information can be an even more dangerous thing. The reason is that progress in therapeutics, which is relatively difficult, always lags far behind progress in diagnosis, which is relatively easy. This imbalance results in a tension which forces premature treatment which often does more harm than good. It is well to note that each new diagnostic modality brings with it a flood of new information which will at first be grossly misused before anyone understands what it means (Harris’s Law of Diagnostics Advance).

A recent example of this sort of thing is the EKG machine, which for the first time showed that many seemingly normal people had strange cardiac rhythms, some of which were seen also around the time people died suddenly of heart problems. Because of this association, for the last 15 years, a number of very powerful drugs have been used to treat people with such rhythms. Many drug-induced fatalities resulted. Unfortunately, only now is it beginning to be understood that most people with good heart function are less in danger from such rhythms than they are from the drugs used to treat them — a finding of little consolation to the people already killed by the drugs.

And on to the economics:

There is a second downside to advanced medicine, of course, besides the danger, that is the cost of “middlingly advanced technology” (such as what we’ll see in the next fifty years) in a society takes a socialistic view of health care. Such as ours.

Non-molecular technology is expensive. It should be obvious to the reader, with a bit of thought that in a world of non-molecular technology, the potential demand for medical care as technology advances, is (for all intents and purposes) Iinfinite. In America, we have adopted the unfortunate policy of letting everyone pay for everyone else’s medical care, which has had exactly the same result as if we had let everyone pool their money and pay for each other’s lunch: everyone orders lobster. We have paid for the lobster only by spreading the costs around to places where they are not obvious. For instance, when you buy an American car, you pay more money for the health care costs of the people who built it than you do the steel that goes into it. This kind of thing can continue very subtly and very insidiously until a very large fraction of the gross national product is eaten up by health care costs. (In our country, it is already 11% and rising).

One day, you may find that you have had to forego your family vacation in order to buy Granny that new AUTODOC which measures 245 different chemicals in her blood every minute and transmits all of the results to Medical Multivac in Bethesda. Of course you may not realize this: all you will know is that the vacation went because money is so tight, taxes are so high, and inflation is so bad. But your money went to Granny nevertheless. The only answer to this problem, short of nanotechnology, is rationing.

But rationing itself becomes the last great social cost of advanced medical technology under socialism, because history shows that it is never done on an individual (person by person) basis. When people do not pay for their own medical care, no one (not doctors, families, or the government) has ever been willing to make the decision of who should benefit from a given technology, and who should not. Therefore, all systems of rationing to control medical costs ultimately have come down in the past to rationing technology across the board…So all of the rosy predictions made in this article must be tempered with the “social” realities that medicine will have to deal with in the next 20 years. Many of the advances we have discussed may simply not materialize because we are not wealthy enough to afford them collectively. That will be a great tragedy.

2 Reactions

On reading all the foregoing, I commented: that was a depressing read. As far as I can tell, they were dead on about the dismal economics, somewhat right about the diagnostics, and fairly wrong about everything else. Which is better than the old predictions listed, only one of which struck me as obviously right (but in a useless way, who actually uses perfluorocarbons for liquid breathing?).

To which Darwin said:

At the time I wrote it I kept saying to myself, almost none of this stuff is going to happen in 20 years - not here anyway. However, I have to come up with something.

Ironically, in the area of cerebral resuscitation, where I am a supposed “expert,” I tried very hard to be realistic and to be both accurate and precise. That was arguably the area where I did the worst - exactly the way other experts fare when they try to predict the future of their fields…So, here I am, 24 years out from making those predictions and I read the crap posted on Less Wrong and on Cryonet and I don’t whether to scream in rage and anger, or weep. How is possible to reach and convince this new generation of cryonics “passivists” that Yudkowsky and Alcor are breeding and make them understand that progress will continue to be unacceptably slow unless the system itself is changed?

See also Fight Aging!’s post, “Overestimating the Near Future”:

…Many of the specific predictions in the article were in fact demonstrated in the laboratory to some degree, and were technically feasible to develop as commercial products by the year 2000, and in some cases earlier but at much greater expense. Certainly there are partial hits for many of the predictions by 2010, in the sense of it being possible, somewhat demonstrated, or in the early stages of being shown to be a practical goal. Yet the regulatory environment in much of the developed world essentially rules out any form of adventurous, rapid, highly competitive development in clinical medicine - such as exists in the electrical engineering, computing, and other worlds. We are cursed therefore with the passage of many years between a new medical technology being demonstrated possible and then attempted in the marketplace … if it ever makes it to the marketplace at all. This must change if we are to see significant progress.

Darwin comments there:

I’ve been going over my original manuscript and surfing the web for specific applications (approved or in process) which meet the criteria of my predictions of 24 years ago. While many of my “lesser” predictions are in fact being realized (often in ways totally unforeseen by us when we wrote the article) overall it is a profoundly depressing experience.

Perhaps nowhere has that been more true than in the areas of aging and cerebral resuscitation - two fields of endeavor I’ve spent a lifetime working on, or intimately involved with those who are. In 1999, we announced that we had achieved repeatable recovery of dogs following 16+ minutes of whole body noromothermic cardiac arrest with no neurological deficit. The enabling molecules and techniques (principally a combination of melatonin, alpha-phenyl-n-tert-butyl-nitrone (PBN), and mild post-cardiac arrest therapeutic hypothermia) all seemed eminently applicable in the (then) immediate future. Indeed, an analog of PBN, 2,3-dihydroxy–6-nitro–7-sulfamoyl-benzo(F)quinoxaline (NBQX) had passed Phase I and II clinical trials for the treatment of stroke with flying colors, and seemed destined for approval.

That was 13 years, ago, and there is still not a single drug available (approved or otherwise) anywhere in the world to treat cerebral ischemia-reperfusion injury - the real killer in cardiac arrest and stroke! Do a literature search on Pubmed for melatonin + cerebral ischemia and you will get ~130 hits - almost all of them dramatically positive. Melatonin is a naturally occurring bioregulatory molecule which is inexpensive and freely available as an over the counter “nutrient.” Even as a stand alone molecule, melatonin is powerfully protective in both global and regional cerebral ischemia, and yet no human application has been forthcoming. It’s been 15 years since our patent on melatonin and other cerebroprotective molecules was issued, 17 years since the patent was applied for, and over 20 years since I made the discovery! Indeed, mild therapeutic hypothermia, made the supposed standard of care for post cardiac arrest neuroinjury nearly a decade ago, is still largely ignored and is used well in only a handful of hospitals worldwide.

What kind of black irony is it that I live in terror of stroke and cardiac arrest (for both myself and my loved ones) and yet the very molecules I discovered to combat them are as unavailable as if they had never been found? Change? Yes, change is certainly needed, and soon.

3 Further reading

Previous Darwin-related posts:

See also Tyler Cowen's The Great Stagnation and “Peter Thiel warns of upcoming (and current) stagnation”.

46 comments

A Crash Course in the Neuroscience of Human Motivation

Sat, 20 Aug 2011 07:15:05 +1000

Submitted by lukeprog • 107 votes • 89 comments

[PDF of this article updated Aug. 23, 2011]

[skip to preface]

Whenever I write a new article for Less Wrong, I'm pulled in two opposite directions.

One force pulls me toward writing short, exciting posts with lots of brain candy and just one main point. Eliezer has done that kind of thing very well many times: see Making Beliefs Pay Rent, Hindsight Devalues Science, Probability is in the Mind, Taboo Your Words, Mind Projection Fallacy, Guessing the Teacher's Password, Hold Off on Proposing Solutions, Applause Lights, Dissolving the Question, and many more.

Another force pulls me toward writing long, factually dense posts that fill in as many of the pieces of a particular argument in one fell swoop as possible. This is largely because I want to write about the cutting edge of human knowledge but I keep realizing that the inferential gap is larger than I had anticipated, and I want to fill in that inferential gap quickly so I can get to the cutting edge.

For example, I had to draw on dozens of Eliezer's posts just to say I was heading toward my metaethics sequence. I've also published 21 new posts (many of them quite long and heavily researched) written specifically because I need to refer to them in my metaethics sequence.¹ I tried to make these posts interesting and useful on their own, but my primary motivation for writing them was that I need them for my metaethics sequence.

And now I've written only four posts² in my metaethics sequence and already the inferential gap to my next post in that sequence is huge again. :(

So I'd like to try an experiment. I won't do it often, but I want to try it at least once. Instead of writing 20 more short posts between now and the next post in my metaethics sequence, I'll attempt to fill in a big chunk of the inferential gap to my next metaethics post in one fell swoop by writing a long tutorial post (a la Eliezer's tutorials on Bayes' Theorem and technical explanation).³

So if you're not up for a 20-page tutorial on human motivation, this post isn't for you, but I hope you're glad I bothered to write it for the sake of others. If you are in the mood for a 20-page tutorial on human motivation, please proceed.

Who knows what I want to do? Who knows what anyone wants to do? How can you be sure about something like that? Isn’t it all a question of brain chemistry, signals going back and forth, electrical energy in the cortex? How do you know whether something is really what you want to do or just some kind of nerve impulse in the brain. Some minor little activity takes place somewhere in this unimportant place in one of the brain hemispheres and suddenly I want to go to Montana or I don’t want to go to Montana.

- Don DeLillo, White Noise

Preface

How do we value things, and choose between options? Philosophers, economists, and psychologists have long tried to answer these questions. But human behavior continues to defy our most subtle models of it, and the algorithms producing our behavior remained hidden in a black box.

But now, neuroscientists are directly measuring the neurons whose firing rates encode value and produce our choices. We know a lot more about the neuroscience of human motivation than you might think. Now we can peer directly into the black box of human motivation, and begin (dimly) to read our own source code.

The neuroscience of human motivation has implications for philosophy of mind and action, for scientific self-help, and for metaethics and Friendly AI. (We don't really know what we want, and looking directly at the algorithms that produce human wanting might help in solving this mystery.)

So, I wrote a crash course in the neuroscience of human motivation.

The purpose of this document is not to argue for any of the conclusions presented within it. That would require not a long blog post but instead a couple 500-page books — say, Foundations of Neuroeconomic Analysis and Handbook of Reward and Decision Making (my two greatest sources for this post).⁴

Instead, I merely want to summarize the current mainstream scientific picture on the neuroscience of human motivation, explain some of the concepts it uses, and tell a few stories about how our current picture of human motivation developed.

As you read this, I hope that many questions and objections will come to mind, because it's not the full story. That's why I went to the trouble of linking to PDFs of almost all my sources (see References): so you can check the full data and the full arguments yourself if you like.

This document is long. You may prefer to read it in sections.

Folk Psychology
Neoclassical Economics
Behaviorism and Reinforcement Learning
Reinforcement Learning and Decision Theory
The Turn to the Brain
Hebbian Learning
Expected Utility in Neurons
Real-Time Updates to Expected Utility
Argmax and Reservation Price
Random Utility
Discounting
Relative and Absolute Utility
Normalization
Are Actions Choices?
The Primate Choice Mechanism: A Brief Review
Marginal Utility and Reference Dependence
Valuation in the Brain
Summary and Research Directions

Notes
References

Folk Psychology

There are these things called 'humans' on planet Earth. They undergo metabolism and cell growth. They produce waste. They maintain homeostasis. They reproduce. They move. They communicate. Sometimes they have pillow fights.

Some of these human processes are 'automatic', like cell growth and breathing. Other processes are 'intentional' or 'willed', like moving and communicating and having pillow fights. We call these latter processes intentional actions, or simply actions. Sometimes we're not sure where to draw the line between automatic processes and actions, but this should become clearer as we learn more. In the meantime, we ask...

How can we explain human actions?

One popular explanation is 'folk psychology.' Folk psychology posits that we humans have beliefs and desires, and that we are motivated to do what we believe will fulfill our desires.

I desire to eat a cookie. I believe I can fulfill that desire if I walk to the kitchen and put one of the cookies there into my mouth. So I am motivated to walk to the kitchen and put a cookie in my mouth.

Of course there are complications. For example I have multiple desires. Suppose I desire to eat a cookie and believe there are cookies in the kitchen. But I also desire to remain sitting comfortably in the living room. Can I satisfy both desires? I also believe that if I nicely ask my friend in the kitchen to bring me a cookie, she will. So I ask her to bring me a cookie and I begin to eat it, without having to leave the comfy living room sofa. We still explain my behavior with constructs like 'beliefs' and 'desires', but we consider more than one of each to do so.

Most of us use folk psychology every day to successfully predict human behavior. I believe that my friend desires to do nice things for me on occasion if they're not too much trouble, and I believe that my friend, once I tell her I want a cookie, will believe she can be nice to me without much trouble if she brings me a cookie from the kitchen. So, I predict that my friend will bring me a cookie when I ask her. So I ask her, and behold! My prediction was correct. I am happily eating a cookie on the sofa.

But folk psychology (FP) faces some problems.⁵ Consider its context in history:

The presumed domain of FP used to be much larger than it is now. In primitive cultures, the behavior of most of the elements of nature were understood in intentional terms. The wind could know anger, the moon jealousy, the river generosity… These were not metaphors… the animistic approach to nature has dominated our history, and it is only in the last two or three thousand years that we have restricted FP’s literal application to the domain of the higher animals.

[Even still,] the FP of the Greeks is essentially the FP we uses today… This is a very long period of stagnation and infertility for any theory to display, especially when faced with such an enormous backlog of anomalies and mysteries in its own explanatory domain… To use Imre Lakatos’ terms, FP is a stagnant or degenerating research program, and has been for millennia.

Consider also its prospects for inter-theoretic reduction:

If we approach homo sapiens from the perspective of natural history and the physical sciences, we can tell a coherent story of its constitution, development, and behavioral capacities which encompasses particle physics, atomic and molecular theory, organic chemistry, evolutionary theory, biology, physiology, and materialistic neuroscience. The story, though still radically incomplete, is already extremely powerful, outperforming FP at many points even in its own domain. And it is deliberately… coherent with the rest of our developing world picture. In short, the greatest theoretical synthesis in [history] is currently in our hands…

But FP is no part of this growing synthesis. Its intentional categories stand magnificently alone, without visible prospect of reduction to that larger corpus. A successful reduction cannot be ruled out, in my view, but FP’s explanatory impotence and long stagnation inspire little faith that its categories will find themselves neatly reflected in the framework of neuroscience. On the contrary, one is reminded of how alchemy must have looked as elemental chemistry was taking form, how Aristotelean cosmology must have looked as classical mechanics was being articulated, or how the vitalist conception of life must have looked as organic chemistry marched forward.

Finally, consider the problem of habit. I sit at my computer and want to type my name, 'Luke.' However, I have just used a special program to switch the function of the keys labeled L and P so that they will input the other character instead (so that I can play a prank on my friend, who will be using my computer shortly). I believe that typing the key labeled L will input P instead, but nevertheless when I type my name my fingers fall into their familiar habit and I end up typing my name as 'Puke.' My act of typing was intentional, and yet I didn't do what I believed would fulfill my desire to type my name.

Folk psychology faces both successes and failures in explaining human action. Hopefully we can do better.

Neoclassical Economics

Folk psychology was updated and quantified by neoclassical economics. To summarize:

One [assumption of] neoclassical economics is "rationality," in which individuals are said to choose alternatives that maximize expected utilities. In particular, the neoclassical view is that individuals rank all possible alternatives according to how much satisfaction they will bring and then choose the alternative that [they expect] will bring the most satisfaction or utility...⁶

Let's review this notion of maximizing expected utility. Suppose I can choose one of two boxes sitting before me, red and blue. There is a 10% chance the red box contains a million dollars, and a 90% chance it contains nothing. As for the blue box, I am certain it contains $10,000. The 'expected value' of choosing the red box is (0.1 × $1,000,000) + (0.9 × $0), which is equal to $100,000. The expected value of choosing the blue box is !1 × $10,000), or $10,000. An agent that chose whatever had the highest expected value would choose the red box, which has 10 times the expected value of the blue box ($100,000 vs. $10,000).

But humans don't value things only according to their dollar value. A million dollars might have 10 times the objective value of $100,000, but it might have less than 10 times the subjective value of $100,000 because after $100,000 you only care a little how much more wealthy you are.

Or, you might be risk averse. You might prefer a sure thing to something that is uncertain. So a 10% chance of a million dollars might be worth less — in subjective value — than a 100% chance of $10,000. If you are risk averse you might choose the blue box because it has higher expected subjective value even though it has lower expected objective value.

We call objective value simply 'value'. We call subjective value 'utility.'

Neoclassical economics quantifies folk psychology by measuring the strength of belief with probability and by measuring the strength of desire with utility. It then says that humans act so as to maximize expected utility, a measure that combines the utility of particular thing with your subjective probability of getting it.⁷

This neoclassical model of human behavior has faced many challenges, and is regularly revised in the face of new evidence.⁸ For example, Loewenstein (1987) found that if students were asked to place a value on the opportunity to kiss a celebrity of their choice 1-5 days in the future, they placed the highest value on a kiss in 3 days. This didn't fit any existing neoclassical models of utility, but was explained in 2001 when Caplin & Leahy (2001) incorporated "anticipatory feeling" into the neoclassical model, explaining that the students got some utility from anticipating the kiss with the celebrity (but also, as usual, discounted the utility of a reward the further away it was in the future), and this is why they didn't want the kiss right away.

Keep in mind that economists don't argue that we actually compute the expected utility of each option before us and then choose the best one, but that we always act "as if" we were doing that.⁹

But sometimes we don't even act "as if" we are obeying the axioms of neoclassical economics. For example, the independence axiom of expected utility theory says that if you prefer an apple over an orange, then you must prefer the Gamble A (72% chance you get an apple, otherwise you get a cat) over the Gamble B (72% chance you get an orange, otherwise you get a cat). But Allais (1953) found that subjects do violate this basic assumption under some conditions.

Such violations of the basic axioms of neoclassical economics led to the development of behavioral economics and theories like Kahneman and Tversky's (1979) prospect theory,¹⁰ which transcends some assumptions of the neoclassical model. But these new theories don't fit the data perfectly, either.¹¹

The models of human motivation we've surveyed so far are conceptually related to decision theory (beliefs and desires, or probabilities and utilities), so I'll call them 'decision-theoretic models' of human motivation. We'll discuss decision-theoretic models again when we finally get to the topic of neuroscience, but for now I want to discuss a different approach to motivation.

Behaviorism and Reinforcement Learning

While neoclassical economists formulated expected utility theory, behaviorist psychologists developed a different set of explanations for human action. Though behaviorists were wrong when they said that science can't talk about mental activity or mental states, you can charitably think of behaviorists as playing a game of Rationalist's Taboo with constructs of folk psychology like "want" or "fear" in order to get at phenomena more appropriate for quantification in technical explanation. Also, the behaviorist approach led to 'reinforcement learning', an important concept in the neuroscience of human motivation.

Before I explain reinforcement learning, let's recall operant conditioning:

Stick a pigeon in a box with a lever and some associated machinery (a "Skinner box"). The pigeon wanders around, does various things, and eventually hits the lever. Delicious sugar water squirts out. The pigeon continues wandering about and eventually hits the lever again. Another squirt of delicious sugar water. Eventually it percolates into its tiny pigeon brain that maybe pushing this lever makes sugar water squirt out. It starts pushing the lever more and more, each push continuing to convince it that yes, this is a good idea.

Consider a second, less lucky pigeon. It, too, wanders about in a box and eventually finds a lever. It pushes the lever and gets an electric shock. Eh, maybe it was a fluke. It pushes the lever again and gets another electric shock. It starts thinking "Maybe I should stop pressing that lever." The pigeon continues wandering about the box doing anything and everything other than pushing the shock lever.

The basic concept of operant conditioning is that an animal will repeat behaviors that give it reward, but avoid behaviors that give it punishment.

Behaviorism died in the wake of cognitive psychology, but its approach to motivation turned out to be very useful in the field of artificial intelligence, where it is called reinforcement learning:

Reinforcement learning is learning what to do — how to map situations to actions — so as to maximize a numerical reward signal. The learner is not told which actions to take, as in most forms of machine learning, but instead must discover which actions yield the most reward by trying them. In the most interesting and challenging cases, actions may affect not only the immediate reward, but also the next situation and, through that, all subsequent rewards. These two characteristics — trial-and-error search and delayed reward — are the two most important distinguishing features of reinforcement learning.

To obtain a lot of reward, a reinforcement learning agent must prefer actions that it has tried in the past and found to be effective in producing reward. But to discover such actions it has to try actions that it has not selected before. The agent has to exploit what it already knows in order to obtain reward, but it also has to explore in order to make better action selections in the future. The dilemma is that neither exploitation nor exploration can be pursued exclusively without failing at the task. The agent must try a variety of actions and progressively favor those that appear to be best..¹²

In addition to the agent and its environment, there are four major components of a reinforcement learning system:

...a policy, a reward function, a value function, and, optionally, a model of the environment.

A policy defines the learning agent's way of behaving at a given time. Roughly speaking, a policy is a mapping from perceived states of the environment to actions to be taken when in those states...

A reward function defines the goal in a reinforcement learning problem. Roughly speaking, it maps perceived states (or state-action pairs) of the environment to a single number, a reward, indicating the intrinsic desirability of the state. A reinforcement-learning agent's sole objective is to maximize the total reward it receives in the long run. ...[A reward function may] be used as a basis for changing the policy. For example, if an action selected by the policy is followed by low reward, then the policy may be changed to select some other action in that situation in the future...

Whereas a reward function indicates what is good in an immediate sense, a value function specifies what is good in the long run. Roughly speaking, the value of a state is the total amount of reward an agent can expect to accumulate over the future starting from that state. Whereas rewards determine the immediate, intrinsic desirability of environmental states, values indicate the long-term desirability of states after taking into account the states that are likely to follow, and the rewards available in those states. For example, a state might always yield a low immediate reward, but still have a high value because it is regularly followed by other states that yield high rewards. Or the reverse could be true...

Rewards are in a sense primary, whereas values, as predictions of rewards, are secondary. Without rewards there could be no values, and the only purpose of estimating values is to achieve more reward. Nevertheless, it is values with which we are most concerned when making and evaluating decisions. Action choices are made on the basis of value judgments. We seek actions that bring about states of highest value, not highest reward, because these actions obtain for us the greatest amount of reward over the long run...

...The fourth and final element of some reinforcement learning systems is a model of the environment. This is something that mimics the behavior of the environment. For example, given a state and action, the model might predict the resultant next state and next reward. Models are used for planning, by which we mean any way of deciding on a course of action by considering possible future situations before they are actually experienced.

Want an example? Here is how a reinforcement learning agent would learn to play Tic-Tac-Toe:

First we set up a table of numbers, one for each possible state of the game. Each number will be the latest estimate of the probability of our winning from that state. We treat this estimate as the state's current value, and the whole table is the learned value function. State A has higher value than state B, or is considered 'better' than state B, if the current estimate of the probability of our winning from A is higher than it is from B. Assuming we always play Xs, then for all states with three Xs in a row the probability of winning is 1, because we have already won. Similarly, for all states with three Øs in a row... the correct probability is 0, as we cannot win from them. We set the initial values of all the other states, the nonterminals, to 0.5, representing an informed guess that we have a 50% chance of winning.

Now we play many games against the opponent. To select our moves we examine the states that would result from each of our possible moves (one for each blank space on the board) and look up their current values in the table. Most of the time we move greedily, selecting the move that leads to the state with greatest value, that is, with the highest estimated probability of winning. Occasionally, however, we select randomly from one of the other moves instead; these are called exploratory moves because they cause us to experience states that we might otherwise never see.

A sequence of Tic-Tac-Toe moves might look like this:¹³

Solid lines are the moves our reinforcement learning agent made, and dotted lines are moves it considered but did not make. The second move was an exploratory move: it was taken even though another sibling move, that leading to e*, was ranked higher.

While playing, the agent changes the values assigned to the states it finds itself in. To improve its estimates concerning the probability of winning from various states, it 'backs up' the value of state after each greedy move to the state before the move (as suggested by the arrows.) What this means is that the value of the earlier state is adjusted to be closer to the value of the later state.

If we let s denote the state before the greedy move, and s' the state after, then the update to the estimated value of s, denoted V(s), can be written:

V(s) <- V(s) + α[V(s') - V(s)]

where α is a small positive fraction called the step-size parameter, which influences the rate of learning. The update rule is an example of a temporal difference learning method, so called because its changes are based on a difference... between [value] estimates at two different times.

...if the step-size parameter is reduced properly over time, this method converges, for any [unchanging] opponent, to the true probabilities of winning from each state give optimal play by the agent.

And that's how a simple version of temporal difference (TD) reinforcement learning works.

Reinforcement Learning and Decision Theory

You may have noticed a key advantage of reinforcement learning: an agent using it can be 'dumber' than a decision-theoretic agent. It can just start with guesses ("What the hell; let's try 50%!") for the value of various states, and then it learns their true values by running through many, many trials.

But what if you don't have many trials to run through, and you need to make an important decision right now?

Then you have to be smart. You need to have a good model of the world and use decision theory to choose the action with the highest expected utility.

This is precisely what rationality — being good at building correct models of the world — is especially good for:

For some tasks, the world provides rich, inexpensive empirical feedback. In these tasks you hardly need reasoning. Just try the task many ways... and take care to notice what is and isn’t giving you results.

Thus, if you want to learn to sculpt, [studying rationality] is a bad way to go about it. Better to find some clay and a hands-on sculpting course. The situation is similar for small talk, cooking, selling, programming, and many other useful skills.

Unfortunately, most of us also have goals for which we can obtain no such ready success/failure data. For example, if you want to know whether cryonics is a good buy, you can’t just try buying it and not-buying it and see which works better. If you miss your first bet, you’re out for good.

Reinforcement learning can be a good strategy if you have time to learn from many trials. If you've only got one shot at a problem, you'd better build up a really accurate model of the world first and then try to maximize expected utility.

Now, back to our story.

It turns out that reinforcement learning seems to underlie many of our mental processes. (More on this later.)

The lesson Yvain drew from this discovery was:

Reinforcement learning is evolution writ small; behaviors propagate or die out based on their consequences to reinforcement in a mind, just as mutations propagate or die out based on their consequences to reproduction in an organism. In the behaviorist model, our mind is not an agent, but a flourishing ecosystem of behaviors both physical and mental, all scrabbling for supremacy and mutating into more effective versions of themselves.

Just as evolving organisms are adaptation-executors and not fitness-maximizers, so minds are behavior-executors and not utility-maximizers.

But things are a bit more complicated than that, as we'll now see.

The Turn to the Brain

I hesitate to say that men will ever have the means of measuring directly the feelings of the human heart. It is from the quantitative effects of the feelings that we must estimate their comparative amounts.

William Jevons (1871)

It turns out that Jevons was wrong. Modern neuroscience allows us to peer into the black box of the human value system and measure directly "the feelings of the human heart."¹⁴

We'll begin with the experiments of Wolfram Shultz. Schultz recorded the activity of single dopamine neurons in monkeys who sat in front of a water spout. At irregular intervals, a speaker played a tone and a drop of water dropped from the spout.¹⁵ The monkeys' dopamine neurons normally fired at the baseline rate, but responded with a burst of activity when water was delivered. Over time, though, the neurons responded less and less to the water and more and more to the tone.

But if Schultz delivered water without first giving the tone, then the dopamine neurons responded with a burst of activity again. And if he played the tone and didn't provide water, the neurons reduced their firing rates below the baseline. The neurons weren't responding to the water itself but to a difference between expected reward and actual reward — a reward prediction error (RPE).

Two other researchers, Read Montague and Peter Dayan, noticed that these patterns of neuronal activity were exactly predicted by TD reinforcement learning theory from computer science.¹⁶ In particular, the RPE observed in neurons appeared to play the same role in monkey learning as the difference between value estimates at two different times did in TD reinforcement learning theory.

Since then, researchers have done many more single-neuron recording studies to test particular versions of TD reinforcement learning and revise the theory until it predicts more and more behavior while also predicting novel experimental discoveries.

Caplin & Dean¹⁷ provided another way to test the hypothesis that dopamine neurons encoded RPE in a TD-class model. They showed that all existing RPE-models could be reduced to three axiomatic statements. If a system violated one of these axioms, it could not be an RPE system. Later, Caplin et al. (2010) tested the axioms on actual brain activity to see if they held up. They did. This is another reason why so many scientists working in this field believe the current 'dopamine hypothesis' — that dopamine neurons encode RPE in a TD-class reinforcement learning system in the brain.

TD-class reinforcement learning works in computers by updating numbers that represent the values of states. How does reinforcement learning work when using nerve cells?

Hebbian Learning

By Hebbian learning, of course. "Cells that fire together, wire together."

Imagine a neural pathway (in one of Pavlov's dogs) that connects the neural circuits that sense the ringing of a bell to the neural circuits for salivation. This is a weak connection at first, which is why the bell doesn't initially elicit salivation.

Also imagine a third neuron that connects the salivation circuit to a circuit that detects food. This is a strong connection, and that's why food does elicit salivation right away:¹⁸

Donald Hebb proposed:

When an axon of cell A is near enough to excite cell B and repeatedly or persistently take part in firing it, a growth process of metabolic change takes place in one or both cells such that A's efficacy, as one of the cells firing B, is increased.¹⁹

In short, whenever two connected cells are active at the same time, the synapses connecting them are strengthened.

Consider Pavlov's experiment. At first, the Bell cell will fire whenever bells ring, but probably not when the salivation cells happen to be active. So, the connection between the Bell cell and the Salivation cell remains weak. But then, Pavlov intervenes and causes the Bell cell and the Salivation cell to fire at the same time by ringing the bell and presenting food at the same time (the Food detector cell already has a strong connection to the Salivation cell). Whenever the Bell cell and the Salivation cell happen to fire at the same time, the synapse between them is strengthened. Once the connection is strong enough, the Bell cell can cause the Salivation cell to fire on its own, just like the Food detector cell can.

It was a fine theory, but it wasn't observed until Bliss & Lomo (1973) observed Hebb's mechanism at work in the rabbit hippocampus. Today, we know how some forms of Hebb's mechanism work at the molecular level.²⁰

Later, Wickens (1993) proposed a similar mechanism called the three-factor rule, according to which some synapses are strengthened whenever presynaptic and postsynaptic activity occurred in the presence of dopamine. These same synapses might be weakened when activity occurred in the absence of dopamine. Later studies confirmed this hypothesis.²¹

Suppose a monkey receives an unexpected reward and encodes a large positive RPE. Glimcher explains:

The TD model tells us that under these conditions we want to increment the value attributed to all actions or sensations that have just occurred. Under these conditions, we know that the dopamine neurons release dopamine throughout the frontocortical-basal ganglia loops, and do so in a highly homogenous manner. That means we can think of any neuron equipped with dopamine receptors as 'primed' for synaptic strengthening. When this happens, any segment of the frontocortical-basal ganglia loop that is already active will have its synapses strengthened.²²

We will return to the dopamine system later, but for now let us back up and pursue the neoclassical economic path into the brain.

Expected Utility in Neurons

Ever since Friedman (1953), economists have insisted that humans only behave as if they are utility maximizers, not that they actually compute expected utility and try to maximize it.

It was a surprise, then, when neuroscientists stumbled upon the neurons that were encoding expected utility in their firing rates.

Tanji & Evarts (1976) did their experiments with rhesus monkeys because they are our closest relative besides the apes, and this kind of work is usually forbidden on apes for ethical reasons (we need to implant a recording electrode in the brain).

The monkeys were trained to know that a colored light on the screen meant they would soon be offered a reward (a drop of water) either for pushing or pulling, but not for both. This was the ‘ready’ cue. A second later, researchers gave a ‘direction’ cue that told the monkeys which action — push or pull — was going to be rewarded. The third cue was the 'go' signal: if the monkey made the previously indicated movement, it was rewarded.

This is what they saw:

At the ‘ready’ cue, the neurons associated with a pushing motion became weakly active (but fired above the baseline rate), and so did the neurons associated with a pulling motion. When the ‘direction’ cue was given, the neurons associated with the to-be-rewarded motion doubled their firing rate, and the neurons associated with the opposite motion fell back to the baseline rate. Then at the ‘go’ cue, the neurons associated with the to-be-rewarded movement increased again rapidly, up past the threshhold required to produce movement, and the movement was produced shortly thereafter.

One tempting explanation of the data is that after the ‘ready’ cue, the monkey’s brain 'decides' there’s a 50% chance that pulling will get the reward, and a 50% chance that pushing will get the reward. That’s why we see the neuron firing rates associated with those two actions each jump to slightly less than 50% of the movement threshold when the ‘ready’ cue is given. But then, when the ‘direction’ cue is given, those expectations shift to 100%/0% or 0%/100%, depending on which action is about to be rewarded according to the ‘direction’ cue. That’s why activity in the circuit associated with the to-be-rewarded action doubles and the other one drops to baseline. And then the ‘go’ cue is delivered and firing rates blast past the movement threshold, and movement is produced.

Let's jump ahead to Basso & Wurtz (1997), who did a similar experiment except that they used voluntary eye movements (called ‘saccades’) instead of voluntary arm movements. And this time, they presented each monkey with one, two, four, or eight possible targets, instead of just two targets (push and pull) like Tanji & Evarts did.

What they found was that as more potential targets were presented, the magnitude of the preparatory activity associated with each target systematically decreased. And again, once the ‘direction’ and ‘go’ cues were presented, the activity associated with those other potential targets dropped rapidly and activity burst rapidly in neurons associated with the to-be-rewarded movement. It was as though the monkeys’ brains were distributing their probability mass evenly across the potentially rewarded actions, and then once they knew which action should in fact be rewarded, they moved all their probability mass to that action and performed the action and got the reward.

Real-Time Expected Utility Updates

Other researchers showed monkeys a black screen with flickering white dots on it. In each frame of the video, the computer moved each dot in a random direction. The independent variable was a measure called 'coherence.' In a 100% leftward coherence condition, all dots moved to the left. In a 60% rightward condition, 60% of the dots move rightward while the rest moved randomly. And so on.

In a typical experiment, the researchers would identify a neuron in a monkey's brain that increased its firing rate in response to rightward coherence of the dots, and decreased its firing rate in response to leftward coherent of the dots. Then they would present the monkey with a sequence (in random order) of every possible leftward and rightward coherence condition.

A leftward coherence (of any magnitude) meant the monkey would be rewarded for leftward eye movement, and a rightward coherence meant the monkey would be rewarded for rightward eye movement. But, the monkey had to wait two seconds before being rewarded.

In this experiment, the probabilities always started at 50% but then updated continuously. A 100% rightward coherence condition allowed the monkey to very quickly know which voluntary eye movement would be rewarded, but in a 5% rightward coherence condition the expected utility of the rightward target grew more slowly.

The results? The greater the coherence of rightward motion of the dots, the faster the neurons associated with rightward eye movement increased their firing rate. (A higher coherence meant the monkey was able to update its probabilities more quickly.)

Argmax and Reservation Price

Many studies show that the brain controls movement by way of a 'winner take all' mechanism that is isomorphic to the argmax operation from economics.²³ That is, there are many possibilities competing for your final choice, but just before your choice the single strongest signal remains after all the others are inhibited.

This choice mechanism was investigated in more detail by Michael Shadlen and others.²⁴ Shadlen gave monkeys the same eye movement task as above, except that the monkeys could make their choice at any time instead of waiting for two seconds. He found that:

When the direction of the dots is unambiguous, monkeys make their choices quickly.
As the direction of the dots becomes more ambiguous, they take longer to make their choices.
Throughout the experiment, the firing rates of neurons in the LIP (part of the 'final common path' for generating eye movement) grew toward a specific threshold level.

The threshold level acts as a kind of criterion of choice. Once the criterion is met, action is taken. Or in economic terms, the monkeys seemed to set a reservation price on making certain movements.²⁵

Random Utility

When deciding between goods of different expected utilities, humans exhibit a stochastic transfer function:

Consider a human subject choosing between two objects of highly different expected utilities, such as a first lottery with a 50% chance of winning $5 and a second lottery with a 25% chance of winning $5. We observe highly deterministic behavior under these conditions: basically all subjects always choose the 5o% chance of winning $5. But what happens when we increment the value of the 25% lottery? As the amount one stands to win from that lottery is incremented, individual subjects eventually switch their preference. Exactly when they make that switch depends on their idiosyncratic degree of risk aversion. What is most interesting about this behavior for these purposes, though, is that actual human subjects, when presented with this kind of choice repeatedly, are never completely deterministic. As the value of the 25% lottery increases, they begin to show probabilistic behavior — selecting the 25% lottery sometimes, but not always.²⁶

Our behavior has an element of randomness in it. Daniel McFadden won a Nobel Prize in economics for capturing such behavior using a random utility model.²⁷ The way he did it is to suppose that when a chooser asks himself what a thing is worth, he doesn't get a fixed answer but a variable one. That is, there is actual variation in his preferences. Thus, his expected utility for a particular lottery is drawn from a distribution of possible utilities, usually one with a Gaussian variance.²⁸

This behavior makes sense when we think about the human choice mechanism at the neuronal level, because neuron firing rates are stochastic.²⁹ When a neurobiologist says "The neuron was firing at 200 Hz," what she means is that the mean firing rate of the neuron over a long time and stable conditions would have been close to 200 Hz. So the neurons that encode utility (wherever they are) will exhibit stochasticity, and thereby introduce some randomness into our choices. In this way, neurobiological data constrains our economic models of human behavior. An economic model without some randomness in it will have difficulty capturing human choices for as long as humans run on neurons.³⁰

Discounting

Louie & Glimcher (2010) examined temporal discounting in the brain. The two monkeys in this study were repeatedly asked to choose between a small, immediately available reward and a larger reward available after a small delay. For example, on one day they were asked to choose between 0.13 millileters of juice right now, or else 0.2 millileters of juice available after a delay of 2, 4, 8, or 12 seconds. A monkey might be willing to wait 2, 4, or 8 seconds for the larger reward, but not 12 seconds.

After many, many measurements of this kind, Louie and Glimcher were able to describe the discounting function being used by each monkey. (One of them was more impatient than the other.)

Moreover, the neurons in the relevant section of the brain fired at rates that reflected each monkey’s discounting function. If 0.2 millileters of juice was offered with no delay, the neurons were highly active. If the same reward was offered at a delay of 2 seconds, they were slightly less active. If the same reward was offered after 4 seconds, the neurons were less active still. And so on. As it turned out, the discounting function that captured their choices was identical to the discounting function that captured the firing rates of these neurons.

This shouldn't be a surprise at this point, but just to confirm: Yes, we can observe discounting in the firing rates of neurons involved in the choice-making process.

Relative and Absolute Utility

Dorris & Glimcher (2004) observed monkeys and their choice mechanism neurons while the monkeys engaged in repeated plays of the inspection game. The study is too involved for me to explain here, but the results suggested that choice mechanism neurons encode relative expected utilities (relative to other actions under consideration) rather than absolute expected utilities.

Tobler et al. (2005) suggested that the brain only encodes relative expected utilities. But there is reason to suspect this can't be right. If we stored only relative expected utilities, then we would routinely violate the axiom of transitivity (if you prefer A to B and B to C, you can't also prefer C to A). To see why this is the case, consider Glimcher's example (he says 'expected value' instead of 'utility'):

...consider a subject trained to choose between objects A and B, where A is $1,000,000 worth of goods and B is $100,000 worth of goods... A system that represented only the relative expected subjective value of A and B would represent SV(A) > SV(B). Next, consider training the same subject to choose between C and D, where C is $1,000 worth of goods and D is $100 worth of goods. Such a system would represent SV(C) > SV(D). What happens when we ask a chooser to select between B and C? For a chooser who represents only relative expected subjective value, the choice should be C: she should pick $1,000 worth of goods over $100,000 worth of goods because it has a higher learned relative expected subjective value. In order for our chooser to... construct transitive preferences across choice sets (and to obey the continuity axiom)... it is required that somewhere in the brain she represent the absolute subjective values of her choices.³¹

And we mostly do seem to obey the axiom of transitivity.

So if the choice mechanism neurons do represent relative utilities, then some other neurons elsewhere must encode a more absolute form of utility. Other implications of this are explored in the next section.

Normalization

David Heeger showed³² that the firing rates of 'feature detector' neurons in the visual cortex captured a response to a feature in the visual field divided by the sum of the activation rates of nearby neurons sensitive to the same image. Thus, these neurons encode not only whether they 'see' the feature they are built to detect, but also how unique it is in the visual field.

The effect of this is that neurons reacting to the edge of a visual object fire more actively than others do. Behold! Edge detection!

It's also an efficient way to encode information about the world. Consider a world where orange dots are ubiquitous. For an animal in that world, it would be wasteful to fire action potentials to represent orange dots. Better to represent the absence of orange dots, or the transition from orange dots to something else. An optimally efficient encoding method would be sensitive not to the 'alphabet' of all possible inputs, but to a smaller alphabet of the inputs that actually appear in the world. This insight was mathematically formalized by Schwartz & Simoncelli (2001).

The efficiency of this normalization technique may explain why we've discovered it at work in so many different places in the brain.³³ And given that we've found it almost everywhere we've looked for it, it wouldn't be a surprise to see it show up in our choice-making circuits. Indeed, Simoncelli & Schwartz's normalization equation may be what our brains use to encode expected utilities that are relative to the other choices under consideration.

One implication of their equation is that a chooser's errors become more frequent as the size of the choice set grows. Thus, behavioral errors on small choice sets should be rarer than might be predicted by most random utility models, but error rates will increase rapidly with choice set size (and beyond a certain choice set size, choices will appear random).

Preliminary evidence that choice set size effects error rates has arrived from behavioral economics. For example, consider Iyengar & Lepper's (2000) study of supermarket shoppers. They set up a table showing either 6 or 24 flavors of jams, allowing shoppers to sample as many as they wanted. Customers who saw 24 flavors had a 3% chance of buying a jar, while those who saw only 6 flavors had a 30% chance!

In another experiment, Iyengar & Lepper let subjects choose one of either 6 or 30 different chocolates. Those who chose from among only 6 options were more satisfied with their selection than those who had been presented with 30 different chocolates.

These data fit our expectation that as the choice set grows, the frequency of errors in our behavior rises and the likelihood that an option will rise above the threshold for purchase drops. When Louie & Glimcher (2010) investigated this phenomena in monkey choice mechanism neurons, they found it at work there, too. But the process of choice-set editing is still poorly understood, and some recent studies have failed to replicate Iyengar & Lepper's results (Scheibehenne et al. 2010).

Perhaps the most surprising implication of these findings is that because of neuronal stochasticity, and because errors increase as the choice set grows, we should expect stochastic violations of the independence axiom, and that when choosers face very large choice sets they will essentially ignore the independence axiom.

This is a prediction about human behavior not made by earlier models from neoclassical economics, but it is suggested by looking at the neurons involved in human choice-making.

Are Actions Choices?

But all these data come from experiments where the choices are actions, and from our knowledge of the brain's "final common path" for producing actions. How do actions map on to choices about lovers and smartphones?

Studies by Greg Horowitz have provided some relevant data, because monkeys had to choose options identified by color rather than by action.³⁴ For example in one trial, a 'red' option might offer one reward and a 'green' option might offer a different reward. On each trial, the red and green options would appear at random places on the computer screen, and the monkey could choose a reward with a voluntary eye movement. The key here is that rewards were chosen by color and not by a (particular) action.

Horowitz found that the choice mechanism neurons showed the same pattern of activation under these conditions as was the case under action-based choice tasks.

So, it looks like the valuation circuits can store the value of a colored target, and these valuations can be mapped to the choice mechanism. But we don't know much about how this works, yet.

The Primate Choice Mechanism: A Brief Review

Thus far, we have mostly discussed the primate brain's choice mechanism. To review:

The choice circuit resides in the final common pathway for action.
It takes as its input a signal that encodes stochastic expected utility, a concept aligned to the random utility term in economic models proposed by McFadden (2005) and Gul & Pesendorfer (2006).
This input signal is represented by a normalized firing rate (with Poisson variance, like all neurons).
As the choice set size grows, so does the error rate.
Final choice is implemented by an argmax function or a reservation price mechanism. (A single circuit can achieve both modes.³⁵)

But how are probability and utility calculated such that they can be fed into the expected utility representations of the choice mechanism? I won't discuss how the brain forms probabilistic beliefs in this article,³⁶ so let us turn to the study of how utility is calculated in the brain: the question of valuation.

Marginal Utility and Reference Dependence

Consider the following story:

Imagine an animal exploring a novel environment from a nest on a day when both (1) its blood concentration is dilute (and thus its need for water is low) and (2) its blood sugar level is low (and thus its need for food is high). The animal travels west one kilometer from the nest and emerges from the undergrowth into an open clearing at the shores of a large lake. Not very thirsty, the animal bends down to sample the water and finds it... unpalatable... the next day the same animal leaves its nest in the same metabolic state and travels one kilometer to the east, where it discovers a grove of trees that yield a dry but nutritious fruit, a grove of dried apricot trees. It samples the fruit and finds it sweet and highly palatable.

What has the animal actually learned about the value of going west and the value of going east? It has had a weakly negative experience, in the psychological sense, when going west and a very positive experience when going east. Do these subjective properties of its experience influence what it has learned? Do the stored representations derived from these experiences encode the actual objective values of going west and east, or do they encode the subjective experiences? That is a critical question about what the animal has learned, because it determines what it does when it wakes up thirsty. When it wakes up thirsty it should, in a normative sense, go west towards the... lake, despite the fact that its previous visit west was a negative experience.³⁷

Economists have known this problem for a long time, and solved it with an idea called marginal utility.

In neoclassical economics, we view the animal as having two kinds of 'wealth': a sugar wealth and a water wealth (the total store of sugar and water in the animal's body at a given time). A piece of fruit or a sip of water is an increment in the animal's total sugar or water wealth. The utility of a piece of fruit or a sip of water, then, depends on its current levels of sugar and water wealth.

On day one, the animal's need for water is low and its need for sugar is high. On that day, the marginal utility of a piece of fruit is greater than the marginal utility of a sip of water. But suppose during the next week the animal has a high blood sugar level. At that time, the marginal utility of a piece of fruit is low. Thus, the marginal utility of a consumable resource depends on wealth. The wealthier the chooser, the lower the marginal utility provided by a fixed amount of gain ('diminishing marginal utility').

In neoclassical economics, the animal faced with the option of going east or west in the morning would first estimate how much the water and the fruit would change its objective wealth level, and then it would estimate how much those objective changes in wealth would change its utility. That is, it would use objective values to compute its marginal (subjective) utility. If it only had access to the subjective experiences in our story, it couldn't compute a new marginal utilities when it finds itself unexpectedly thirsty.

The problem with this solution is that the brain does not appear to encode the objective values of stimuli, and humans behaviorally don't seem to respect the objective values of options either, as discussed here.

In response to the behavioral evidence, Kahneman & Tversky (1979) developed a reference dependent utility function to describe human behavior: prospect theory. Their suggestion was, basically:

Rather than computing marginal utilities against [objective] wealth as in [standard neoclassical economic models], utilities (not marginal utilities) could be computed directly as deviations from a baseline level of wealth, and then choices could be based on direct comparisons of these utilities rather than on comparisons of marginal utilities. Their idea was to begin with something like the chooser's status quo, how much wealth he thinks he has. Each gamble is then represented as the chance of winning or losing utilities relative to that status-quo-like reference point.³⁸

This fits with the neurobiological fact that we encode signals from external stimuli relative to reference points, and don't have access to the objective values of stimuli.

The advantage of the neoclassical economic model is that it keeps a chooser's choices consistent. The advantage of the reference-dependent approach is that it better fits human behavior and human neurobiology.

Most neoclassical economists seem to ignore the problems for their theories that are presented by reference dependence in human behavior and human neurobiology, but two neoclassical economists at Berkeley, Matthew Rabin and Botond Koszegi, have begun to take reference dependence seriously. As they put it:

...while an unexpected monetary windfall in the lab may be assessed as a gain, a salary of $5o,000 to an employee who expected $60,000 will not be assessed as a large gain relative to status-quo wealth, but rather as a loss relative to expectations of wealth. And in nondurable consumption — where there is no object with which the person can be endowed — a status-quo-based theory cannot capture the role of reference dependence at all: it would predict, for instance, that a person who misses a concert she expected to attend would feel no differently than somebody who never expected to see the concert.³⁹

Their reference-dependent model makes particular predictions:

[Our theory] shows that a consumer's willingness to pay a given price for shoes depends on the probability with which she expected to buy them and the price she expected to pay. On the one hand, an increase in the likelihood of buying increases a consumer's sense of loss of shoes if she does not buy, creating an "attachment effect" that increases her willingness to pay. Hence, the greater the likelihood she thought prices would be low enough to induce purchase, the greater is her willingness to buy at higher prices. On the other hand, holding the probability of getting the shoes fixed, a decrease in the price a consumer expected to pay makes paying a higher price feel like more of a loss, creating a "comparison effect" that lowers her willingness to pay the high price. Hence, the lower the prices she expected among those prices that induce purchase, the lower is her willingness to buy at higher prices.

Thus, the cost of accepting the human fact of reference-dependence is that we have to admit that humans are irrational (in the sense of 'rationality' defined by the axioms of revealed preference):

The fact that a consumer will pay more for shoes she expected to buy than for shoes she did not expect to buy, or that an animal would prefer inferior fruit it expected to eat over superior fruit it did not expect to eat, is exactly the kind of irrational behavior that we might hope the pressures of evolution would preclude. What observations tell us, however, is that these behaviors do occur. The neuroscience of sensory encoding tells us that these behaviors are an inescapable product of the fundamental structure of our brains.⁴⁰

But really, shouldn't it have been obvious all along that humans are irrational? Perhaps it is, to everyone but neoclassical economists and Aristoteleans. (Okay, enough teasing...)

One thing to keep in mind is that the brain encodes information about the external world in a reference-dependent way because that method makes a more efficient use of neurons. So evolution traded away some rationality for greater efficiency in the encoding mechanism.

Valuation in the Brain

Back to dopamine. Earlier, we learned that the brain learns the values of their actions with a dopaminergic reward system that uses something like temporal difference (TD) reinforcement learning. This reward system updates the stored values for actions by generating a reward prediction error (RPE) from the difference between expected reward and experience reward, and propagating this learning throughout relevant structures of the brain using the neurotransmitter dopamine. In particular, some synapses are strengthened whenever presynaptic and postsynaptic activity occur in the presence of dopamine, as proposed by Wickens (1993).

But we haven't yet discussed how utilities for actions are generated in the first place, or how they are stored (independent of the expected utilities represented during the choice process). It feels like I generally want ice cream a little bit and hot sex a lot more. Where is that information stored?

Dozens⁴¹ of fMRI studies show that two brain regions in particular are correlated with subjective value: the ventral striatum and the medial prefrontal cortex. Other studies suggest that at least five more brain regions probably also contribute to the valuation process: the orbitofrontal cortex, the dorsolateral prefrontal cortex, the amygdala, the insula, and the anterior cingulate cortex.

There are many theories about how the human brain generates and stores utilities, but these theories are far more speculative and in their infancy than everything else I've presented in this tutorial, so I won't discuss them here. Instead, let us conclude with a summary of what neuroscientists know about the human brain's motivational system, and what some of the greatest open questions are.

Summary and Research Directions

Here's what we've learned:

Utilities are real numbers ranging from 0 to 1,000 that take action potentials per second as their natural units. (By 'utility' here I don't mean what's usually meant by the term, I just mean 'utility' for the purpose of predicting choice by measuring the firing rates of certain populations of neurons in the final common path of the choice circuit in the human brain.)
Mean utilities are mean firing rates of specific populations of neurons in the final common path of human choice circuits.
Mean utilities predict choice stochastically, similar to random utility models from economics.
Utilities are encoded cardinally in firing rates relative to neuronal baseline firing rates. (This is opposed to post-Pareto, ordinal notions of utility.)
The choice circuit takes as its input a firing rate that encodes relative (normalized) stochastic expected utility.
As the choice set size grows, so does the error rate.
Final choice is implemented by an argmax function or a reservation price mechanism.

Paul Glimcher lists⁴² the greatest open questions in the field as:

Where is utility stored and how does it get to the choice mechanism?
How does the brain decide when it's time to choose?
What is the neural mechanism that allows us to substitute between two goods at a certain point?
How are probabilistic beliefs represented in the brain?
Utility functions are state-dependent, so how do state and utility function interact?

Later, we'll explore the implications of our findings for metaethics. As of August 2011, if you've read this then you probably know more about how human values actually work than almost every professional metaethicist on Earth. The general lesson here is that you can often out-pace most philosophers simply by reading what today's leading scientists have to say about a given topic instead of reading what philosophers say about it.

Notes

¹ They are: Less Wrong Rationality and Mainstream Philosophy, Philosophy: A Diseased Discipline, On Being Okay with the Truth, The Neuroscience of Pleasure, The Neuroscience of Desire, How You Make Judgments: The Elephant and its Rider, Being Wrong About Your Own Subjective Experience, Intuition and Unconscious Learning, Inferring Our Desires, Wrong About Our Own Desires, Do Humans Want Things?, Not for the Sake of Pleasure Alone, Not for the Sake of Selfishness Alone, Your Evolved Intuitions, When Intuitions Are Useful, Cornell Realism, Railton's Moral Reductionism (Part 1), Railton's Moral Reductionism (Part 2), Jackson's Moral Functionalism, Moral Reductionism and Moore's Open Question Argument, and Are Deontological Moral Judgments Rationalizations?

² Heading Toward: No-Nonsense Metaethics, What is Metaethics?, Conceptual Analysis and Moral Theory, and Pluralistic Moral Reductionism.

³ I tried something similar before, with Cognitive Science in One Lesson.

⁴ Glimcher (2010) offers the best coverage of the topic in a single book. Tobler & Kobayashi (2009) offer the best coverage in a single article.

⁵ The quotes in this section are from Churchland (1981).

⁶ Allen & Ng (2004).

⁷ This perspective goes back at least as far back as Arnauld (1662), who wrote:

To judge what one must do to obtain a good or avoid an evil, it is necessary to consider not only the good and the evil in itself, but also the probability that it happens or does not happen: and to view geometrically the proportion that all these things have together.

⁸ In addition to Caplin & Leahy (2001), see Kreps & Porteus' (1978, 1979) incroporation of the "utility of knowing", Loomes & Sugden's (1982) incorporation of "regret", Gul & Pesendorfer's (2001) incorporation of "the cost of self-control", and Koszegi & Rabin's (2007, 2009) incorporation of the "reference point".

⁹ Friedman (1953).

¹⁰ See a review in Fox & Poldrack (2009).

¹¹ For one difficulty with prospect theory, see Laury & Holt (2008).

¹² Sutton & Barto (2008), p. 3. All quotes from this section are from the early pages of this book.

¹³ From Sutton & Barto (2008).

¹⁴ Much of the rest of this post is basically a summary and paraphrase of Glimcher (2010).

¹⁵ Mirenowicz & Schultz (1994).

¹⁶ Schultz et al. (1997).

¹⁷ Caplin & Dean (2007)

¹⁸ From Glimcher (2010).

¹⁹ Hebb (1949).

²⁰ Malenka & Bear (2004).

²¹ Reynolds & Wickens (2002).

²² Glimcher (2010), p. 341.

²³ Edelman & Keller (1996); Van Gisbergen et al. (1987).

²⁴ Gold and Shadlen (2007); Roitman and Shadlen (2002).

²⁵ Simon (1957).

²⁶ Glimcher (2010), p. 215.

²⁷ McFadden (2000). The behavior of gradually transitioning between two choices is described by Selten (1975).

²⁸ For a probably improved random utility model, see Gul & Pesendorfer (2006).

²⁹ Dean (1983); Werner & Mountcastle (1963).

³⁰ Unless some other feature of the brain turns out to 'smooth out' the stochasticity of neurons involved in valuation and choice-making.

³¹ Glimcher (2010).

³² Heeger (1992, 1993); Carandini & Heeger (1994); Simoncelli & Heeger (1998).

³³ Carandini & Heeger (1994); Britten & Heuer (1999); Zoccolan et al. (2005); Louie & Glimcher (2010).

³⁴ Horowitz & Newsome (2001a, 2001b, 2004).

³⁵ Liu & Wang (2008).

³⁶ But, see Deneve (2009).

³⁷ Glimcher (2010), p. 281.

³⁸ Glimcher (2010), p. 283.

³⁹ This quote and the next quote are from Koszegi & Rabin (2006).

⁴⁰ Glimcher (2010), p. 292.

⁴¹ I won't list them all here. For an overview, see Glimcher (2010), ch. 14.

⁴² Glimcher (2010), ch. 17. I've paraphrased his open questions. I also excluded his 6th question: What Is the Neural Organ for Representing Money?

References

Allais (1953). Le comportement de l'homme rationel devant le risque. Critique des postulates et axiomes de l'ecole americaine. Econometrica, 21: 503-546.

Allen & Ng (2004). Economic behavior. In Spielberger (ed.), Encyclopedia of Applied Psychology, Vol. 1 (pp. 661-666). Academic Press.

Arnauld (1662). Port-Royal Logic.

Basso & Wurtz (1997). Modulation of neuronal activity in superior colliculus by changes in target probability. Journal of Neuroscience, 18: 7519-7534.

Britten & Heuer (1999). Spatial summation in the receptive fields of MT neurons. Journal of Neuroscience, 19: 5074-5084.

Caplin & Dean (2007). Axiomatic neuroeconomics.

Caplin, Dean, Glimcher, & Rutledge (2010). Measuring beliefs and rewards: a neuroeconomic approach. Quarterly Journal of Economics, 125: 3.

Caplin & Leahy (2001). Psychological expected utility theory and anticipatory feelings. Quarterly Journal of Economics, 116: 55-79.

Carandini & Heeger (1994). Summation and devision by neurons in primate visual cortex. Science, 264: 1333-1336.

Churchland (1981). Eliminative materialism and the propositional attitudes. The Journal of Philosophy, 78: 67-90.

Dean (1983). Adaptation-induced alteration of the relation between response amplitude and contrast in cat striate cortical neurons. Vision Research, 23: 249-256.

Deneve (2009). Bayesian decision making in two-alternative forced choices. In Dreher & Tremblay (eds.), Handbook of Reward and Decision Making (pp. 441-458). Academic Press.

Dorris & Glimcher (2004). Activity in posterior parietal cortex is correlated with the subjective desireability of an action. Neuron, 44: 365-378.

Edelman & Keller (1996). Activity of visuomotor burst neurons in the superior colliculus accompanying express saccades. Journal of Neurophysiology, 76: 908-926.

Fox & Poldrack (2009). Prospect theory and the brain. In Glimcher, Camerer, Fehr, & Poldrack (eds.), Neuroeconomics: Decision Making and the Brain (pp. 145-173). Academic Press.

Friedman (1953). Essays in Positive Economics. University of Chicago Press.

Glimcher (2010). Foundations of Neuroeconomic Analysis. Oxford University Press.

Gold and Shadlen (2007). The neural basis of decision making. Annual Review of Neuroscience, 30: 535-574.

Gul & Pesendorfer (2001). Temptation and self-control. Econometrica, 69: 1403-1435.

Gul & Pesendorfer (2006). Random expected utility. Econometrica, 74: 121-146.

Hebb (1949). The organization of behavior. Wiley & Sons.

Heeger (1992). Normalization of cell responses in cat striate cortex. Visual Neuroscience, 9: 181-197.

Heeger (1993). Modeling simple-cell direction selectivity with normalized, half-squared linear operators. Journal of Neurophysiology, 70: 1885-1898.

Horowitz & Newsome (2001a). Target selection for saccadic eye movements: direction selective visual responses in the superior colliculus induced by behavioral training. Journal of Neurophysiology, 86: 2527-2542.

Horowitz & Newsome (2001b). Target selection for saccadic eye movements: prelude activity in the superior colliculus during a direction discrimination task. Journal of Neurophysiology, 86: 2543-2558.

Iyengar & Lepper (2000). When choice is demotivating: Can one desire too much of a good thing? Journal of Personality and Social Psychology, 79: 995-1006.

Jevons (1871). The Theory of Political Economy. Macmillan and Co.

Kahneman & Tversky (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47: 263-291.

Koszegi & Rabin (2006). A model of reference-dependent preferences. Quarterly Journal of Economics, 121: 1133-1165.

Koszegi & Rabin (2007). Reference-dependent risk attitudes. American Economic Review, 97: 1047-1073.

Koszegi & Rabin (2009). Reference-dependent consumption plans. American Economic Review, 99: 909-936.

Kreps & Porteus (1978). Temporal resolution of uncertainty and dynamic choice theory. Econometrica, 46: 185-200.

Kreps & Porteus (1979). Dynamic choice theory and dynamic programming. Econometrica, 47: 91-100.

Laury & Holt (2008). Payoff scale effects and risk preference under real and hypothetical conditions. In Plott & Smith (eds.), Handbook of Experimental Economic Results, Vol. 1 (pp. 1047-1053). Elsevier Press.

Loewenstein (1987). Anticipation and the valuation of delayed consumption. Economic Journal, 97: 666-684.

Liu & Wang (2008). A common cortical circuit mechanism for perceptual categorical discrimination and veridical judgment. PLOS Computational Biology, 4: 1-14.

Loomes & Sugden (1982). Regret theory: An alternative theory of rational choice under uncertainty. Economic Journal, 92: 805-824.

Louie & Glimcher (2010). Separating value from choice: delay discounting activity in the lateral intraparietal area. Journal of Neuroscience, 30: 5498-5507.

Malenka & Bear (2004). LTP and LTD: an embarrassment of riches. Neuron, 44: 5–21.

Mirenowicz & Schultz (1994). Importance of unpredictability for reward responses in primate dopamine neurons. Journal of Neurophysiology, 72: 1024-1027.

Reynolds & Wickens (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks, 15: 507-521.

Roitman and Shadlen (2002). Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. Nature Neuroscience, 22: 9475-9489.

Scheibehenne, Greifeneder, & Todd (2010). Can there ever be too many options? A meta-analytic review of choice overload. Journal of Consumer Research, 37: 409-425.

Schultz, Dayan, & Montague (1997). A neural substrate of prediction and reward. Science, 275: 1593–1599.

Schwartz & Simoncelli (2001). Natural signal statistics and sensory gain control. Nature Neuroscience, 4: 819-825.

Selten (1975). Reexamination of perfectness concept for equilibrium points in extensive games. International Journal of Game Theory, 4: 25-55.

Simoncelli & Heeger (1998). A model of neuronal responses in visual area MT. Vision Research, 38: 743-761.

Sutton & Barto (2008). Reinforcement Learning: An Introduction. MIT Press.

Tanji & Evarts (1976). Anticipatory activity of motor cortex neurons in relation to direction of an intended movement. Journal of Neurophysiology, 39: 1062-1068.

Tobler & Kobayashi (2009). Electrophysiological correlates of reward processing in dopamine neurons. In Dreher & Tremblay (eds.), Handbook of Reward and Decision Making (pp. 29-50). Academic Press.

Van Gisbergen, Opstal, & Tax (1987). Collicular ensemble coding of saccades based on vector summation. Neuroscience, 21: 651.

Werner & Mountcastle (1963). The variability of central neural activity in a sensory system, and its implications for central reflection of sensory events. Journal of Neurophysiology, 26: 958-977.

Wickens (1993). A Theory of the Striatum. Pergamon Press.

Zoccolan, Cox, & DiCarlo (2005). Multiple object response normalization in monkey inferotemporal cortex. Journal of Neuroscience, 25: 8150-8164.

89 comments

Do Humans Want Things?

Thu, 04 Aug 2011 15:00:40 +1000

Submitted by lukeprog • 23 votes • 52 comments

Summary: Recent posts like The Neuroscience of Desire and To what degree do we have goals? have explored the question of whether humans have desires (or 'goals'). If we don't have desires, how can we tell an AI what kind of world we 'want'? Recent work in economics and neuroscience has clarified the nature of this problem.

We begin, as is so often the case on Less Wrong, with Kahneman & Tversky.

In 1981, K&T found that human choice was not always guided by the objective value of possible outcomes, but by the way those outcomes were 'framed'.¹ For example in one study, K&T told subjects the following story:

Imagine that the U.S. is preparing for the outbreak of an unusual Asian disease, which is expected to kill 600 people. Two alternative programs to combat the disease have been proposed.

Half the participants were given the following choice:

If program A is adopted, 200 people will be saved. If Program B is adopted, there is a 1/3 probability that 600 people will be saved and a 2/3 probability that no people will be saved.

The second half of participants were given a different choice:

If Program C is adopted 400 people will die. If Program D is adopted there is a 1/3 probability that nobody will die, and a 2/3 probability that 600 people will die.

Each of these choice sets is identical, except that one is framed with language about people being saved, and the other is framed with language about people dying.

In the first group, 72% of subjects chose Program A. In the second group, only 22% of people chose the numerically identical option: Program C.

K&T explained the difference by noting that in option A we consider the happy thought of saving 200 people, but in option C we confront the dreadful thought of 400 deaths. Our choice seems to depend not only on the objective properties of the options before us, but also on the reference point used to frame the options.

But if this is how human desire works, we are left with a worrying problem about how to translate human desires into the goals of an AI. Surely we don't want an AI to realize one state of affairs over another based merely on how the options are framed!

Before we begin to solve this problem, though, let's look at a similar result from neurobiology.

Reference-Dependence in Neurobiology

A different kind of reference-dependence has been discovered in the way that neurons encode value.

Imagine sitting in a windowless room with Mark, who is wearing blue jeans and a green t-shirt. Your perception of Mark results from about 10¹⁷ photons/second with a mean wavelength of 450 nanometers coming from every square centimeter of Mark's blue jeans, and about 10¹⁷ photons/second with a mean wavelength of 550 nanometers coming from every square centimeter of his green shirt.

Now, you and Mark step outside, and are briefly blinded by the sun. A minute later you sit on a park bench. Mark looks the same as before: blue jeans, green shirt. But now, in the bright sun, your identical subjective perceptual experience of Mark results from about 10²³ 450-nm photons/second/cm² coming from his blue jeans, and about 10²³ 550-nm photons/second/cm² coming off his green shirt.

A six-order-of-magnitude shift in the objective reality of the stimulus has resulted in no change in your subjective experience of Mark.²

How did this happen?

What changed was the illuminant, the sun. But for Earth-bound mammals, changes in an object millions of miles away are not very important. What matters for our survival and reproduction is information about the objects immediately around us. So our brains subtract away the changing effects of the sun as we move in and out of direct sunlight.

This 'illuminant subtraction' process occurs during the first step of visual processing, during transduction. The rods and cones of the retina compute an average of local light intensity, which is used as a reference point.³ Changes of light intensity from this reference point are what the rods and cones communicate to the rest of the nervous system.

Thus: information about the objective intensity of incoming light is irretrievably lost at the transducer. Light intensity is stored in the brain only in a reference-dependent way.

The same is true of our other senses. Sound intensity can differ between a quiet room and a rock concert by as much as 10 order of magnitude,⁴ and our ears respond by shifting the reference point and encoding sound intensity relative to that reference point.⁵ A rose may smell sweet in my bedroom, but its scent will be hidden in a field of roses.⁶ The somatosensory system appears to operate with the same principle. You feel your clothes when you first put them on, but the nerve endings in your skin stop reporting their existence except where your clothes are shifting across your skin or their pressure on your skin is changing.⁷ And the same is true for taste. How salty something tastes, for example, depends on the amount of sodium in your blood and in surrounding tissue in your mouth.⁸

I wrote before about how neurons encode value. But now it seems that, as neuroscientist Paul Glimcher puts it:

All sensory encoding is reference dependent: nowhere in the nervous system are the objective values of consumable rewards encoded.⁹

Thus we smack headlong into another constraint for our theories about human values and their extrapolation. Human brains can't (directly) encode value for the objective intensities of stimuli because that information is lost at the transducer.

It's beginning to seem that our folk theories about humans 'wanting' things in the world were naive.

Do Humans Want Things?

It has traditionally been thought that humans desire (or value) states of affairs:

A desire for tea is a desire for a certain state of affairs one has in mind: that one drink some tea. A desire for a new pair of skates is likewise a desire for another state of affairs: that one own a new pair of skates. And so on.¹⁰

Intuitively, when we think about what we want, it seems that we want certain states of affairs to obtain. We want to be smarter. We want there to be world peace. We want to live forever while having fun.

But as far as we can tell, our behavior is often not determined by our wanting a particular state of affairs, but by how our options are framed.

Moreover, neurons in the parietal and orbitofrontal corticies encode value in a reference-dependent way — that is, they do not encode value for objective states of affairs.¹¹ So in what sense do humans 'want' objective states of affairs?

(Compare: In what sense does the blue-minimizing robot 'want' anything?)

In a later post, I'll explain in greater detail how brains do (and don't) encode value for states of affairs. In the meantime, you might want to try to figure out on your own in what sense the brain might want things.

Notes

¹ Tversky & Kahneman (1981).

² This example, and the outline of this post, is taken from Glimcher (2010), ch. 12.

³ Burns & Baylor (2001).

⁴ Bacchus (2006); Robinson & McAlpine (2009).

⁵ Squire et al. (2008), ch. 26.

⁶ Mountcastle (2005); Squire et al. (2008), pp. 565-567.

⁷ Squire et al. (2008), ch. 25.

⁸ Squire et al. (2008), pp. 555-556.

⁹ Glimcher (2010), p. 278. Moreover, objective properties of the real world are not even linearly related to our subjective experience. The intensity of our perception of the world grows as a power law, the exact rate of which depends on the kind of stimulus (Stevens 1951, 1970, 1975). For example, we've found that:

Perceived warmth of a patch of skin = (temp. of that skin)^0.7

And, another example:

Perceived intensity of an electrical shock = (electrical current)^3.5

¹⁰ Schroeder (2009).

¹¹ It's less certain how values are encoded in the medial prefrontal cortex and in the temporal cortex, but Paul Glimcher predicts (in personal communication with me from June 2011) that this will also be a largely reference-dependent process.

References

Baccus (2006). From a whisper to a roar: Adaptation to the mean and variance of naturalistic sounds. Neuron, 51: 682-684.

Burns & Baylor (2001). Activation, deactivation, and adaptation in vertebrate photoreceptor cells. Annual Review of Neuroscience, 24: 779-805.

Glimcher (2010). Foundations of Neuroeconomic Analaysis. Oxford University Press.

Mountcastle (2005). The Sensory Hand: Neural Mechanisms of Somatic Sensation. Harvard University Press.

Robinson & McAlpine (2009). Gain control mechanisms in the auditory pathway. Current Opinion in Neurobiology, 19: 402-407.

Schroeder (2009). Desire. Stanford Encyclopedia of Philosophy.

Squire, Berg, Bloom, du Lac, & Ghosh, eds. (2008). Fundamental Neuroscience, Third Edition. Academic Press.

Stevens (1951). Handbook of Experimental Psychology, 1st edition. John Wiley & Sons.

Stevens (1970). Neural events and the psychophysical law. Science, 170: 1043-1050.

Stevens (1975). Psychophysics: Introduction to its Perceptual, Neural and Social Prospects. Wiley.

Tversky & Kahneman (1981). The framing of decisions and the psychology of choice. Science, 211: 453-458.

52 comments

Assuming Nails

Tue, 06 Jul 2010 08:26:00 +1000

Submitted by Psychohistorian • 6 votes • 28 comments

Tangential followup to Defeating Ugh Fields in Practice.
Somewhat related to Privileging the Hypothesis.

Edited to add:
I'm surprised by negative/neutral reviews. This means that either I'm simply wrong about what counts as interesting, or I haven't expressed my point very well. Based on commenter response, I think the problem is the latter. In the next week or so, expect a much more concise version of this post that expresses my point about epistemology without the detour through a criticism of economics.

At the beginning of my last post, I was rather uncharitable to neoclassical economics:

If I had to choose a single piece of evidence off of which to argue that the rationality assumption of neoclassical economics is totally, irretrievably incorrect, it's this article about financial incentives and medication compliance.... [to maintain that this theory is correct] is to crush reality into a theory that cannot hold it.

Some mistook this to mean that I believe neoclassical economists honestly, explicitly believe that all people are always totally rational. But, to quote Rick Moranis, "It's not what you think. It's far, far worse." The problem is that they often take the complex framework of neoclassical economics and believe that a valid deduction within this framework is a valid deduction about the real world. However, deductions within any given framework are entirely uninformative unless the framework corresponds to reality. But, because such deductions are internally valid, we often give them far more weight than they are due. Testing the fit of a theoretical framework to reality is hard, but a valid deduction within a framework feels so very satisfying. But even if you have a fantastically engineered hammer, you cannot go around assuming everything you want to use it on is a nail. It is all too common for experts to assume that their framework applies cleanly to the real world simply because it works so well in its own world.

If this concept doesn't make perfect sense, that's what the rest of this post is about: spelling out exactly how we go wrong when we misuse the essentially circular models of many sciences, and how this matters. We will begin with the one discipline in which this problem does not occur. The one discipline which appears immune to this type of problem is mathematics, the paragon of "pure" academic disciplines. This is principally because mathematics appears to have perfect conformity with reality, with no research or experimentation needed to ensure said conformity. The entire system of mathematics exists, in a sense, in its own world. You could sit in windowless room (perhaps one with a supercomputer) and, theoretically, derive every major theorem of mathematics, given the proper axioms. The answer to the most difficult unsolved problems in mathematics was determined the moment the terms and operators within them were defined - once you say a "circle" is "a convex polygon with every point equidistant from a center," you have already determined every single digit of pi. The problem is finding out exactly how this model works - making calculations and deductions within this model. In the case of mathematics, for whatever reason, the model conforms perfectly to the real world, so any valid mathematical deduction is a valid deduction in the real world.

This is not the case in any true science, which by necessity must rely on experiment and observation. Every science operates off of some simplified model of the world, at least with our current state of knowledge. This creates two avenues of progress: discoveries within the model, which allow one to make predictions about the world, and refinements of the model, which make such predictions more accurate. If we have an internally consistent framework, theoretical manipulation within our model will never show us our error, because our model is circular and functions outside the real world. It would be like trying to predict a stock market crash by analyzing the rules of Monopoly, except that it doesn't feel absurd. There's nothing wrong with the model qua the model, the problem is with the model qua reality, and we have to look at both of them to figure that out.

Economics is one of the fields that most suffers from this problem. Our mathematician in his windowless room could generate models of international exchange rates without ever having seen currency, once we gave him the appropriate definitions and assumptions. However, when we try using these models to forecast the future, life gets complicated. No amount of experimenting within our original model will fix this without looking at the real world. At best, we come up with some equations that appear to conform to what we observe, but we run the risk that the correspondence is incidental or that there were some (temporarily) constant variables we left out that will suddenly cease to be constant and break the whole model. It is all too easy to forget that the tremendous rigor and certainty we feel when we solve the equations of our model does not translate into the real world. Getting the "right" answer within the model is not the same thing as getting the real answer.

As an obvious practical example, an individual with a serious excess of free time could develop a model of economics which assumes that agents are rational paper-clip maximizers - that agents are rational and their ultimate concern is maximizing the number of existing paper-clips. Given even more free time and a certain amount of genius, you could even model the behaviour of irrational paper-clip maximizers, so long as you had a definition of irrational. But however refined these models are, they models will remain entirely useless unless you actually have some paper-clip maximizers whose behaviour you want to predict. And even then, you would need to evaluate your predictions after they succeed or fail. Developing a great hammer is relatively useless if the thing you need to make must be put together with screws.

There is an obvious difference in the magnitude of this problem between the sciences, and it seems to be based on the difficulty of experimenting within them. In harder sciences where experiments are fairly straightforwards, like physics and chemistry, it is not terribly difficult to make models that conform well with reality. The bleeding edge of, say, physics, tends to like in areas that are either extremely hard to observe, like the subatomic, or extremely computation-intensive. In softer sciences, experiments are very difficult, and our models rely much more on powerful assumptions, social values, and armchair reasoning.

As humans, we are both bound and compelled to use the tools we have at our disposal. The problem here is one of uncertainty. We know that most of our assumptions in economics are empirically off, but we don't know how wrong or how much that matters when we make predictions. But the model nevertheless seeps into the very core of our model of reality itself. We cannot feel this disconnect when we try to make predictions; a well-designed model feels so complete that there is no feeling of error when we try to apply it. This is likely because we are applying it correctly, but it just doesn't apply to reality. This leads people to have high degrees of certainty and yet frequently be wrong. It would not surprise me if the failure of many experts to appreciate the model-reality gap is responsible for a large proportion of incorrect predictions.

This, unfortunately, is not the end of the problem. It gets much worse when you add a normative element into your model, when you get to call some things, "efficient" or "healthful," or "normal," or "insane." There is also a serious question as to whether this false certainty is preferable to the vague unfalsifiability of even softer social sciences. But I shall save these subjects for future posts.

28 comments

Cryonics Wants To Be Big

Mon, 05 Jul 2010 17:50:29 +1000

Submitted by lsparrish • 26 votes • 160 comments

Cryonics scales very well. People who argue from the perspective that cryonics is costly are probably not aware of this fact. Even assuming you needed to come up with the lump sum all at once rather than steadily pay into life insurance, the fact is that most people would be able to afford it if most people wanted it. There are some basic physical reasons why this is the case.

So long as you keep the shape constant, for any given container the surface area is based on a square law while the volume is calculated as a cube law. For example with a simple cube shaped object, one side squared times 6 is the surface area; one side cubed is the volume. Spheres, domes, and cylinders are just more efficient variants on this theme. For any constant shape, if volume is multiplied by 1000, surface area only goes up by 100 times.

Surface area is where heat gains entry. Thus if you have a huge container holding cryogenic goods (humans in this case) it costs less per unit volume (human) than is the case with a smaller container that is equally well insulated. A way to understand why this works is to realize that you only have to insulate and cool the outside edge -- the inside does not collect any new heat. In short, by multiplying by a thousand patients, you can have a tenth of the thermal transfer to overcome per patient with no change in r-value.

But you aren't limited to using equal thickness of insulation. You can use thicker insulation, but get a much smaller proportional effect on total surface area when you use bigger container volumes. Imagine the difference between a marble sized freezer and a house-sized freezer. What happens when you add an extra foot of insulation to the surface of each? Surface area is impacted much as diameter is -- i.e. more significantly in the case of the smaller freezer than the larger one. The outer edge of the insulation is where it begins collecting heat. With a truly gigantic freezer, you could add an entire meter (or more) of insulation without it having a significant proportional impact on surface area, compared to how much surface area it already has. (This is one reason cheaper materials can be used to construct large tanks -- they can be applied in thicker layers.)

Another factor to take into account is that liquid nitrogen, the super-cheap coolant used by cryonics facilities around the world, is vastly cheaper (more than a factor of 10) when purchased in huge quantities of several tons. The scaling factors for storage tanks and high-capacity tanker trucks are a big part of the reason for this. CI has used bulk purchasing as a mechanism for getting their prices down to $100 per patient per year for their newer tanks. They are actually storing 3,000 gallons of the stuff and using it slowly over time, which implies there is a boiloff rate associated with the 3,000 gallon tank in addition to the tanks.

The conclusion I get from this is that there is a very strong self-interested case (as well as the altruistic case) to be made for the promotion of megascale cryonics towards the mainstream, as opposed to small independently run units for a few of us die-hard futurists. People who say they won't sign up for cost reasons may actually (if they are sincere) be reachable at a later date. To deal with such people's objections and make sure they remain reachable, it might be smart to get them to agree with some particular hypothetical price point at which they would feel it is justified. In large enough quantities, it is conceivable that indefinite storage costs would be as low as $50 per person, or 50 cents per year.

That is much cheaper than saving a life any other way. Of course there's still the risk that it might not work. However, given a sufficient chance of it working it could still be morally superior to other life saving strategies that cost more money. It also has inherent ecological advantages over other forms of life-saving in that it temporarily reduces the active population, giving the environment a chance to recover and green tech more time to take hold so that they can be supported sustainably and comfortably. And we might consider the advent of life-health extension in the future to be a reason to think it a qualitatively better form of life-saving.

Note: This article only looks directly at cooling energy costs; construction and ongoing maintenance do not necessarily scale as dramatically. The same goes for stabilization (which I view as a separate though indispensable enterprise). Both of these do have obvious scaling factors however. Other issues to consider are defense and reliability. Given the large storage mass involved, preventing temperature fluctuations without being at the exact boiling temperature of LN2 is feasible; it could be both highly failsafe and use the ideal cryonics temperature of -135C rather than the -196C that LN2 boiloff as a temperature regulation mechanism requires. Feel free to raise further issues in the comments.

160 comments

What Cost for Irrationality?

Fri, 02 Jul 2010 04:25:06 +1000

Submitted by Kaj_Sotala • 54 votes • 105 comments

This is the first part in a mini-sequence presenting content from Keith E. Stanovich's excellent book What Intelligence Tests Miss: The psychology of rational thought. It will culminate in a review of the book itself.

People who care a lot about rationality may frequently be asked why they do so. There are various answers, but I think that many of ones discussed here won't be very persuasive to people who don't already have an interest in the issue. But in real life, most people don't try to stay healthy because of various far-mode arguments for the virtue of health: instead, they try to stay healthy in order to avoid various forms of illness. In the same spirit, I present you with a list of real-world events that have been caused by failures of rationality, so that you might better persuade others of this being important.

What happens if you, or the people around you, are not rational? Well, in order from least serious to worst, you may...

Have a worse quality of living. Status Quo bias is a general human tendency to prefer the default state, regardless of whether the default is actually good or not. In the 1980's, Pacific Gas and Electric conducted a survey of their customers. Because the company was serving a lot of people in a variety of regions, some of their customers suffered from more outages than others. Pacific Gas asked customers with unreliable service whether they'd be willing to pay extra for more reliable service, and customers with reliable service whether they'd be willing to accept a less reliable service in exchange for a discount. The customers were presented with increases and decreases of various percentages, and asked which ones they'd be willing to accept. The percentages were same for both groups, only with the other having increases instead of decreases. Even though both groups had the same income, customers of both groups overwhelmingly wanted to stay with their status quo. Yet the service difference between the groups was large: the unreliable service group suffered 15 outages per year of 4 hours' average duration and the reliable service group suffered 3 outages per year of 2 hours' average duration! (Though note caveats.)

A study by Philips Electronics found that one half of their products had nothing wrong in them, but the consumers couldn't figure out how to use the devices. This can be partially explained by egocentric bias on behalf of the engineers. Cognitive scientist Chip Heath notes that he has "a DVD remote control with 52 buttons on it, and every one of them is there because some engineer along the line knew how to use that button and believed I would want to use it, too. People who design products are experts... and they can't imagine what it's like to be as ignorant as the rest of us."

Suffer financial harm. John Allen Paulos is a professor of mathematics at Temple University. Yet he fell prey to serious irrationality which began when he purchased WorldCom stock at $47 per share in early 2000. As bad news about the industry began mounting, WorldCom's stock price started falling - and as it did so, Paulos kept buying, regardless of accumulating evidence that he should be selling. Later on, he admitted that his "purchases were not completely rational" and that "I bought shares even though I knew better". He was still buying - partially on borrowed money - when the stock price was $5. When it momentarily rose to $7, he finally decided to sell. Unfortunately, he didn't get off from work until the market closed, and on the next market day the stock had lost a third of its value. Paulos finally sold everything, at a huge loss.

Stock market losses due to irrationality are not atypical. From the beginning of 1998 to the end of 2001, the Firsthand Technology Value mutual fund had an average gain of 16 percent per year. Yet the average investor who invested in the fund lost 31.6 percent of her money over the same period. Investors actually lost a total of $1.9 billion by investing in a fund which was producing 16 percent of a profit per year. That happened because the fund was very volatile, causing people to invest and cash out at exactly the wrong times. When it gained, it gained a lot, and when it lost, it lost a lot. When people saw that it had been making losses, they sold, and when they saw it had been making gains, they bought. In other words, they bought when high and sold when low - exactly the opposite of what you're supposed to do if you want to make a profit. Reporting on a study of 700 mutual funds during 1998-2001, finanical reporter Jason Zweig noted that "to a remarkable degree, investors underperformed their funds' reported returns - sometimes by as much as 75 percentage points per year."

Be manipulated and robbed of personal autonomy. Subjects were asked to divide 100 usable livers to 200 children awaiting a transplant. With two groups of children, group A with 100 children and group B with 100 children, the overwhelming response was to allocate 50 livers to each, which seems reasonable. But when group A had 100 children, each with an 80 percent chance of surviving when transplanted, and group B had 100 children, each with a 20 percent chance of surviving when transplanted, people still chose the equal allocation method even if this caused the unnecessary deaths of 30 children. Well, that's just a question of values and not rationality, right? Turns out that if the patients were ranked from 1 to 200 in terms of prognosis, people were relatively comfortable with distributing organs to the top 100 patients. It was only when the question was framed as "group A versus group B" that people suddenly felt they didn't want to abandon group B entirely. Of course, these are exactly the same dilemma. One could almost say that the person who got to choose which framing to use was getting to decide on behalf of the people being asked the question.

Two groups of subjects were given information about eliminating affirmative action and adopting a race-neutral policy at several universities. One group was told that under race-neutral conditions, the probability of a black student being admitted would decline from 42 percent to 13 percent and the probability of a white student being admitted would rise from 25 percent to 27 percent. The other group was told that under race-neutral admissions, the number of black students being admitted would decrease by 725 and the number of white students would increase by 725. These two framings were both saying the same thing, but you can probably guess the outcome: support for affirmative action was much higher in the percentage group.

In a hypothetical country, a family with no children and an income of $35,000 pays $4,000 in tax, while a family with no children and an income of $100,000 pays $26,000 in tax. Now suppose that there's a $500 tax reduction for having a child for a family with an income of $35,000. Should the family with an income of $100,000 be given a larger reduction because of their higher income? Here, most people would say no. But suppose that instead, the baseline is that a family of two children with an income of $35,000 pays $3,000 in tax and that a family of two children with an income of $100,000 pays $25,000 in tax. We propose to make the families with no children pay more tax - that is, have a "childless penalty". Say that the family with the income of $100,000 and one child has their taxes set at $26,000 and the same family with no children has their taxes set at $27,000 - there's a childless penalty of $1,000 per child. Should the poorer family which makes $35,000 and has no children also pay the same $2,000 childless penalty as the richer family? Here, most people would also say no - they'd want the "bonus" for children to be equal for low- and high-income families, but they do not want the "penalty" for lacking children to be the high for same and low income.

End up falsely accused or imprisoned. In 2003, an attorney was released from prison in England when her conviction of murdering her two infants was overturned. Five months later, another person was released from prison when her charge of having murdered her children was also overturned. In both cases, the evidence presented against them had been ambiguous. What had convinced the jury was that in both cases, a pediatrician had testified that the odds of two children in the same family dying of infant death syndrome was 73 million to 1. Unfortunately, he had arrived to this figure by squaring the odds of a single death. Squaring the odds of a single event to arrive at the odds for it happening twice only works if the two events are independent. But that assumption is likely to be false in the case of multiple deaths in the same family, where numerous environmental and genetic factors may have affected both deaths.

In the late 1980s and early 1990s, many parents were excited and overjoyed to hear of a technique coming out of Australia that enabled previously totally non-verbal autistic children to communicate. It was uncritically promoted in highly visible media such as 60 Minutes, Parade magazine and the Washington Post. The claim was that autistic individuals and other children with developmental disabilities who'd previously been nonverbal had typed highly literate messages on a keyboard when their hands and arms had been supported over by the typewriter by a sympathetic "facilitator". As Stanovich describes: "Throughout the early 1990s, behavioral science researchers the world over watched in horrified anticipation, almost as if observing cars crash in slow motion, while a predictable tragedy unfolded before their eyes." The hopes of countless parents were dashed when it was shown that the "facilitators" had been - consciously or unconsciously - directing the children's hands on the right keys. It should have been obvious that spreading such news before the technique had been properly scientifically examined was dangerously irresponsible - and it gets worse. During some "faciliation" sessions, children "reported" having been sexually abused by their parents, and were removed from their homes as a result. (Though they were eventually returned.)

End up dead. After 9/11, people became afraid of flying and started doing so less. Instead, they began driving more. Unfortunately, car travel has a much higher chance of death than air travel. Researchers have estimated that over 300 more people died in the last months of 2001 because they drove instead of flying. Another group calculated that for flying to be as dangerous as driving, there would have to be an incident on the scale of 9/11 once a month!

Have your society collapse. Possibly even more horrifying is the tale of Albania, which had previously been a communist dictatorship but had made considerable financial progress from 1992 to 1997. In 1997, however, one half of the adult population had fallen victim to Ponzi schemes. In a Ponzi scheme, the investment itself isn't actually making any money, but rather early investors are paid off with the money from late investors, and eventually the system has to collapse when no new investors can be recruited. But when schemes offering a 30 percent monthly return began to become popular in Albania, competitors offering a 50-60 or even a 100 percent monthly return soon showed up, and people couldn't resist the temptation. Eventually both the government and economy of Albania collapsed. Stanovich describes:

People took out mortgages on their homes in order to participate. Others sold their homes. Many put their entire life savings into the schemes. At their height, an amount equal to 50 percent of the country's GDP was invested in Ponzi schemes. Before the schemes collapsed, they actually began to compete with wage income and distort the economy. For example, one business owner saw his workforce quickly slip from 130 employees to 70 because people began to hink they could invest in the Ponzi schemes instead of actually working for their income.

The estimated death toll was between 1,700 and 2,000.

105 comments

Defeating Ugh Fields In Practice

Sun, 20 Jun 2010 05:37:44 +1000

Submitted by Psychohistorian • 60 votes • 92 comments

Unsurprisingly related to: Ugh fields.

If I had to choose a single piece of evidence off of which to argue that the rationality assumption of neoclassical economics is totally, irretrievably incorrect, it's this article about financial incentives and medication compliance. In short, offering people small cash incentives vastly improves their adherence to life-saving medical regimens. That's right. For a significant number of people, a small chance at winning $10-100 can be the difference between whether or not they stick to a regimen that has a very good chance of saving their life. This technique has even shown promise in getting drug addicts and psychiatric patients to adhere to their regimens, for as little as a $20 gift certificate. This problem, in the aggregate, is estimated to cost about 5% of total health care spending -$100 billion - and that may not properly account for the utility lost by those who are harmed beyond repair. To claim that people are making a reasoned decision between the payoffs of taking and not-taking their medication, and that they be persuaded to change their behaviour by a payoff of about $900 a year (or less), is to crush reality into a theory that cannot hold it. This is doubly true when you consider that some of these people were fairly affluent.

A likely explanation of this detrimental irrationality is something close to an Ugh field. It must be miserable having a life-threatening illness. Being reminded of it by taking a pill every single day (or more frequently) is not pleasant. Then there's the question of whether you already took the pill. Because if you take it twice in one day, you'll end up in the hospital. And Heaven forfend your treatment involves needles. Thus, people avoid taking their medicine because the process becomes so unpleasant, even though they know they really should be taking it.

As this experiment shows, this serious problem has a simple and elegant solution: make taking their medicine fun. As one person in the article describes it, using a low-reward lottery made taking his meds "like a game;" he couldn't wait to check the dispenser to see if he'd won (and take his meds again). Instead of thinking about how they have some terrible condition, they get excited thinking about how they could be winning money. The Ugh field has been demolished, with the once-feared procedure now associated with a tried-and-true intermittent reward system. It also wouldn't surprise me the least if people who are unlikely to adhere to a medical regimen are the kind of people who really enjoy playing the lottery.

This also explains why rewarding success may be more useful than punishing failure in the long run: if a kid does his homework because otherwise he doesn't get dessert, it's labor. If he gets some reward for getting it done, it becomes a positive. The problem is that if she knows what the reward is, she may anchor on already having the reward, turning it back into negative reinforcement - if you promise your kid a trip to Disneyland if they get above a 3.5, and they get a 3.3, they feel like they actually lost something. The use of a gambling mechanism may be key for this. If your reward is a chance at a real reward, you don't anchor as already having the reward, but the reward still excites you.

I believe that the fact that such a significant problem can be overcome with such a trivial solution has tremendous implications, the enumeration of all of which would make for a very unwieldy post. A particularly noteworthy issue is the difficulty of applying such a technique to one's own actions, a problem which I believe has a fairly large number of workable solutions. That's what comments, and, potentially, follow-up posts are for.

92 comments

Blue- and Yellow-Tinted Choices

Fri, 14 May 2010 08:35:42 +1000

Submitted by Yvain • 43 votes • 54 comments

_{A man comes to the rabbi and complains about his life: "I have almost no money, my wife is a shrew, and we live in a small apartment with seven unruly kids. It's messy, it's noisy, it's smelly, and I don't want to live."
The rabbi says, "Buy a goat."
"What? I just told you there's hardly room for nine people, and it's messy as it is!"
"Look, you came for advice, so I'm giving you advice. Buy a goat and come back in a month."
In a month the man comes back and he is even more depressed: "It's gotten worse! The filthy goat breaks everything, and it stinks and makes more noise than my wife and seven kids! What should I do?"
The rabbi says, "Sell the goat."
A few days later the man returns to the rabbi, beaming with happiness: "Life is wonderful! We enjoy every minute of it now that there's no goat - only the nine of us. The kids are well-behaved, the wife is agreeable - and we even have some money!"}

-- traditional Jewish joke

Related to: Anchoring and Adjustment

Biases are “cognitive illusions” that work on the same principle as optical illusions, and a knowledge of the latter can be profitably applied to the former. Take, for example, these two cubes (source: Lotto Lab, via Boing Boing):

The “blue” tiles on the top face of the left cube are the same color as the “yellow” tiles on the top face of the right cube; if you're skeptical you can prove it with the eyedropper tool in Photoshop (in which both shades come out a rather ugly gray).

The illusion works because visual perception is relative. Outdoor light on a sunny day can be ten thousand times greater than a fluorescently lit indoor room. As one psychology book put it: for a student reading this book outside, the black print will be objectively lighter than the white space will be for a student reading the book inside. Nevertheless, both students will perceive the white space as subjectively white and the black space as subjectively black, because the visual system returns to consciousness information about relative rather than absolute lightness. In the two cubes, the visual system takes the yellow or blue tint as a given and outputs to consciousness the colors of each pixel compared to that background.

So this optical illusion occurs when the brain judges quantities relative to their surroundings rather than based on some objective standard. What's the corresponding cognitive illusion?

In Predictably Irrational (relatively recommended, even though the latter chapters sort of fail to live up to the ones mentioned here) Dan Ariely asks his students to evaluate (appropriately) three subscription plans to the Economist:

Ariely asked his subjects which plan they'd buy if they needed an Economist subscription. 84% wanted the combo plan, 16% wanted the web only plan, and no one wanted the print only plan. After all, the print plan cost exactly the same as the print + web plan, but the print + web plan was obviously better. Which raises the question: why even include a print-only plan? Isn't it something of a waste of space?

Actually, including the print-only plan turns out to be a very good business move for the Economist. Ariely removed the print-only plan from the choices. Now the options looked like this.

There shouldn't be any difference. After all, he'd only removed the plan no one chose, the plan no sane person would choose.

This time, 68% of students chose the web only plan and 32% the combo plan. That's a 52% shift in preferences between the exact same options.

The rational way to make the decision is to compare the value of a print subscription to the Economist (as measured by the opportunity cost of that money) to the difference in cost between the web and combo subscriptions. But this would return the same answer in both of the above cases, so the students weren't doing it that way.

What it looks like the students were doing was perceiving relative value in the same way the eye perceives relative color. The ugly gray of the cube appeared blue when it was next to something yellow, and yellow when it was next to something blue. In the same way, the $125 cost of the combo subscription looks like good value next to a worse deal, and bad value next to a better deal.

When the $125 combo subscription was placed next to a $125 plan with fewer features (print only instead of print plus web) it looked like a very good deal – the equivalent of placing an ugly gray square next to something yellow to make it look blue. Take away the yellow, or the artificially bad deal, and it doesn't look nearly as attractive.

This is getting deep into Dark Arts territory, and according to Predictably Irrational, the opportunity to use these powers for evil has not gone unexploited. Retailers will deliberately include in their selection a super deluxe luxury model much fancier and more expensive than they expect anyone to ever want. The theory is that consumers are balancing a natural hedonism that tells them to get the best model possible against a commitment to financial prudence. So most consumers, however much they like television, will have enough good sense to avoid buying a $2000 TV. But if the retailer carries a $4000 super-TV, the $2000 TV suddenly doesn't look quite so bad.

The obvious next question is “How do I use this knowledge to trick hot girls or guys into going out with me?” Dan Ariely decided to run some experiments on his undergraduate class. He took photographs of sixty students, then asked other students to rate their attractiveness. Next, he grouped the photos into pairs of equally attractive students. And next, he went to Photoshop and made a slightly less attractive version of each student: a blemish here, an asymmetry there.

Finally, he went around campus, finding students and showing them three photographs and asking which person the student would like to go on a date with. Two of the photographs were from one pair of photos ranked equally attractive. The third was a version of one of the two, altered to make it less attractive. So, for example, he might have two people, Alice and Brenda, who had been ranked equally attractive, plus a Photoshopped ugly version of Brenda.

The students overwhelmingly (75%) chose the person with the ugly double (Brenda in the example above), even though the two non-Photoshopped faces were equally attractive. Ariely then went so far as to recommend in his book that for best effect, you should go to bars and clubs with a wingman who is similar to you but less attractive. Going with a random ugly person would accomplish nothing, but going with someone similar to but less attractive than you would put you into a reference class and then bump you up to the top of the reference class, just like in the previous face experiment.

Ariely puts these studies in a separate chapter from his studies on anchoring and adjustment (which are also very good) but it all seems like the same process to me: being more interested in the difference between two values than in the absolute magnitude of them. All that makes anchoring and adjustment so interesting is that the two values have nothing in common with one another.

This process also has applications to happiness set points, status seeking, morality, dieting, larger-scale purchasing behavior, and akrasia which deserve a separate post

54 comments

Articles Tagged ‘economics’ - Less Wrong

New report: Intelligence Explosion Microeconomics

Is Sunk Cost Fallacy a Fallacy?

Prediction is hard, especially of medicine

1 The Future of Medicine

1.1 Part 1

1.1.1 Diagnostics

1.1.2 Resuscitation

1.1.3 Antibiotics

1.1.4 Immunology and cancer

1.1.5 Atherosclerosis

1.2 Part 2

1.2.1 Anesthesia

1.2.2 Surgery

1.2.3 Geriatrics

1.2.4 Psychiatry & Behavior

1.2.5 Implants & Prosthetics

1.2.6 Hemodialysis

1.2.7 Organ Preservation

1.2.8 Other Approaches to Organ Preservation

1.2.9 Genetic therapy

1.2.10 Prevention

1.2.11 The Downside

2 Reactions

3 Further reading

A Crash Course in the Neuroscience of Human Motivation

Preface

Contents:

Folk Psychology

Neoclassical Economics

Behaviorism and Reinforcement Learning

Reinforcement Learning and Decision Theory

The Turn to the Brain

Hebbian Learning

Expected Utility in Neurons

Real-Time Expected Utility Updates

Argmax and Reservation Price

Random Utility

Discounting

Relative and Absolute Utility

Normalization

Are Actions Choices?

The Primate Choice Mechanism: A Brief Review

Marginal Utility and Reference Dependence

Valuation in the Brain

Notes

References

Do Humans Want Things?

Reference-Dependence in Neurobiology

Do Humans Want Things?

Notes

References

Assuming Nails

Cryonics Wants To Be Big

What Cost for Irrationality?

Defeating Ugh Fields In Practice

Blue- and Yellow-Tinted Choices