UC Berkeley launches Center for Human-Compatible Artificial Intelligence
Source article: http://news.berkeley.edu/2016/08/29/center-for-human-compatible-artificial-intelligence/
UC Berkeley artificial intelligence (AI) expert Stuart Russell will lead a new Center for Human-Compatible Artificial Intelligence, launched this week.
Russell, a UC Berkeley professor of electrical engineering and computer sciences and the Smith-Zadeh Professor in Engineering, is co-author of Artificial Intelligence: A Modern Approach, which is considered the standard text in the field of artificial intelligence, and has been an advocate for incorporating human values into the design of AI.
The primary focus of the new center is to ensure that AI systems are beneficial to humans, he said.
The co-principal investigators for the new center include computer scientists Pieter Abbeel and Anca Dragan and cognitive scientist Tom Griffiths, all from UC Berkeley; computer scientists Bart Selman and Joseph Halpern, from Cornell University; and AI experts Michael Wellman and Satinder Singh Baveja, from the University of Michigan. Russell said the center expects to add collaborators with related expertise in economics, philosophy and other social sciences.
The center is being launched with a grant of $5.5 million from the Open Philanthropy Project, with additional grants for the center’s research from the Leverhulme Trust and the Future of Life Institute.
Russell is quick to dismiss the imaginary threat from the sentient, evil robots of science fiction. The issue, he said, is that machines as we currently design them in fields like AI, robotics, control theory and operations research take the objectives that we humans give them very literally. Told to clean the bath, a domestic robot might, like the Cat in the Hat, use mother’s white dress, not understanding that the value of a clean dress is greater than the value of a clean bath.
The center will work on ways to guarantee that the most sophisticated AI systems of the future, which may be entrusted with control of critical infrastructure and may provide essential services to billions of people, will act in a manner that is aligned with human values.
“AI systems must remain under human control, with suitable constraints on behavior, despite capabilities that may eventually exceed our own,” Russell said. “This means we need cast-iron formal proofs, not just good intentions.”
One approach Russell and others are exploring is called inverse reinforcement learning, through which a robot can learn about human values by observing human behavior. By watching people dragging themselves out of bed in the morning and going through the grinding, hissing and steaming motions of making a caffè latte, for example, the robot learns something about the value of coffee to humans at that time of day.
“Rather than have robot designers specify the values, which would probably be a disaster,” said Russell, “instead the robots will observe and learn from people. Not just by watching, but also by reading. Almost everything ever written down is about people doing things, and other people having opinions about it. All of that is useful evidence.”
Russell and his colleagues don’t expect this to be an easy task.
“People are highly varied in their values and far from perfect in putting them into practice,” he acknowledged. “These aspects cause problems for a robot trying to learn what it is that we want and to navigate the often conflicting desires of different individuals.”
Russell, who recently wrote an optimistic article titled “Will They Make Us Better People?,” summed it up this way: “In the process of figuring out what values robots should optimize, we are making explicit the idealization of ourselves as humans. As we envision AI aligned with human values, that process might cause us to think more about how we ourselves really should behave, and we might learn that we have more in common with people of other cultures than we think.”
European Soylent alternatives
A person at our local LW meetup (not active at LW.com) tested various Soylent alternatives that are available in Europe and wrote a post about them:
______________________
Over the course of the last three months, I've sampled parts of the
european Soylent alternatives to determine which ones would work for me
longterm.
- The prices are always for the standard option and might differ for
e.g. High Protein versions.
- The prices are always for the amount where you get the cheapest
marginal price (usually around a one month supply, i.e. 90 meals)
- Changing your diet to Soylent alternatives quickly leads to increased
flatulence for some time - I'd recommend a slow adoption.
- You can pay for all of them with Bitcoin.
- The list is sorted by overall awesomeness.
So here's my list of reviews:
Joylent:
Taste: 7/10
Texture: 7/10
Price: 5eu / day
Vegan option: Yes
Overall awesomeness: 8/10
This one is probably the european standard for nutritionally complete
meal replacements.
The texture is nice, the taste is somewhat sweet, the flavors aren't
very intensive.
They have an ok amount of different flavors but I reduced my orders to
Mango (+some Chocolate).
They offer a morning version with caffeine and a sports version with
more calories/protein.
They also offer Twennybars (similar to a cereal bar but each offers 1/5
of your daily needs), which everyone who tasted them really liked.
They're nice for those lazy times where you just don't feel like pouring
the powder, adding water and shaking before you get your meal.
They do cost 10eu per day, though.
I also like the general style. Every interaction with them was friendly,
fun and uncomplicated.
Veetal:
Taste: 8/10
Texture: 7/10
Price: 8.70 / day
Vegan option: Yes
Overall awesomeness: 8/10
This seems to be the "natural" option, apparently they add all those
healthy ingredients.
The texture is nice, the taste is sweeter than most, but not very sweet.
They don't offer flavors but the "base taste" is fine, it also works
well with some cocoa powder.
It's my favorite breakfast now and I had it ~54 of the last 60 days.
Would have been first place if not for the relatively high price.
Mana:
Taste: 6/10
Texture: 7/10
Price: 6.57 / day
Vegan option: Only Vegan
Overall awesomeness: 7/10
Mana is one of the very few choices that don't taste sweet but salty.
Among all the ones I've tried, it tastes the most similar to a classic meal.
It has a somewhat oily aftertaste that was a bit unpleasent in the
beginning but is fine now that I got used to it.
They ship the oil in small bottles seperate from the rest which you pour
into your shaker with the powder. This adds about 100% more complexity
to preparing a meal.
The packages feel somewhat recycled/biodegradable which I don't like so
much but which isn't actually a problem.
It still made it to the list of meals I want to consume on a regular
basis because it tastes so different from the others (and probably has a
different nutritional profile?).
Nano:
Taste: 7/10
Texture: 7/10
Price: 1.33eu / meal
*I couldn't figure out whether they calculate with 3 or 5 meals per day
** Price is for an order of 666 meals. I guess 222 meals for 1.5eu /meal
is the more reasonable order
Vegan option: Only Vegan
Overall awesomeness: 7/10
Has a relatively sweet taste. Only comes in the standard vanilla-ish flavor.
They offer a Veggie hot meal which is the only one besides Mana that
doesn't taste sweet. It tastes very much like a vegetable soup but was a
bit too spicy for me. (It's also a bit more expensive)
Nano has a very future-y feel about it that I like. It comes in one meal
packages which I don't like too much but that's personal preference.
Queal:
Taste: 7/10
Texture: 6/10
Price: 6.5 / day
Vegan option: No
Overall awesomeness: 7/10
Is generally similar to Joylent (especially in flavor) but seems
strictly inferior (their flavors sound more fun - but don't actually
taste better).
Nutrilent:
Taste: 6/10
Texture: 7/10
Price: 5 / day
Vegan option: No
Overall awesomeness: 6/10
Taste and flavor are also similar to Joylent but it tastes a little
worse. It comes in one meal packages which I don't fancy.
Jake:
Taste: 6/10
Texture: 7/10
Price: 7.46 / day
Vegan option: Only Vegan
Overall awesomeness: 6/10
Has a silky taste/texture (I didn't even know that was a thing before I
tried it). Only has one flavor (vanilla) which is okayish.
Also offers a light and sports option.
Huel:
Taste: 1/10
Texture: 6/10
Price: 6.70 / day
Vegan option: Only Vegan
Overall awesomeness: 4/10
The taste was unanimously rated as awful by every single person to whom
I gave it for trying. The Vanilla flavored version was a bit less awful
then the unflavored version but still...
The worst packaging - it's in huge bags that make it hard to pour and
are generally inconvenient to handle.
Apart from that, it's ok, I guess?
Ambronite:
Taste: ?
Texture: ?
Price: 30 / day
Vegan option: Only Vegan
Overall awesomeness: ?
Price was prohibitive for testing - they advertise it as being very
healthy and natural and stuff.
Fruiticio:
Taste: ?
Texture: ?
Price: 5.76 / day
Vegan option: No
Overall awesomeness: ?
They offer a variety for women and one for men. I didn't see any way for
me to find out which of those I was supposed to order. I had to give up
the ordering process at that point. (I guess you'd have to ask your
doctor which one is for you?)
Conclusion:
Meal replacements are awesome, especially when you don't have much time
to make or eat a "proper" meal.
I generally don't feel full after drinking them but also stop being hungry.
I assume they're healthier than the average European diet.
The texture and flavor do get a bit dull after a while if I only use
meal replacements.
On my usual day I eat one serving of Joylent, Veetal and Mana at the
moment (and have one or two "non-replaced" meals).
A Review of Signal Data Science
I took part in the second signal data science cohort earlier this year, and since I found out about Signal through a slatestarcodex post a few months back (it was also covered here on less wrong), I thought it would be good to return the favor and write a review of the program.
The tl;dr version:
Going to Signal was a really good decision. I had been doing teaching work and some web development consulting previous to the program to make ends meet, and now I have a job offer as a senior machine learning researcher1. The time I spent at signal was definitely necessary for me to get this job offer, and another very attractive data science job offer that is my "second choice" job. I haven't paid anything to signal, but I will have to pay them a fraction of my salary for the next year, capped at 10% and a maximum payment of $25k.
The longer version:
Obviously a ~12 week curriculum is not going to be a magic pill that turns a nontechnical, averagely intelligent person into a super-genius with job offers from Google and Facebook. In order to benefit from Signal, you should already be somewhat above average in terms of intelligence and intellectual curiosity. If you have never programmed and/or never studied mathematics beyond high school2 , you will probably not benefit from Signal in my opinion. Also, if you don't already understand statistics and probability to a good degree, they will not have time to teach you. What they will do is teach you how to be really good with R, make you do some practical machine learning and learn some SQL, all of which are hugely important for passing data science job interviews. As a bonus, you may be lucky enough (as I was) to explore more advanced machine learning techniques with other program participants or alumni and build some experience for yourself as a machine learning hacker.
As stated above, you don't pay anything up front, and cheap accommodation is available. If you are in a situation similar to mine, not paying up front is a huge bonus. The salary fraction is comparatively small, too, and it only lasts for one year. I almost feel like I am underpaying them.
This critical comment by fluttershy almost put me off, and I'm glad it didn't. The program is not exactly "self-directed" - there is a daily schedule and a clear path to work through, though they are flexible about it. Admittedly there isn't a constant feed of staff time for your every whim - ideally there would be 10-20 Jonahs, one per student; there's no way to offer that kind of service at a reasonable price. Communication between staff and students seemed to be very good, and key aspects of the program were well organised. So don't let perfect be the enemy of good: what you're getting is an excellent focused training program to learn R and some basic machine learning, and that's what you need to progress to the next stage of your career.
Our TA for the cohort, Andrew Ho, worked tirelessly to make sure our needs were met, both academically and in terms of running the house. Jonah was extremely helpful when you needed to debug something or clarify a misunderstanding. His lectures on selected topics were excellent. Robert's Saturday sessions on interview technique were good, though I felt that over time they became less valuable as some people got more out of interview practice than others.
I am still in touch with some people I met on my cohort, even though I had to leave the country, I consider them pals and we keep in touch about how our job searches are going. People have offered to recommend me to companies as a result of Signal. As a networking push, going to Signal is certainly a good move.
Highly recommended for smart people who need a helping hand to launch a technical career in data science.
1: I haven't signed the contract yet as my new boss is on holiday, but I fully intend to follow up when that process completes (or not). Watch this space.
2: or equivalent - if you can do mathematics such as matrix algebra, know what the normal distribution is, understand basic probability theory such as how to calculate the expected value of a dice roll, etc, you are probably fine.
Superintelligence and physical law
It's been a few years since I read http://lesswrong.com/lw/qj/einsteins_speed/ and the rest of the quantum physics sequence, but I recently learned about the company Nutonian, http://www.nutonian.com/. Basically it's a narrow AI system that looks at unstructured data and tries out billions of models to fit it, favoring those that use simpler math. They apply it to all sorts of fields, but that includes physics. It can't find Newton's laws from three frames of a falling apple, but it did find the Hamiltonian of a double pendulum given its motion data after a few hours of processing: http://phys.org/news/2009-12-eureqa-robot-scientist-video.html
Two forms of procrastination
I noticed something about myself when comparing two forms of procrastination:
a) reading online discussions,
b) watching movies online.
Reading online discussions (LessWrong, SSC, Reddit, Facebook) and sometimes writing a comment there, is a huge sink of time for me. On the other hand, watching movies online is almost harmless, at least compared with the former option. The difference is obvious when I compare my productivity at the end of the day when I did only the former, or only the latter. The interesting thing is that at the moment it feels the other way round.
When I start watching a movie that is 1:30:00 long, or start watching a series where each part is 40:00 long but I know I will probably watch more than one part a day, I am aware from the beginning that I am going to lose more than one hour of time; possibly several hours. On the other hand, when I open the "Discussion" tab on LessWrong, the latest "Open Thread" on SSC, my few favorite subreddits, and/or my Facebook "Home" page, it feels like it will only take a few minutes -- I will click on the few interesting links, quickly skim through the text, and maybe write a comment or two -- it certainly feels like much less than an hour.
But the fact is, when I start reading the discussions, I will probably click on at least hundred links. Most of the pages I will read just as quickly as I imagined, but there will be a few that will take disproportionally more time; either because they are interesting and long, or because they contain further interesting links. And writing a comment sometimes takes more time than it seems; it can easily be a half an hour for a three-paragraphs-long comment. (Ironically, this specific article gets written rather quickly, because I know what I want to write, and I write it directly. But there are comments where I think a lot, and keep correcting my text, to avoid misunderstanding when debating a sensitive topic, etc.) And when I stop doing it, because I want to make something productive for a change, I will feel tired. Reading many different things, trying to read quickly, and formulating my answers, that all makes me mentally exhausted. So after I close the browser, I just wish I could take a nap.
On the other hand, watching a movie does not make me tired in that specific way. The movies runs at its own speed and doesn't require me to do anything actively. Also, there is no sense of urgency; none of the "if I reply to this now, people will notice and respond, but if I do it a week later, no one will care anymore". So I feel perfectly comfortable pausing the movie at any moment, doing something productive for a while, then unpausing the movie and watching more. I know I won't miss anything.
I think it's the mental activity during my procrastination that both makes me tired and creates the illusion that it will take less time than it actually does. When the movie says 1:30:00, I know it will be 1:30:00 (or maybe a little less because of the final credits). With a web page, I can always tell myself "don't worry, I will read this one really fast", so there is the illusion that I have it under control, and can reduce the time waste. The fact that I am reading an individual page really fast makes me underestimate how much time it took to read all those pages.
On the other hand, sometimes I do inverse procrastination -- I start watching a movie, pause it a dozen times and do some useful work during the breaks -- and at the end of the day I spent maybe 90% of the time working productively, while my brain tells me I just spent the whole day watching a movie, so I almost feel like I had a free day.
Okay, so how could I use this knowledge to improve my productivity?
1) Knowing the difference between the two forms of procrastination, whenever I feel a desire to escape to the online world, I should start watching a movie instead of reading some debate, because thus I will waste less time, even if it feels the other way round.
2) Integrate it with pomodoro? 10 minutes movie, 50 minutes work, then again, and at the end of the day my lying brain will tell me "dude, you didn't work at all today, you were just watching movies, of course you should feel awesome!".
Do you have a similar experience? No idea how typical is this. No need to hurry with responding, I am going to watch a movie now. ;-)
Thoughts on hacking aromanticism?
Several years ago, Alicorn wrote an article about how she hacked herself to be polyamorous. I'm interested in methods for hacking myself to be aromantic. I've had some success with this, so I'll share what's worked for me, but I'm really hoping you all will chime in with your ideas in the comments.
Motivation
Why would someone want to be aromantic? There's the obvious time commitment involved in romance, which can be considerable. This is an especially large drain if you're in a situation where finding suitable partners is difficult, which means most of this time is spent enduring disappointment (e.g. if you're heterosexual and the balance of singles in your community is unfavorable).
But I think an even better way to motivate aromanticism is by referring you to this Paul Graham essay, The Top Idea in Your Mind. To be effective at accomplishing your goals, you'd like to have your goals be the most interesting thing you have to think about. I find it's far too easy for my love life to become the most interesting thing I have to think about, for obvious reasons.
Subproblems
After thinking some, I came up with a list of 4 goals people try to achieve through engaging in romance:
- Companionship.
- Sexual pleasure.
- Infatuation (also known as new relationship energy).
- Validation. This one is trickier than the previous three, but I think it's arguably the most important. Many unhappy singles have friends they are close to, and know how to masturbate, but they still feel lousy in a way people in post-infatuation relationships do not. What's going on? I think it's best described as a sort of romantic insecurity. To test this out, imagine a time when someone you were interested in was smiling at you, and contrast that with the feeling of someone you were interested in turning you down. You don't have to experience companionship or sexual pleasure from these interactions for them to have a major impact on your "romantic self-esteem". And in a culture where singlehood is considered a failure, it's natural for your "romantic self-esteem" to take a hit if you're single.
To remove the need for romance, it makes sense to find quicker and less distracting ways to achieve each of these 4 goals. So I'll treat each goal as a subproblem and brainstorm ideas for solving it. Subproblems 1 through 3 all seem pretty easy to solve:
- Companionship: Make deep friendships with people you're not interested in romantically. I recommend paying special attention to your coworkers and housemates, since you spend so much time with them.
- Sexual pleasure: Hopefully you already have some ideas on pleasuring yourself.
- Infatuation: I see this as more of a bonus than a need to be met. There are lots of ways to find inspiration, excitement, and meaning in life outside of romance.
Subproblem 4 seems trickiest.
Hacking Romantic Self-Esteem
I'll note that what I'm describing as "validation" or "romantic self-esteem" seems closely related to abundance mindset. But I think it's useful to keep them conceptually distinct. Although alieving that there are many people you could date is one way to boost your romantic self-esteem, it's not necessarily the only strategy.
The most important thing to keep in mind about your romantic self-esteem is that it's heavily affected by the availability heuristic. If I was encouraged by someone in 2015, that won't do much to assuage the sting of discouragement in 2016, except maybe if it happens to come to mind.
Another clue is the idea of a sexual "dry spell". Dry spells are supposed to get worse the longer they go on... which simply means that if your mind doesn't have a recent (available!) incident of success to latch on, you're more likely to feel down.
So to increase your romantic self-esteem, keep a cherished list of thoughts suggesting your desirability is high, and don't worry too much about thoughts suggesting your desirability is low. Here's a freebie: If you're reading this post, it's likely that you are (or will be) quite rich by global standards. I hear rich people are considered attractive. Put it on your list!
Other ideas for raising your romantic self-esteem:
- Take steps to maintain your physical appearance, so you will appear marginally more desirable to yourself when you see yourself in the mirror.
- Remind yourself that you're not a victim if you're making a conscious choice to prioritize other aspects of your life. Point out to yourself things you could be doing to find partners that you're choosing not to do.
I think this is a situation where prevention works better than cure--it's best to work pre-emptively to keep your romantic self-esteem high. In my experience, low romantic self-esteem leads to unproductive coping mechanisms like distracting myself from dark thoughts by wasting time on the Internet.
The other side of the coin is avoiding hits to your romantic self-esteem. Here's an interesting snippet from a Quora answer I found:
In general specialized contemplative monastic organisations that tend to separate from the society tend to be celibate while ritual specialists within the society (priests) even if expected to follow a higher standard of ethical and ritual purity tend not to be.
So, it seems like it's easier for heterosexual male monks to stay celibate if they are isolated on a monastery away from women. Without any possible partners around, there's no one to reject (or distract) them. Participating in a monastic culture in which long-term singlehood is considered normal & desirable also removes a romantic self-esteem hit.
Retreating to a monastery probably isn't practical, but there may be simpler things you can do. I recently switched from lifting weights to running in order to get exercise, and I found that running is better for my concentration because I'm not distracted by attractive people at the gym.
It's not supposed to be easy
I shared a bunch of ideas in this post. But my overall impression is that instilling aromanticism is a very hard problem. Based on my research, even monks and priests have a difficult time of things. That's why I'm curious to hear what the Less Wrong community can come up with. Side note: when possible, please try to make your suggestions gender-neutral so we can avoid gender-related flame wars. Thanks!
[link] Disjunctive AI Risk Scenarios
Arguments for risks from general AI are sometimes criticized on the grounds that they rely on a series of linear events, each of which has to occur for the proposed scenario to go through. For example, that a sufficiently intelligent AI could escape from containment, that it could then go on to become powerful enough to take over the world, that it could do this quickly enough without being detected, etc.
The intent of my following series of posts is to briefly demonstrate that AI risk scenarios are in fact disjunctive: composed of multiple possible pathways, each of which could be sufficient by itself. To successfully control the AI systems, it is not enough to simply block one of the pathways: they all need to be dealt with.
I've got two posts in this series up so far:
AIs gaining a decisive advantage discusses four different ways by which AIs could achieve a decisive advantage over humanity. The one-picture version is:

AIs gaining the power to act autonomously discusses ways by which AIs might come to act as active agents in the world, despite possible confinement efforts or technology. The one-picture version (which you may wish to click to enlarge) is:
These posts draw heavily on my old paper, Responses to Catastrophic AGI Risk, as well as some recent conversations here on LW. Upcoming posts will try to cover more new ground.
"3 Reasons It’s Irrational to Demand ‘Rationalism’ in Social Justice Activism"
The lead article on everydayfeminism.com on March 25:
3 Reasons It’s Irrational to Demand ‘Rationalism’ in Social Justice Activism
The scenario is always the same: I say we should abolish prisons, police, and the American settler state— someone tells me I’m irrational. I say we need decolonization of the land — someone tells me I’m not being realistic.... When those who are the loudest, the most disruptive — the ones who want to destroy America and all of the oppression it has brought into the world — are being silenced even by others in social justice groups, that is unacceptable.
(The link from "decolonization" is to "Decolonization is not a metaphor", to make it clear s/he means actually giving the land back to the Native Americans.)
I regularly see people who describe how social justice activists act accused of setting up a straw man. This article show that the bias of some SJWs against reason is impossible to strawman. The author argues at length that rationality is bad, and that justice arguments shouldn't be rational or be defended rationally. Ze is, or was, confused about what "rationality" means, but clearly now means it to include reason-based argumentation.
This isn't just some wacko's blog; it was chosen as the headline article for the website. I had to click around to a few other articles to make sure it wasn't a parody site.
But it isn't just a sign of how irrational the social justice movement is—it has clues to how it got that way.
The Thyroid Madness : Core Argument, Evidence, Probabilities and Predictions
I've made a couple of recent posts about hypothyroidism:
http://lesswrong.com/lw/nbm/thyroid_hormones_chronic_fatigue_and_fibromyalgia/
http://lesswrong.com/lw/n8u/a_medical_mystery_thyroid_hormones_chronic/
It appears that many of those who read them were unable to extract the core argument, and few people seem to have found them interesting.
They seem extremely important to me. Somewhere between a possible palliative for some cases of Chronic Fatigue Syndrome, and a panacea for most of the remaining unexplained diseases of the world.
So here I've made the core argument as plain as I can. But obviously it misses out many details. Please read the original posts to see what I'm really saying. They were written as I thought, and the idea has crystallised somewhat in the process of arguing about it with friends and contributors to Less Wrong. In particular I am indebted to the late Broda Barnes for the connection with diabetes, which I found in his book 'Hypothyroidism: The Unsuspected Illness', and which makes the whole thing look rather more plausible.
CORE ARGUMENT
(1.1) Hypothyroidism is a disease with very variable symptoms, which can present in many different ways.
It is an endocrine hormone disease, which causes the metabolism to run slow. A sort of general systems failure. Which parts fail first seems random.
It is extraordinarily difficult to diagnose by clinical symptoms.
(1.2) Chronic Fatigue Syndrome and Fibromyalgia look very like possible presentations of Hypothyroidism
(1.3) The most commonly used blood test (TSH) for Hypothyroidism is negative in CFS/FMS
=>
EITHER
(2.1) CFS/FMS/Hypothyroidism are extremely similar diseases which are nevertheless differently caused.
OR
(2.2) The blood test is failing to detect many cases of Hypothyroidism.
It seems that one is either forced to accept (2.1), or to believe that blood hormone levels can be normal in the presence of Hypothyroidism.
There is precedent for this:
Diabetes, another endocrine disorder (this time the hormone is insulin), comes in two forms:
type I : the hormone producing gland is damaged, the blood hormone levels go wrong. (Classical Diabetes)
type II: the blood hormone levels are normal, but for some reason the hormone does not act. (Insulin Resistance)
I therefore hypothesize:
(3) That there is at least one mechanism interfering with the action of the thyroid hormones on the cells.
and
(4) The same, or similar mechanisms can interfere with the actions of other hormones.
A priori, I'd give these hypotheses a starting chance of 1%. They do not seem unreasonable. In fact they are obvious.
The strongest evidence against them is that they are so very obvious, and yet not believed by those whose job it is to decide.
CURRENT STATUS (Estimated probability)
(1.1) Uncontroversial, believed by everyone involved (~100%)
(1.2) Similarly uncontroversial (~100%)
(1.3) By definition. With abnormal TSH, you'd have hypothyroidism (~100%)
(2.1) Universal belief of conventional medicine and medical science, some alternative medicine disagrees (~90%)
(2.2) The idea that the TSH test is inaccurate is widely believed in alternative medicine, and by thyroid patient groups, but largely rejected by conventional medicine (~10%)
(3) There is some evidence from alternative medicine that this might be true (~10%)
(4) My own idea. A wild stab in the dark. But if it happens twice, you bet it happens thrice [1] (~0.000001%)
Some Details
(1.1) Clinical diagnosis of Hypothyroidism is very out of fashion, considered hopelessly unreliable, doctors are actually trained to ignore the symptoms. There is a famous medical sin of 'Overdiagnosing Hypothyroidism', and doctors who fall into sin are regularly struck off.
(1.2) I don't think you'll find anyone who knows about both diseases to dispute this.
(1.3) True by definition. CFS/FMS symptoms plus abnormal TSH would be Hypothyroidism proper, almost no-one would disagree.
(2.1) This is the belief of conventional medicine. But the cause of CFS/FMS is unknown.
Generally the symptoms are blamed on 'stress', but 'stress' seems to be 'that which causes disease'. This 'explanation' seems to be doing little explanatory work. In fact it looks like magical thinking to me.
Medical Scientists know much more about all this than I do, and they believe it.
On the other hand, scientific ideas without verified causal chains often turn out to be wrong.
(2.2) (The important bit: If the TSH test is not solid, there are a number of interesting consequences.)
I've been looking for a few months through the endocrinological literature for evidence that the sensitivity of the TSH test was properly checked before its introduction or since, and I can't find any. It seems to have been an unjustified assumption. At the very least, my medical literature search skillz are not up to it. I appeal for help to those with better skillz.
It is beyond doubt that atrophy or removal of the thyroid gland causes the TSH value to go extremely high, and such cases are uncontroversial.
The actual interpretation of the TSH test is curiously wooly.
It has proved very difficult to pin down the 'normal range' for TSH, and they have been arguing about it for nearly forty years, over which the 'normal range' has been repeatedly narrowed
The AACB report of 2012 concluded that the normal range was so narrow that huge numbers of people with no symptoms would be outside it, and this range is not widely accepted for obvious reasons
There are many other possible blood hormone tests for hypothyroidism. All are considered to be less accurate or less sensitive than the TSH test. It does seem to be the best available blood test. It does not correlate particularly well with clinical symptoms.
(3) Broda Barnes, a conventional endocrinologist working before the introduction of reliable blood tests, was convinced that the most accurate test was the peripheral basal body temperature on waking.
He considered measuring the basal metabolic rate, and rejected it for good reasons. He considered that desiccated thyroid was a good treatment for the disease, and thought the disease very common. He estimated its prevalence at 40% in the American population. His work is nowadays considered obsolete, and ignored. But he seems to have been a careful, thoughtful scientist, and the best arguments against his conclusions are placebo-effect and confirmation bias. He treated thousands of patients, his treatments were not controversial at the time, and he reported great success. He wrote a popular book 'Hypothyroidism: The Unsuspected Illness', and his conclusions have fathered a large and popular alternative medicine tradition.
John Lowe, a chiropractor who claimed that fibromyalgia could be cured with desiccated thyroid, found that many (25%) of his patients did not respond to the treatment. He hypothesised peripheral resistance, thought it genetic, and used very high doses of the thyroid hormone T3 on many of his patients, which should have killed them. I have read many of his writings, they seem thoughtful and sane. I am not aware of any case in which John Lowe is thought to have done harm. There must be some, even if he was right. But if he was wrong he should have killed many of his patients, including himself. He was either a liar, or a serial murderer, or he was right. He was likely seeing an extremely biased sample of patients, those who could not be helped by conventional approaches.
(4) I just made it up by analogy.
There is the curious concept of 'adrenal fatigue', widespread in alternative medicine but dismissed as fantasy outside it, where the adrenal glands (more endocrine things) are supposed to be 'tired out' by 'excessive stress'. That could conceivably be explained by peripheral resistance to adrenal hormones.
CONSEQUENCES
If (3) is true but (4) is not:
There are a number of mysterious 'somatoform' disorders, collectively known as the central sensitivity syndromes, with many symptoms in common, which could be explained as type 2 hypothyroidism. Obvious cases are Chronic Fatigue Syndrome, Fibromyalgia Syndrome, Major Depressive Disorder and Irritable Bowel Syndrome, but there are many others. Taken together they would explain Broda Barnes' estimate of 40% of Americans.
If (4) is true:
Then we can probably explain most of the remaining unexplained human diseases as endocrine resistance disorders.
HOW CAN THIS BE TRUE, BUT HAVE BEEN MISSED?
This is the million-dollar question!
My favourite explanation is that in order to overwhelm 'peripheral resistance to thyroid hormones', one needs to give the patient both T4 and T3 in exactly the right proportions and dose.
Supplementation with T4 alone will not increase the levels of T3 in the system, since the conversion is under the body's normal control, and the body defends T3 levels.
But T3 is the 'active hormone'. Without significantly increasing the circulating levels of T3, the resistance cannot be overwhelmed.
On the other hand, any significant overdosing of T3 will massively overstimulate the body, causing the extremely unpleasant symptoms of hyperthyroidism.
This seems to me to be sufficient explanation for why various trials of T4 supplementation on the central sensitivity disorders have all failed. In almost all cases, the patients will either have seen no improvement, or have experienced the symptoms of over-treatment. Only in very few cases will any improvement have occurred, and standard trials are not designed to detect such effects.
It is actually just luck that the T4/T3 proportion in desiccated thyroid is about right for some people.
Alternatively, there may just be some component in desiccated thyroid whose action we don't understand.
PERSONAL EXPERIENCE
I displayed symptoms of mild-to-moderate Chronic Fatigue Syndrome, and my wonderful NHS GP checked everything it could possibly be. All my blood tests normal, TSH=2.51. I was heading for a diagnosis of CFS.
After four months I mysteriously partially recovered after trying the iron/vitamin B supplement Floradix, even though I wasn't anaemic.
I started researching on the basis that things that go away on their own tend to come back on their own.
I noticed that I had recorded, in records kept at the time of the illness, thirty out of a list of forty possible symptoms of Hypothyroidism, drew the obvious conclusions as so many others have, and purchased a supply of desiccated thyroid in case it came back.
It did come back, and after one month, I began to self-treat with desiccated thyroid, very carefully titrating small doses against symptoms, and quickly noted immediate huge improvement in all symptoms. In fact I'd say they were gone.
My basal temperature rose over a few weeks from 36.1 to ~36.6 (average, rise slow over several weeks, noise ~ +-0.3 day to day).
One week, holding the dose steady in anticipation of more blood tests, I overdid it by the truly minute amount of 3mg/day of desiccated thyroid, which caused all of the symptoms of the manic phase of bipolar disorder (whose down phase is indistinguishable from CFS, and whose up phase looks terribly like the onset of hyperthyroidism), The manic symptoms disappeared within twelve hours of ceasing thyroid supplementation, to be replaced by overwhelming tiredness.
I resumed thyroid supplementation at a slightly lower dose, and feel as well as I have done for ten years. It's now been ten weeks and I am becoming reasonably confident that it is having some effect.
POSSIBLE CAUSATION
Such catastrophic failures of the body's central control system CANNOT be evolutionarily stable unless they are extremely rare or have compensating advantages.
I am thus drawn to the idea of either:
(a) recent environmental change (which seems to be the alternative medicine explanation)
(b) immune defence (which would explain why e.g. CFS often presents as extended version of the normal post-viral fatigue)
If the alternative is being eaten alive, it seems all too plausible that an immune mechanism might be to 'wall off' cells in some way until the emergency is past, even if catastrophic damage is a side effect.
STRONG PREDICTIONS
Low Body Temperature
It is a very strong prediction of this theory that low basal metabolic rates, and thus low basal peripheral temperatures will be found in many sufferers of Chronic Fatigue Syndrome and Fibromyalgia.
If this is not true, then the idea is refuted unambiguously.
Thyroid Hormone Supplementation as Palliative
It is a less strong prediction, but still fairly strong, that supplementation of the hormones T4 and T3 in carefully titrated doses and proportions will relieve some of the symptoms of CFS/FMS.
Note that T4 supplementation alone is unlikely to work. And that unless the doses and proportions are carefully adjusted to relieve symptoms, the treatment is likely to either not work, or be worse than the disease!
SOME SELECTED POSSIBLE IMPLICATIONS / PREDICTIONS
I've been very reluctant to draw my wilder speculative conclusions in public, since they have the potential to do great harm whether or not the idea is true, but here are some of the less frightening ones that I feel safe stating:
I state them only to encourage people to believe that this problem is worth thinking about.
Endocrinology appears not to be too interested, and my crank emails to endocrinologists have gone unanswered.
One of the reasons that I feel safe stating these four in public is that Broda Barnes thought them obvious and published popular books about them, so they are unlikely to come as a surprise to anyone outside endocrinology:
Dieting/Exercise/Weight Loss
Dieting and Exercise don't work long term as treatments for weight loss. The function of the thyroid system is to adapt metabolism to available resources. Starvation will cause mild transient hypothyroidism as the body attempts to survive the famine it infers. This may be the explanation for Anorexia Nervosa.
Diabetes
Diagnosis of diabetes was once a death sentence. With the discovery of insulin, allowing diabetics to control their blood sugar levels, it became survivable.
However it still has terrible complications, a lot of which look like the complications of hypothyroidism.
If a hormone-resistance mechanism interferes with both insulin and thyroid hormones, the reason for this is obvious. Diabetics with well-controlled blood sugar are dying in their millions from a treatable condition.
Heart Disease
One of the very old tests for hypothyroidism was blood cholesterol. It was thought to be a reliable indicator of hypothyroidism if present, but it was not always present.
A known symptom of hypothyroidism is atherosclerosis and weakness of the heart.
I would imagine that hypothyroidism initially presents as low blood pressure, due to the weakness of the heart. As the arteries clog, the weakened heart is forced to work harder and harder. Blood pressure goes higher and higher, and eventually the heart collapses under the strain.
Blood pressure reducing medications may actually be doing harm. A promising treatment might be to correct the underlying hypothyroidism.
Smoking
Cigarettes are full of poisons, and smoking is correlated with very many diseases.
It could be that smoking causes amongst its effects peripheral resistance, which causes clinical hypothyroidism, which then causes everything it usually causes. And that would be my bet!
It could be that hypothyroidism causes a very great number of bad things, including depression, which then causes smoking.
Smoking may not actually be that dangerous, and it might be possible to mitigate its bad effects.
[1] Madonna, "Pretender", Like A Virgin, Power Station Studios, New York, New York (1984)
I'm going to stop there. There are quite a lot of similar conclusions to be drawn. Read Barnes.
I also have some novel ones of my own which I am not telling anyone about just yet.
What the hell do I, or any of the quacks who have been screaming about this for forty years, have to say in order that someone with real expertise in this area takes this idea seriously enough to have a go at refuting it?
EDIT: This keeps confusing people (including me): Low Basal Metabolic Rates. The amount of oxygen you use once you have been asleep for a while. That's what the thyroid apparently controls in adult animals. Daytime won't do, that's probably under the control of something else. And peripheral temperatures. Not core. We're interested in the amount of heat flowing out of the body. Which is not quite the same thing as temperature....
EDIT : WHY THIS IS WORTH A CLOSE LOOK, EVEN THOUGH IT IS LIKELY WRONG!
Thanks to HungryHobo for making me make this point explicitly:
This is a very simple and obvious explanation of an awful lot of otherwise confusing data, anecdotes, quackery, expert opinion and medical research.
And it is obviously false! Of course medicine has tried using thyroid supplementation to fix 'tired all the time'. It doesn't work!
But there really is an awful lot unexplained about all this T4/T3 business, and why different people think it works differently. I refer you to the internet for all the unexplained things.
In just the endocrinological literature there is a long fight going on about T4/T3 ratios in thyroid supplementation, and about the question of whether or not to treat 'subclinical hypothyroidism'. Some people show symptoms with very low TSH values. Some people have extremely high TSH values and show no symptoms at all.
I've been trying various ways of explaining it all for nearly four months now. And I've found lots of magical thinking in conventional medicine, and lots of waving away of the reports of honest-sounding empiricists, who have made no obvious errors of reasoning, most of whom are taking terrible risks with their own careers in order to, as they see it, help their patients.
I've read lots of people saying 'we tried this, and it works', and no people saying 'we tried this, and it makes no difference'. The explanation favoured by conventional medicine strongly predicts 'we tried this, and it makes no difference'. But they've never tried it! It's really confusing. A lot of people are very confused.
I think that simple explanations are extra-worth looking at because they are simple.
Of course that doesn't mean they're right. Consequence and experiment are the only judge of that.
I do not think I am right! There is no way I can have got the whole picture. I can't explain, for instance 'euthyroid sick syndrome'. But I don't predict that it doesn't exist either.
But you should look very carefully at the simple beautiful ideas that seem to explain everything, but that look untrue.
Firstly because Solomonoff induction looks like a good way to think about the world. Or call it Occam's Razor if you prefer. It is straightforward Bayesianism, as David Mackay points out in Information Theory, Inference, and Learning Algorithms.
Secondly because all the good ideas have turned out to be simple, and could have been spotted, (and often were) by the Ancient Greeks, and could have been demonstrated by them, if only they'd really thought about it.
Thirdly because experiments not done with the hypothesis in mind have likely neglected important aspects of the problem. (In this case T3 homeostasis and possible peripheral resistance and the difference between basal metabolic rate and waking rate, and the difference between core and peripheral temperature and the possibility of a common DIO2 mutation causing people's systems to react differently to T4 monotherapy).
So that even if there are things you can't explain (I can't explain hot daytime fibro-turks...), you should keep plugging away, to see if you can explain them, if you think hard enough.
Good ideas should be given extra-benefit of the doubt. Not ignored because they prove (slightly) too much!
I reckon that we should be able to refute or strongly support the general idea from reports in the published literature. Here is some stuff that I have found recently. There is a comment that looks like this. Add anything you find to it, and I'll move it up here.
ADD EVIDENCE FOR OR AGAINST HERE
Found this for "Wilson's syndrome", but can only see the abstract:
http://www.ncbi.nlm.nih.gov/pubmed/16883675
It looks like it might be supportive, but it also looks crap. No mention of blinding, randomising, or placebo in the abstract.
Can anyone see the actual paper and link to it here? And can anyone work out whether these guys are allies of Wilson, or trying to break him? Because that matters.
This, on the other hand:
http://www.ncbi.nlm.nih.gov/pubmed/9513740
Looks solid, and looks like refutation. They claim normal average core temperatures in CFS. I have quibbles, of course:
I'd expect the core temperature to be well defended. So I'm not worried by that per se, but they do talk about relation to oral temperature, and they do talk about metabolic rate, so they've obviously thought about it, and I can't quite work out what they did there.
Also, the reason that they're measuring this is because their CFS patients have all been complaining about low oral temperatures and the fact that even when they've got a fever, they're not hot. So errr?? Do all the CFS patients believe this theory and are (un)consciously faking? I mean, I can believe that, but is it true that all CFS patients think this theory is true? Who is telling CFS patients to take their temperatures and why?
On the other hand, their actual graphs do look funny. There's a strange shape to the CBT vs time graph in CFS, but n=7, I think, so maybe that's just noise.
These guys:
http://www.sciencedirect.com/science/article/pii/S0024320515301223
Are actually claiming HIGHER peripheral temperatures in Fibromyalgia. But I think they're measuring during the day. I've no idea how to explain that, or what it might mean.
Barnes claimed: Measure axillary temperature on waking. Should be 98.6+/-0.2F (so 37C+/-0.1), lower is bad. Treat with lots of thyroid (1/2-2 grains).
I claim (from just me, and I am perfectly capable of fooling myself): measure oral temperature on waking. Was low (~36.1), has gone higher (36.6-7-8-9) under influence of small amounts of thyroid (1/3 grain). Feel fine now.
Can anyone find: Large numbers of CFS/FMS patients have normal metabolic rate while sleeping or just after waking, no exercise allowed, or normal axillary or oral temperature on waking, again no exercise allowed?
Because that's what I'm looking for at the moment, and it is refutation. I will have to pull off some clever moves indeed to get round that.
Oh, yes, and there's a paper by Lowe himself, finding exactly what I expect him to find:
http://www.ncbi.nlm.nih.gov/pubmed/16810133
Can anyone dig up quibbles with this that can make me discount it?
Oh Jesus:
Clinical Response to Thyroxine Sodium in Clinically Hypothyroid but Biochemically Euthyroid Patients G. R. B. SKINNER MD DSc FRCPath FRCOG, D. HOLMES, A. AHMAD PhD, J. A. DAVIES BSc and J. BENITEZ MSc Vaccine Research Trust, 22 Alcester Road, Moseley, Birmingham B13 8BE, UK
This I can't explain at all! He treated CFS people with tiny amounts of T4, and worked up the dose until they were all better. Worked a treat, apparently. Can anyone break it?
It simultaneously breaks me and proves that CFS is a thyroid problem. I think. Help! Again, no placebos, but a large clinical trial that seems to have worked, by a careful man.
I wouldn't dream of suggesting that anyone steal this using sci-hub.io by typing the title into the search box and then solving the easy CAPTCHA which is in English even though the instructions are all in Russian. You should write to the authors and request a copy instead.
Four 2003 Studies of
Thyroid Hormone Replacement Therapies:
Logical Analysis and Ethical Implications
Dr. John C. Lowe
Lowe again, my rationalist hero, publishing in his own journal, referencing his own papers and books. This time I think he's made maths mistakes. But that's my department, so I'm going to go away and think about it. I mention the paper here to avoid the obvious mistake of deciding whether to mention it after I've had a proper look.
Effective Treatment of Chronic Fatigue Syndrome and Fibromyalgia—A Randomized, Double-Blind, Placebo-Controlled, Intent-To-Treat Study
- Jacob E. Teitelbaum*, Barbara Bird, Robert M. Greenfield, Alan Weiss, Larry Muenz & Laurie Gould
- DOI:10.1300/J092v08n02_02
- ABSTRACT
OK, how do we discount this one? I haven't even read it yet. Can anyone see it?
Thyroid Insufficiency. Is Thyroxine the Only Valuable Drug?
- DOI:10.1080/13590840120083376
- W. V. Baisier, J. Hertoghe & W. Eeckhaut
- ABSTRACT
- Bored now. Anyone find me anything that says this doesn't work?
I've even heard rumours that Lowe himself did PCRTs of his treatments. And probably published them in some chiropractic house mag. I can't even find those.
A rich seam of thyroid vs depression papers, all found through: http://psycheducation.org/
Since he's got a cause, I expect to find them all in favour. I'm going to list them here before reading them in order to avoid the obvious mistake of cherry picking from the cherry basket, and then add comments once I've read them / their abstracts.
Further evidence pointing in the opposite direction is very welcome!
I also tried:
https://www.ncbi.nlm.nih.gov/pubmed/?term=thyroxine+major+depression
and some of those are also here. I can't remember which ones I found through psycheducation and which ones through pubmed.
Bloody browser tabs, sorry, I should have been more careful.
J Affect Disord. 2014 Sep;166:353-8. doi: 10.1016/j.jad.2014.04.022. Epub 2014 May 2.
A favorable risk-benefit analysis of high dose thyroid for treatment of bipolar disorders with regard to osteoporosis.
Kelly T1.
ABSTRACT
High dose thyroid hormone has been in use since the 1930s for the treatment of affective disorders. Despite numerous papers showing benefit, the lack of negative trials and its inclusion in multiple treatment guidelines, high dose thyroid has yet to find wide spread use. The major objection to the use of high dose thyroid is the myth that it causes osteoporosis. This paper reviews the literature surrounding the use of high dose thyroid, both in endocrinology and in psychiatry. High dose thyroid does not appear to be a significant risk factor for osteoporosis while other widely employed psychiatric medications do pose a risk. Psychiatrists are uniquely qualified to do the risk-benefit analyses of high dose thyroid for the treatment of the bipolar I, bipolar II and bipolar NOS. Other specialties do not have the requisite knowledge of the risks of alterative medications or of the mortality and morbidity of the bipolar disorders to do a full risk benefit analysis.
J Clin Endocrinol Metab. 2010 Aug;95(8):3623-32. doi: 10.1210/jc.2009-2571. Epub 2010 May 25.
A randomized controlled trial of the effect of thyroxine replacement on cognitive function in community-living elderly subjects with subclinical hypothyroidism: the Birmingham Elderly Thyroid study.
Parle J1, Roberts L, Wilson S, Pattison H, Roalfe A, Haque MS, Heath C, Sheppard M, Franklyn J, Hobbs FD.
Conclusions:
This RCT provides no evidence for treating elderly subjects with SCH with T4 replacement therapy to improve cognitive function.
J Affect Disord. 2002 Apr;68(2-3):285-94.
Effects of supraphysiological thyroxine administration in healthy controls and patients with depressive disorders.
Bauer M1, Baur H, Berghöfer A, Ströhle A, Hellweg R, Müller-Oerlinghausen B, Baumgartner A.
J Affect Disord. 2009 Aug;116(3):222-6. doi: 10.1016/j.jad.2008.12.010. Epub 2009 Feb 11.
The use of triiodothyronine as an augmentation agent in treatment-resistant bipolar II and bipolar disorder NOS.
Kelly T1, Lieberman DZ.
Am J Psychiatry. 2006 Sep;163(9):1519-30; quiz 1665.
A comparison of lithium and T(3) augmentation following two failed medication treatments for depression: a STAR*D report.
Nierenberg AA1, Fava M, Trivedi MH, Wisniewski SR, Thase ME, McGrath PJ, Alpert JE, Warden D, Luther JF, Niederehe G, Lebowitz B, Shores-Wilson K, Rush AJ.
Nord J Psychiatry. 2015 Jan;69(1):73-8. doi: 10.3109/08039488.2014.929741. Epub 2014 Jul 1.
Well-being and depression in individuals with subclinical hypothyroidism and thyroid autoimmunity - a general population study.
Fjaellegaard K1, Kvetny J, Allerup PN, Bech P, Ellervik C.
Mol Biol Rep. 2014;41(4):2419-25. doi: 10.1007/s11033-014-3097-6. Epub 2014 Jan 18.
Thyroid hormones association with depression severity and clinical outcome in patients with major depressive disorder.
Berent D1, Zboralski K, Orzechowska A, Gałecki P.
Mol Psychiatry. 2016 Feb;21(2):229-36. doi: 10.1038/mp.2014.186. Epub 2015 Jan 20.
Levothyroxine effects on depressive symptoms and limbic glucose metabolism in bipolar disorder: a randomized, placebo-controlled positron emission tomography study.
Bauer M1,2, Berman S2, Stamm T3, Plotkin M4, Adli M3, Pilhatsch M1, London ED2, Hellemann GS5, Whybrow PC2, Schlagenhauf F3.
Abstract
Mol Psychiatry. 2005 May;10(5):456-69.
Supraphysiological doses of levothyroxine alter regional cerebral metabolism and improve mood in bipolar depression.
Bauer M1, London ED, Rasgon N, Berman SM, Frye MA, Altshuler LL, Mandelkern MA, Bramen J, Voytek B, Woods R, Mazziotta JC, Whybrow PC.
Minerva Endocrinol. 2013 Dec;38(4):365-77.
Hypothyroidism and depression: salient aspects of pathogenesis and management.
Duntas LH1, Maillis A.
J Psychiatr Res. 2012 Nov;46(11):1406-13. doi: 10.1016/j.jpsychires.2012.08.009. Epub 2012 Sep 7.
The combination of triiodothyronine (T3) and sertraline is not superior to sertraline monotherapy in the treatment of major depressive disorder.
Garlow SJ1, Dunlop BW, Ninan PT, Nemeroff CB.
Mol Psychiatry. 2016 Feb;21(2):229-36. doi: 10.1038/mp.2014.186. Epub 2015 Jan 20.
Levothyroxine effects on depressive symptoms and limbic glucose metabolism in bipolar disorder: a randomized, placebo-controlled positron emission tomography study.
Bauer M1,2, Berman S2, Stamm T3, Plotkin M4, Adli M3, Pilhatsch M1, London ED2, Hellemann GS5, Whybrow PC2, Schlagenhauf F3.
Lesswrong Survey - invitation for suggestions
Given that it's been a while since the last survey (http://lesswrong.com/lw/lhg/2014_survey_results/)
It's now time to open the floor to suggestions of improvements to the last survey. If you have a question you think should be on the survey (perhaps with reasons why, predictions as to the result, or other useful commentary about a survey question)
Alternatively questions that should not be included in the next survey, with similar reasons as to why...
survey is now up (2016-03-26) http://lesswrong.com/lw/nfk/lesswrong_2016_survey/
= 783df68a0f980790206b9ea87794c5b6)

Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)