Comment Permalink

I like the framework through which you're approaching this (i.e. managing complexity)

Something I notice that I wish I had was access to my answers to past year's surveys, to track how my answers have evolved over time. The effort for facilitating that is probably nontrivial, but something you could do to start enabling backwards compatibility for that is including some kind of identifying tag that you're likely to remember in the future but is untraceable (or, leaves it to individuals to decide how to tradeoff "I will easily remember this next year" vs "this can't be used to identify me"

Questions that might be good are "CFAR followup" type questions, like "Have you tried a new thing in the past week/month? When's the last time you made a major career change on purpose?"

Screwtape8y10

Trivial suggestion- include in the survey a question that goes something like "roll a die ten times, and write the results in order here. If you would like to be able to identify yourself on this survey later, make a copy of those results and look for them in the dataset when it is released. If you want to make sure nobody can possibly identify you on this survey, leave this question blank."

If you want to make it something people might remember without writing it down, ask them to generate a couple of words from the diceware list.

1namespace8y

Previous surveys have included this, I just overlooked it. No the effort to facilitate at least the weak form you're talking about is fairly trivial. In terms of looking at your personal results it might be possible to coax the software into sending you a copy of your answers, if I remember correctly the software used last time supports this but I turned it off as an extraneous feature.

See in context

8 Requesting Questions For A 2017 LessWrong Survey

by namespace

9th Apr 2017

3 min read

8

It's been twelve months since the last LessWrong Survey, which means we're due for a new one. But before I can put out a new survey in earnest, I feel obligated to solicit questions from community members and check in on any ideas that might be floating around for what we should ask.

The basic format of the thread isn't too complex, just pitch questions. For best chances of inclusion, however, it's best to include:

A short cost/benefit analysis of including the question. Keep in mind that some questions are too invasive or embarrassing to be reasonably included. Other questions might leak too many bits. There is limited survey space and some items might be too marginal to include at the cost of others.
An example of a useful analysis that could be done with this question(s), especially interesting analysis in concert with other questions. eg. It's best to start with a larger question like "how does parental religious denomination affect the cohorts current religion?" and then translate that into concrete questions about religion.
Some idea of how the question can be done without using write-ins. Unfortunately write-in questions add massive amounts of man-hours to the total analysis time for a survey and make it harder to get out a final product when all is said and done.

The last survey included 148 questions; some sections will not be repeated in the 2017 survey, which gives us an estimate about our question budget. I would prefer to not go over 150 questions, and if at all possible come in at many fewer than that. Removed sections are:

The Basilisk section on the last survey provided adequate information on the phenomena it was surveying, and I do not currently plan to include it again on the 2017 survey. This frees up six questions.
The LessWrong Feedback portion of the last survey also adequately provided information, and I would prefer to replace it on the 2017 survey with a section measuring the site's recovery, if any. This frees up 19 questions.

I also plan to do significant reform to multiple portions of the survey. I'm particularly interested in making changes to:

The politics section. In particular I would like to update the questions about feelings on political issues with new entries and overhaul some of the options on various questions.

I handled the calibration section poorly last year, and would like to replace it this year with an easily scored set of questions. To be more specific, a good calibration section should:
- Good calibration questions should be fermi estimable with no more than a standard 5th grade education. They should not rely on particular hidden knowledge or overly specific information. eg. "Who wrote the foundation novels?" is a terrible calibration question and "What is the height of the Eiffel Tower in meters within a multiple of 1.5?" is decent.
- Good calibration questions should have a measurable distance component, so that even if an answer is wrong (as the vast majority of answers will be) it can still be scored.
- A measure of distance should get proportionately smaller the closer an answer is to being correct and proportionately larger the further it is from being correct.
- It should be easily (or at least sanely) calculable by programmatic methods.
The probabilities section is probably due for some revision, I know in previous years I haven't even answered it because I found the wording of some questions too confusing to even consider.

So for maximum chances of inclusion, it would be best to keep these proposed reforms in mind with your suggestions.

(Note: If you have suggestions on questions to eliminate, I'd be glad to hear those too.)

Surveys

Personal Blog

8

New Comment

30 comments, sorted by

top scoring

Click to highlight new comments since: Today at 6:01 AM

[-]J Thomas Moros8y50

I and some other rationalists have been thinking about cryonics a lot recently and how we might improve the strength of cryonics offerings and the rate of adoption. After some consideration, we came up with a couple suggestions for changes to the survey that we think would be helpful and interesting.

A question along the lines of "What impact do you believe money and attention put towards life extension or other technologies such as cryonics has on the world as a whole?" Answers:
- Very positive
- Positive
- Neutral
- Negative
- Very Negative
The purpose of this question is to evaluate whether the community feels that resources put toward the benefit of individuals through life extension and cryonics has a positive or negative impact on the world. For example, people who expect to live longer may have more of a long term orientation, leading them to do more to improve the future.
Add to the question about being signed up for cryonics an option along the lines of "No, I would like to sign up but can't due to opposition I would face from family or friends". We hear this is one of the reasons people don't sign up for cryonics. It would be great to get some numbers on this, and it doesn't add an extra question, just an extra option for that question.

[-]ChristianKl8y40

On average how many days do you eat meat per week? (Y/N)

Did you eat meat yesterday? (Y/N)

Having the two questions is interesting because in aggregate it allows us to test people's abilities to estimate their average meet consumption.

[-]Screwtape8y30

Perhaps a question like "At what age did you first encounter or begin to develop Rationality?"

I currently think it's very likely the thought patterns that make one a useful programmer are in place sometime in grade school. Given the overlap between rationalists and programmers, I'm curious if something similar is going on. I managed to run a small test by handing HP:MoR to a bunch of kids in one of the local library programs and the ones who were already growing into a geeky/intellectual cluster took to it pretty well.

If we want to do outreach, knowing whether we're convincing people with different kinds of minds to think rationally or just finding people who already sort of think like this sounds useful. Knowing the age at which say, HP:MoR or the sequences could be understood means we have more data whether introducing college students, high school students, or grade school students to them is going to be interesting or boring to them. I don't think we can necessarily get that answer from just asking what age people started developing this mindset, nor do I think their answers will be without error bars since we may be talking about childhood memories. Also, I'm genuinely curious- are the stories of developing this strain of thought in the single digit ages weird exceptions or the typical origin story of a rationalist?

Apologies if that kind of question has already been explored.

[-]Viliam8y20

"At what age did you first encounter or begin to develop Rationality?"

Huh, this is a difficult one, because I had some nebulous ideas in this direction already in childhood, but only a few decades later I read the Sequences and realized that this is a crystallized version of what was already in my mind. Not sure whether it's proper to credit the younger me for "Rationality" or not.

[-]Screwtape8y20

Right, which is why how the question is phrased is tricky. I wouldn't expect someone to devise something as distilled and deliberate as the Sequences independently or to use the title "Rationality" for what they're doing, for example. Like you though, reading the Sequences for the first time felt less like entirely new things and more like finding clearer expressions and vocabulary for a style of thought that was already in my head. It's how common that experience is that I would find interesting- how many people were in this sort of intellectual cluster before encountering the Sequences? Do they usually teach a new kind of approach, or do they upgrade a style of thought that existed beforehand?

I know I'm not phrasing this question well enough for the survey at present. If what I'm getting at makes sense and is interesting, assistance rephrasing it would be appreciated.

[-]Viliam8y20

Sorry, no obviously good ideas how to rephrase the question. Perhaps:

"At what age did you first encounter or begin to develop parts of what you currently recognize as Rationality?"

For me, it was mostly a vague feeling, something like -- of course without these specific words -- "There is a territory... and most people seem not to give a fuck about it, escaping to their fantasies instead (religion, positive thinking, quantum mumbo jumbo, political mindkilling, etc.). The only difference seems to be that some people are proudly and publicly irrational since the very beginning, while others are rational in the near mode of everyday life and only switch to the irrational mode at some later point, suggesting that it is a higher form of thinking (e.g. they first study physics seriously, learn the equations, get a degree... and then they read a book about quantum mumbo jumbo and fall in love with it, continuing to do the equations in their everyday job, but thinking the mumbo jumbo is the higher level of insight). I too understand the temptations of this atractor, I just can't make myself believe that the reality will go away when I close my eyes (and there were moments when I desperately tried this, just to fit in the human society, but it didn't work)." And for a large part of my life, I suspected I might be alone with this condition, which was quite weird.

[-]ChristianKl8y00

I don't think there's one main rationality mindset.

I for example believed in the awful concepts of natural talents the whole time I was in high school. It was the mindset that my teachers and parents had, so I didn't know any better. I think if my teachers would have had the opposite belief, I would also have gotten that mindset at an early age.

When it comes to caring about understanding reality, I think I always did.

When it comes to "it's not enough to have smart arguments, it's more important to be right" I can't say when I picked that up. I don't think I had it at an early age but I don't see why it shouldn't be possible to learn it earlier.

[-]JonahS8y20

I would like to see an (optional) personality test section - email me at jsinick@gmail.com if you're interested in the possibility, as I have some detailed thoughts.

[-]btrettel8y20

I'd be interested in a question about aerobic fitness. My impression is that most rationalists severely underrate aerobic physical activity compared against anaerobic, which is surprising because anaerobic doesn't help cardiovascular capacity much. Presumably given the interest in cryonics and whatnot here, rationalists are interested in living longer. Cardiovascular capacity (VO2max, typically) is strongly correlated with longevity, and it's easy to see the direction of causation.

Possible question: "Over the past month, have you typically met the US federal guidelines for aerobic physical activity? This means at least 150 minutes (2 hours and 30 minutes) a week of moderate-intensity, or 75 minutes (1 hour and 15 minutes) a week of vigorous-intensity aerobic physical activity, or an equivalent combination of moderate- and vigorous-intensity aerobic activity. See health.gov for more information."

There's a lot of data on this question, so it will be easy to compare LessWrongers against other groups.

[-]namespace8y20

I don't know if I'll include this or not yet, but I just wanted to thank you for your awesome presentation of this question. Great sample question, excellent assurance that the data will in fact be analyzable afterward in comparison to other studies, good connection made to 'rationalist' type interests.

I'm impressed, and definitely considering it.

[-]ChristianKl8y10

If there's a question on this topic I don't think the answer should be binary.

[-]btrettel8y00

Good point. Ideal would be entering a number. If I recall correctly the actual guidelines are written in terms of MET-minutes, a weird exercise specific unit of energy. The entire "moderate intensity exercise" thing is a simplification of the actual recommendation. I'm not sure how much participation would decrease if we generalized from binary in this way.

[-]ChristianKl8y00

Even multiple choice with 5 answers might be better than a binary answer.

[-]Raemon8y20

I like the framework through which you're approaching this (i.e. managing complexity)

Questions that might be good are "CFAR followup" type questions, like "Have you tried a new thing in the past week/month? When's the last time you made a major career change on purpose?"

[-]Screwtape8y10

If you want to make it something people might remember without writing it down, ask them to generate a couple of words from the diceware list.

[-]namespace8y10

Previous surveys have included this, I just overlooked it.

The effort for facilitating that is probably nontrivial

No the effort to facilitate at least the weak form you're talking about is fairly trivial. In terms of looking at your personal results it might be possible to coax the software into sending you a copy of your answers, if I remember correctly the software used last time supports this but I turned it off as an extraneous feature.

[-]Viliam8y10

Just please make sure that the more important and shorter questions are first, that you can skip any question, and that you can submit a half completed questionnaire.

(I hate it when I spend 30 minutes filling out some questionnaire, only to find out that there are long nebulous questions at the end, I already don't have more free time, and there is no option to skip those questions, so I just have to throw all my work away.)

[-]gjm8y00

so I just have to throw all my work away

Or write something like "sorry, no time to give a real answer to this question" for each of the long nebulous ones. Probably almost as easy as giving up, and more useful-in-expectation to the people who made the questionnaire. (If there's some reward for filling it in, this may reduce the probability that you're eligible, but I doubt you're filling in questionnaires just for the sake of the rewards offered.)

[-]David Althaus8y10

Here is another question that would be very interesting, IMO:

“For what value of X would you be indifferent about the choice between A) creating a utopia that lasts for one-hundred years and whose X inhabitants are all extremely happy, cultured, intelligent, fair, just, benevolent, etc. and lead rich, meaningful lives, and B) preventing one average human from being horribly tortured for one month?"

[-]David Althaus8y10

I think it's great that you're doing this survey!

I would like to suggest two possible questions about acausal thinking/superrationality:

Newcomb’s problem: one box or two boxes?

Accept: two boxes

Lean toward: two boxes

Accept: one box

Lean toward: one box

Other

(This is the formulation used in the famous PhilPapers survey.)

Would you cooperate or defect against other community members in a one-shot Prisoner’s Dilemma?

Definitely cooperate

Leaning toward: cooperate

Leaning toward: defect

Definitely defect

Other

I think that these questions are not only interesting in and of itself, but that they are also highly important for further research I'd like to conduct. (I can go into more detail if necessary.)

[-]ChristianKl8y10

"Is it useful to identify oneself as an aspiring rationalist?"

[-]ChristianKl8y10

Two possible calibration questions:

How many people who answered this survey are from the US?
How many people who answered this survey are female?

[-]Elo8y00

Which of the following substances do you take on a regular or semi-regular basis? (multiple options possible, more frequently than once a month)

Soylent (purchased or DIY)
Mealsquares
Whey protein (or some other form of concentrated protein)
Fish oil
Vitamin D (in tablet form)
Melatonin
Creatine
Magnesium
Sugary drinks (any sugar-based carbonated drinks, cordial or flavoured water drink)
Diet or "sugar-free" drinks
chocolate
caffeine (coffee, tea, tablets)
nicotene
meat
I decline to answer some of these
I answered all of these correctly

[-]scarcegreengrass8y00

Suggestion: Since the future of intelligent software is a big interest in the community, i'd be curious to get more resolution on people's worldviews. We could try to get numbers for opinions like:

'I don't have much to say about this topic right now.' <- for people who are interested in non-machine-intelligence parts of LW

'There's a probability of __ that Bostrom-style superintelligent software will dominate the history of the 2017 to 2100 CE period.'

Same as the above, but for 2200 to 3000 CE.

'There's a probability of __ that whole brain emulation will dominate the history of the 2017 to 2100 CE period.'

Same as the above, but for 2200 to 3000 CE.

etc

I'd also be curious to see data on worldviews about the future impact of:

Slightly more traditional forms of automation, computing, and software (such as workforce automation, future financial software, etc).

Biotech and genetic engineering

Decentralized manufacturing and nanoscale technologies

The space industry

Weapons of mass destruction

Novel political systems.

This is a lot of questions, so feel free to trim or select as desired.

[-]scarcegreengrass8y00

Suggestion: A question that distinguishes between average and total utilitarianism (i dont remember seeing this previously). This is a little arcane for people who aren't interested in consequentialism & utilitarianism, and that's fine. But within utilitarianism, i'd be interested to know more about this split. It has a bearing on whole brain emulation.

(Just in case i'm using the wrong terminology, let me clarify: I'm referring to whether or not you differentiate between a universe with 10 billion humans at X quality of life or a universe with 20 billion humans at the same X quality of life. I think most people prefer the 20 billion, but some people might lean towards the 10 billion as a matter of degree or in less theoretical contexts.)

[-]Unnamed8y00

I'd like to see some changes to the CFAR-related questions; I've sent a PM with details.

[-]namespace8y00

I have not received one. Please re-send.

[-]ChristianKl8y00

"Do you have mnemonics pegs for the numbers of 1 to 100?"

[-]scarcegreengrass8y00

Just out of curiosity, what sort of things do you mean? Like Schelling points or like information associated with those numbers?

[-]ChristianKl8y10

In mnemonics there's the recommendation to have pegs for numbers. I personally have a person for every number between 1 to 100 and I can visualize that person to make it easier to remember the number.

There are other systems such as https://en.wikipedia.org/wiki/Mnemonic_major_system .

Moderation Log