Fantastic work, very happy to see concrete data collection like this!
Minor methodology note:
the US as a whole is way less dense than NYC so going to less dense neighborhoods is likely more representative
That doesn't work; less dense in NYC generally means significantly wealthier, and NYC already skews much wealthier than average.
"That doesn't work; less dense in NYC generally means significantly wealthier, and NYC already skews much wealthier than average."
I (attempted) to account for that! I went to both generally poorer and less dense neighborhoods using this neighborhood optimizer I built out: https://sladebyrd.com/canvassing-planner
Which fun fact (perhaps unsurprisingly) led to me canvassing in the neighborhood considered to be the murder capital of NYC. (Which was interestingly my single most productive canvassing session)
There is of course the issue that within a given neighborhood I am still selecting the wealthier residents. I tried adding in bias term to attempt to account for that (ie I assume I am getting responses from people that are $X above the neghborhood median), but I do agree it's hard.
I include the collected as well as bayesian estimated demographical data of my responses at the end of the report.
You are rederiving the science and art of survey design. It's surprisingly difficult. Just ask Claude about it, it's read almost all of the literature. Most surveys are hard to interpret because there are major flaws in their design.
Good job rederiving so much quickly!
I'd be interested to know whether you got AI input before launching this project. I wonder if they'd think to tell you about the difficulties of survey design and execution.
Nobody has ever done an in person door to door survey about AI risks[1]. What do people really think about AI? Like really? There have been some surveys on the risks from AI. But there’s a real difference between looking at numbers on page vs. the feeling of talking to our fellow humans.
I[2] asked 101 people what they thought about the impact of AI. Approximately half of the responses were from ringing doorbells, and half were from asking people out and about[3]. Every single one of these was in person face to face. Around 10 respondents only spoke Spanish and were surveyed in Spanish[4].
Here are the results.
The top level results are very strong when it comes to showing interest in regulating AI. Everything else will be reflections on the results/process and all the qualitative data I picked up from this process.
Thoughts on some Specific Questions
This question was the longest and most complex question. I basically have to read out all of the options since it's not on a simple scale. I decided to first split it on whether someone thinks superhuman AI should be developed at all, and then only give the more granular options if they do think it should be developed. This difference in methodology could explain the difference between my results and FLI results, but it seems unlikely because even still the vast majority of respondents who thought it should be developed still thought it should be developed with serious regulation. It’s possible the initial framing of should or should not be developed biased the responses for the subsets of “Should be developed”. Overall these results are extremely strong though. Of the 96 people who answered with an opinion, only 2 of them thought it should be with current regulations or as fast as possible.
This question was terrible for a couple reasons. First people were frequently confused the “limits on developing more powerful AI” clause is just brutal. This was frequently misunderstood and my model is basically people just remember the most recent thing, so they simply remembered “more powerful AI” and discarded the limits part. People would be like yeah I am super worried about AI I hate it I never use it and then answer “much less likely”. I eventually ended up repeating “limits on developing more powerful AI” two times to try to help, but I think this was a good lesson on you want the absolute simplest question construction possible. There is just huge variance in reading comprehension and limiting that as a confounding factor is really important. Similarly, I started with “controls on developing more powerful AI” and I had at least 1 person interpret that as the politicians getting control over the AIs themselves. I tried to be quite careful with my wording, and this is part of why I made the simplest explanation of superhuman AI I could think of for question 3, but even still the takeaway here is when designing surveys you want basically no words that alter meaning, especially if they are farther away in the sentence. This question was also interesting in that it also got confounded by just general anti-politician sentiment. Some people answered less likely on the grounds that you simply can’t trust politicians at all. I think likely if I were to do this again I would simplify and ask if they would support a law instead of support a politician[5]. There has been very little published surveying on the salience of AI risk to electability, so I am still happy I included this question.
Takeaways
Very few people are excited about AI. Almost everyone is worried about superhuman[6] AI becoming too powerful for humans to control. The vast majority support slower development, international treaties limiting AI, and are just generally worried. People are generally worried about AI, even some of the people who said they were more excited than concerned about AI still frequently expressed worries especially in the context of questions about superhuman AI. My vibes based read is many of the people who are more excited are more excited because they don’t think superhuman AI is actually possible, but then when given hypotheticals about superhuman AI they are still worried about it/don’t want it to be developed[7].
I did a pretty good job of getting a sample that is relatively representative of the US along the axes of Ethnicity/Income/Education/Politics, however there is certainly bias from only getting responses in NY/NJ. That being said the responses are generally very indicative of AI risk being both salient and politically advantageous (at least when it comes to electability) for both D and R candidates to try to regulate further.
Surveying is Inherently Biased
I had never canvassed before, it is crazy how many ways there are for bias to sneak in.
People are Weird and Surprising
The beauty of talking to people in person is you get tons of qualitative data to go along with each response. Here are some interesting things that happened:
You Can Do This Too!
I built the survey app as a PWA (progressive web app) that works offline anywhere. I would strongly recommend trying this out yourself if you care about AI and enjoy walking around outside. Doorbells can be kind of depressing so I wouldn’t suggest that for your first time, but if you go to a park on a nice day[13] it’s super easeful! Just walk up and ask whoever looks like they might be open to chat, the worst that happens is they say no! If you want it to be even more fun do it with[14] a friend[15], now you’re walking around a park on beautiful day with someone you like talking to occasionally getting to hear wild takes from strangers on AI!
It’s easy for me to add you as a canvasser, all you have to do is:
As far as Claude can tell at least.
along with two friends of mine.
Primarily in parks, but also on the street.
Which is also, as far as I can tell, the first US based Spanish speaking survey results.
Although I’m not really sure since part of the hope in asking a question specifically voter preferences is that it’s more compelling to a politician taking on an issue if they know it will help them get elected instead of knowing that people support that type of law in general.
Defined as: “Some companies are trying to build superhuman AI that would be far smarter than any human at nearly everything.”
There were a couple people who didn’t answer the question on whether superhuman AI should be developed because they didn’t think it was possible and refused to engage with the hypothetical.
But of course this does add a little bit of bias, there might be some sort of systematic bias between the type of person who turns lights on (or doesn’t fully block out their windows, etc) and the type of person who doesn’t.
Huge range of education, intelligence, etc among respondents
“A computer that does the thinking of humans, like chatgpt, self driving cars, siri, or alexa.” Self driving cars was frequently the thing that was most helpful/understandable. It’s also kind of scary that there are people who have never heard of AI before, but they are certainly still seeing AI generated content on social media etc.
When I talked about how if AI was banned it would be developed far slower since it requires lots of capital and developers etc, he was like yeah that’s irrelevant.
And frequently this is also paired with never actually answering a question or getting mad at me for asking them to specifically choose one of the options. This is a case where I really have to use my best judgement to figure out which answer is most representative of the wild things coming out of their mouth.
Ideally on a weekend, but even on weekdays there can be plenty of people around if it’s nice out.
Thoughts from friend 1: “I had a lot of fun going canvassing around Sunset Park today. I have canvassed for a political candidate before and that felt a lot more standardized than this. Consistently, as we were asking people in the park today about AI, they were surprised and intrigued by the questions. People have lots of opinions, and it seems like some of them jump at the opportunity to have their voice heard. I was particularly intrigued by some of the responses that we got. I know I live in a bubble of over-educated and technologically literate people, but it’s still surprising to talk to people who have never used AI, let alone never have even heard of what it means for something to be “artificially” intelligent. It was also a really great opportunity to be forced to practice my Spanish in a community where 90% of the people we spoke to did not seem to speak English.”
Thoughts from friend 2: “There were responses to some questions that I expected would lead to certain responses on other questions but sometimes I was surprised by what I perceived as lack of cohesion in the logic of people’s answers. It was fun to walk around with you and hang out in a neighborhood I’m never in.”
If you don’t, I will auto-generate a password for you.
Inside the app, it’s very simple