All of jungofthewon's Comments + Replies

Sure! Prior to this survey I would have thought:

  1. Fewer NLP researchers would have taken AGI seriously, identified understanding its risks as a significant priority, and considered it catastrophic. 
    1. I particularly found it interesting that underrepresented researcher groups were more concerned (though less surprising in hindsight, especially considering the diversity of interpretations of catastrophe). I wonder how well the alignment community is doing with outreach to those groups. 
  2. There were more scaling maximalists (like the survey respondents di
... (read more)

This was really interesting, thanks for running and sharing! Overall this was a positive update for me. 

Results are here

I think this just links to PhilPapers not your survey results? 

1Evan R. Murphy
Can you say more about how this was a positive update for you?
1Sam Bowman
Thanks! Fixed link.

and Ought either builds AGI or strongly influences the organization that builds AGI.

 

"strongly influences the organization that builds AGI" applies to all alignment research initiatives right? Alignment researchers at e.g. DeepMind have less of an uphill battle but they still have to convince the rest of DeepMind to adopt their work. 

2elifland
  Yes, I didn't mean to imply this was necessarily an Ought-specific problem and I guess it may have been a bit unfair for me to only do a BOTEC on Ought. I included it because I had the most fleshed-out thoughts on it but it could give the wrong impression about relative promise when others don't have BOTECs. Also people (not implying you!) often take my BOTECs too seriously, they're done in this spirit. That being said, I agree that strong within-organization influence feels more likely than across; not sure to what extent.

I also appreciated reading this.

I found this post beautiful and somber in a sacred way.  Thank you.

This was really helpful and fun to read. I'm sure it was nontrivial to get to this level of articulation and clarity. Thanks for taking the time to package it for everyone else to benefit from. 

If anyone has questions for Ought specifically, we're happy to answer them as part of our AMA on Tuesday.

I think we could play an endless and uninteresting game of "find a real-world example for / against factorization."

To me, the more interesting discussion is around building better systems for updating on alignment research progress -   

  1. What would it look like for this research community to effectively update on results and progress? 
  2. What can we borrow from other academic disciplines? E.g. what would "preregistration" look like? 
  3. What are the ways more structure and standardization would be limiting / taking us further from truth? 
  4. Wh
... (read more)
8johnswentworth
The problem with not using existing real-world examples as a primary evidence source is that we have far more bits-of-evidence from the existing real world, at far lower cost, than from any other source. Any method which doesn't heavily leverage those bits necessarily makes progress at a pace orders of magnitude slower. Also, in order for factorization to be viable for aligning AI, we need the large majority of real-world cognitive problems to be factorizable. So if we can find an endless stream of real-world examples of cognitive problems which humans are bad at factoring, then this class of approaches is already dead in the water.

Thanks for that pointer. It's always helpful to have analogies in other domains to take inspiration from.

I enjoyed reading this, thanks for taking the time to organize your thoughts and convey them so clearly! I'm excited to think a bit about how we might imbue a process like this into Elicit

This also seems like the research version of being hypothesis-driven / actionable / decision-relevant at work. 

love. very beautifully written. today i will also try to scoot n+1 inches. 

2SamuelKnoche
This one should work: https://discord.gg/56cve5YZQH

 Access

Alignment-focused policymakers / policy researchers should also be in positions of influence. 

Knowledge

I'd add a bunch of human / social topics to your list e.g. 

  • Policy 
  • Every relevant historical precedent
  • Crisis management / global logistical coordination / negotiation
  • Psychology / media / marketing
  • Forecasting 

Research methodology / Scientific “rationality,” Productivity, Tools

I'd be really excited to have people use Elicit with this motivation. (More context here and here.)

Re: competitive games of introducing new tools, we di... (read more)

This is exactly what Ought is doing as we build Elicit into a research assistant using language models / GPT-3. We're studying researchers' workflows and identifying ways to productize or automate parts of them. In that process, we have to figure out how to turn GPT-3, a generalist by default, into a specialist that is a useful thought partner for domains like AI policy. We have to learn how to take feedback from the researcher and convert it into better results within session, per person, per research task, across the entire product. Another spin on it: w... (read more)

Ought is building Elicit, an AI research assistant using language models to automate and scale parts of the research process. Today, researchers can brainstorm research questions, search for datasets, find relevant publications, and brainstorm scenarios.  They can create custom research tasks and search engines.  You can find demos of Elicit here and a podcast explaining our vision here.  

We're hiring for the following roles:

... (read more)

"Remember that you are dying."

When Elicit has nice argument mapping (it doesn't yet, right?) it might be pretty cool and useful (to both LW and Ought) if that could be used on LW as well. For example, someone could make an argument in a post, and then have an Elicit map (involving several questions linked together) where LW users could reveal what they think of the premises, the conclusion, and the connection between them.

Yes that is very aligned with the type of things we're interested in!! 

Lots of uncertainty but a few ways this can connect to the long-term vision laid out in the blog post:

  1. We want to be useful for making forecasts broadly. If people want to make predictions on LW, we want to support that. We specifically want some people to make lots of predictions so that other people can reuse the predictions we house to answer new questions. The LW integration generates lots of predictions and funnels them into Elicit.  It can also teach us how to make predicting easier in ways that might generalize beyond LW. 
  2. It's unclear how e
... (read more)
5abramdemski
Ah, a lot of this makes sense! So you're from Ought? Yep, OK, this makes sense to me. Right, OK, this makes sense to me as well, although it's certainly more speculative. When Elicit has nice argument mapping (it doesn't yet, right?) it might be pretty cool and useful (to both LW and Ought) if that could be used on LW as well. For example, someone could make an argument in a post, and then have an Elicit map (involving several questions linked together) where LW users could reveal what they think of the premises, the conclusion, and the connection between them.

I see what you're saying. This feature is designed to support tracking changes in predictions primarily over longer periods of time e.g. for forecasts with years between creation and resolution. (You can even download a csv of the forecast data to run analyses on it.)

It can get a bit noisy, like in this case, so we can think about how to address that. 

you mean because my predictions are noisy and you don't want to see them in that list? 

1jmh
Unless I'm off base it looks like you form four different predictions on the same question. That seems odd to me. I would expect a one prediction per person making a prediction -- so later predictions would update rather than provide a new value along with the prior ones. It looks like you hold all four positions simultaneously. Also, if they are all considered current predictions then that might skew the average. But maybe I am just not getting something about what's going on. Have not really looked beyond the LW post and comments.

try it and let's see what happens! 

1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
11%
12%
13%
14%
15%
16%
17%
18%
19%
20%
21%
22%
23%
24%
25%
26%
27%
28%
29%
30%
31%
32%
33%
34%
35%
36%
37%
38%
39%
40%
41%
42%
43%
44%
45%
46%
47%
48%
49%
50%
51%
52%
53%
54%
55%
56%
57%
58%
59%
60%
61%
62%
63%
64%
65%
66%
67%
68%
69%
70%
71%
72%
73%
74%
75%
76%
77%
78%
79%
80%
81%
82%
83%
84%
85%
86%
87%
88%
89%
90%
91%
92%
93%
94%
95%
96%
97%
98%
99%
1%
99%
6jacobjacob
Your prior should be that "people don't do stuff", community looking way too optimistic at this time

You've just handed me so much power to prove people right or wrong on the internet... 

3Amandango
I'm counting using this to express credence on claims as a non-prediction use!

Haha I didn't find it patronizing personally but it did take me an embarrassingly long time to figure out what Filipe did there :) Resource allocation seems to be a common theme in this thread. 

Yes! For example I am often amazed by people who are able to explain complex technical concepts in accessible and interesting ways 

Yes-anding you: our limited ability to run "experiments" and easily get empirical results for policy initiatives seems to really hinder progress. Maybe AI can help us organize our values, simulate a bunch of policy outcomes, and then find the best win-win solution when our values diverge. 

I love the idea of exploring different minds and seeing how they fit. Getting chills thinking about what it means for humanity's capacity for pleasure to explode. And loving the image of swimming through a vast, clear, blue mind design ocean.  

Doesn't directly answer the question but: AI tools / assistants are often portrayed as having their own identities. They have their own names e.g. Samantha, Clara, Siri, Alexa. But it doesn't seem obvious that they need to be represented as discrete entities. Can an AI system be so integrated with me that it just feels like me on a really really really good day? Suddenly I'm just so knowledgeable and good at math! 

2jefflab
And also the inverse: helping you avoid doing things you don't want to do. For example, observing that you are over-reacting to an ambiguous email rather than giving them the benefit of the doubt. Or more seriously recognizing you are about to fall off the wagon with substance abuse and prompting you to reconsider. (e.g. mitigating the part of human nature described in Romans 7:15, "I do not understand what I do. For what I want to do I do not do, but what I hate I do.")

Instant translation across nueroatypical people, just like instant translation between English and Korean. An AI system that helps me understand what an autistic individual is currently experiencing, helps me communicate more easily with them. 

3jacobjacob
Yes, and something that could build up a model of different people's minds, and figure out the optimal way of transferring a representation between the two. What if had a language for master Go players to communicate their intuitions about board positions? Master composers to communicate how to write a symphony? Mathematicians to communicate the right intuitions for a particular theorem?  I mean we do have human language, which is remarkable in the extent to which it enables these things. But I have an intuition that we could do a lot better here (e.g. doing it in a day instead of 10 years, communicating things the experts can't even put into words themselves yet). Imagining the possibilities her completely awes me. 

An interactive, conversational system that makes currently expensive and highly manual therapy much more accessible. Something that talks you through a cortisol spike, anxiety attack, panic attack. 

2solivagantry
Adapts to the user's preferred method of coping/comfort and blends that with 'scientific' methods?

I tweeted an idea earlier: A tool that explains in words you understand what the other person really meant. maybe has settings for "gently nudge me if i'm unfairly assuming negative intent"

I generally agree with this but think the alternative goal of "make forecasting easier" is just as good, might actually make aggregate forecasts more accurate in the long run, and may require things that seemingly undermine the virtue of precision.

More concretely, if an underdefined question makes it easier for people to share whatever beliefs they already have, then facilitates rich conversation among those people, that's better than if a highly specific question prevents people from making a prediction at all. At least as much, if not more, of the value ... (read more)

You want to change "Your Distribution" to something like "Daniel's 2020 distribution"? 

2Daniel Kokotajlo
Could you help me by writing some instructions for how to use Elicit, to be put in my question on the topic? (See discussion with Ben above)
2Daniel Kokotajlo
Yes, but I don't know how.

Yea this was a lot more obvious to me when I plotted visually: https://elicit.ought.org/builder/om4oCj7jm

(NB: I work on Elicit and it's still a WIP tool) 

2ESRogs
Is the "Your distribution" one Alex's updated estimates, or is it your (jungofthewon's) distribution?

Ahhhh this is so nice! I suspect a substantial fraction of people would revise their timelines after seeing what they look like visually. I think encouraging people to plot out their timelines is probably a pretty cost-effective intervention.