LESSWRONG
LW

All of Omega.'s Comments + Replies

Critiques of prominent AI safety labs: Conjecture
Omega.2-5

Note that we don't criticize Connor specifically, but rather the lack of a senior technical expert on the team in general (including Connor). Our primary criticisms of Connor don't have to do with his leadership skills (which we don't comment on this at any point in the post).

Reply
3Linch
I'm confused about the disagree votes. Can someone who disagree-voted say which of the following claims they disagreed with: 1. Omega criticized the lack of a senior technical expert on Conjecture's team. 2. Omega's primary criticisms of Connor doesn't have to do with his leadership skills. 3. Omega did not comment on Connorship's leadership skills at any point in the post.
Critiques of prominent AI safety labs: Conjecture
Omega.-3-5

(cross-posted from EAF) 

appreciate you sharing your impression of the post. It’s definitely valuable for us to understand how the post was received, and we’ll be reflecting on it for future write-ups.

1) We agree it's worth taking into account aspects of an organization other than their output. Part of our skepticism towards Conjecture – and we should have made this more explicit in our original post (and will be updating it) – is the limited research track record of their staff, including their leadership. By contrast, even if we accept for the sake o... (read more)

Reply
Critiques of prominent AI safety labs: Conjecture
Omega.1-4

(cross-posted from the EA Forum)

Regarding your specific concerns about our recommendations: 

1) We address this point in our response to Marius (5th paragraph)  

2) As we note in the relevant section: “We think there is a reasonable risk that Connor and Conjecture’s outreach to policymakers and media is alarmist and may decrease the credibility of x-risk.” This kind of relationship-building is unilateralist when it can decrease goodwill amongst policymakers.

3) To be clear, we do not expect Conjecture to have the same level of “organizatio... (read more)

Reply
8Garrett Baker
I responded to a very similar comment of yours on the EA Forum. To respond to the new content, I don't know if changing the board of conjecture once a certain valuation threshold is crossed would make the organization more robust (now that I think of it, I don't even really know what you mean by strong or robust here. Depending on what you mean, I can see myself disagreeing about whether that even tracks positive qualities about a corporation). You should justify claims like those, and at least include them in the original post. Is it sketchy they don't have this?
Critiques of prominent AI safety labs: Conjecture
Omega.22

Hi Erik, thanks for your points, we meant to say "at the same level of expertise as alignment leaders and researchers other organizations such as...". This was a typo on our part. 

Reply
Critiques of prominent AI safety labs: Conjecture
Omega.12-3

(crossposted from the EA Forum)

We appreciate your detailed reply outlining your concerns with the post. 

Our understanding is that your key concern is that we are judging Conjecture based on their current output, whereas since they are pursuing a hits-based strategy we should expect in the median case for them to not have impressive output. In general, we are excited by hits-based approaches, but we echo Rohin's point: how are we meant to evaluate organizations if not by their output? It seems healthy to give promising researchers sufficient ... (read more)

Reply2
9Marius Hobbhahn
(cross-posted from EAG) Meta: Thanks for taking the time to respond. I think your questions are in good faith and address my concerns, I do not understand why the comment is downvoted so much by other people.  1. Obviously output is a relevant factor to judge an organization among others. However, especially in hits-based approaches, the ultimate thing we want to judge is the process that generates the outputs to make an estimate about the chance of finding a hit. For example, a cynic might say "what has ARC-theory achieve so far? They wrote some nice framings of the problem, e.g. with ELK and heuristic arguments, but what have they ACtUaLLy achieved?" To which my answer would be, I believe in them because I think the process that they are following makes sense and there is a chance that they would find a really big-if-true result in the future. In the limit, process and results converge but especially early on they might diverge. And I personally think that Conjecture did respond reasonably to their early results by iterating faster and looking for hits.  2. I actually think their output is better than you make it look. The entire simulators framing made a huge difference for lots of people and writing up things that are already "known" among a handful of LLM experts is still an important contribution, though I would argue most LLM experts did not think about the details as much as Janus did. I also think that their preliminary research outputs are pretty valuable. The stuff on SVDs and sparse coding actually influenced a number of independent researchers I know (so much that they changed their research direction to that) and I thus think it was a valuable contribution. I'd still say it was less influential than e.g. toy models of superposition or causal scrubbing but neither of these were done by like 3 people in two weeks.  3. (copied from response to Rohin): Of course, VCs are interested in making money. However, especially if they are angel investors instead
Critiques of prominent AI safety labs: Conjecture
Omega.3-10

Thanks for commenting and sharing your reactions Mishka. 

Some quick notes on what you've shared:

Although one has to note that their https://www.conjecture.dev/a-standing-offer-for-public-discussions-on-ai/ is returning a 404 at the moment. Is that offer still standing?

In their response to us they told us this offer was still standing.

A lot of upvotes on such a post without substantial comments seems... unfair?

As of the time of your comment, we believe there were about 8 votes and 30 karma and the post had been up a few hours. We are not sure what voti... (read more)

Reply
Critiques of prominent AI safety labs: Conjecture
Omega.*10-22

Hi TurnTrout, thanks for asking this question. We're happy to clarify:

  1. 'experts': We do not consider Conjecture at the same level of expertise as [edit] alignment leaders and researchers at other organizations such as Redwood, ARC, researchers at academic labs like CHAI, and the alignment teams at Anthropic, OpenAI and DeepMind. This is primarily because we believe their research quality is low.
  2. 'with stature in the AIS community': Based on our impression (from conversations with many senior TAIS researchers at a range of organizations, including a handful w
... (read more)
Reply
Erik Jenner6242

We do not consider Conjecture at the same level of expertise as other organizations such as Redwood, ARC, researchers at academic labs like CHAI, and the alignment teams at Anthropic, OpenAI and DeepMind. This is primarily because we believe their research quality is low.

This isn't quite the right thing to look at IMO. In the context of talking to governments, an "AI safety expert" should have thought deeply about the problem, have intelligent things to say about it, know the range of opinions in the AI safety community, have a good understanding of AI mor... (read more)

Reply
Omega.'s Shortform
Omega.10

Quick updates: 

  • Our next critique (on Conjecture) will be published in 10 days. 
  • The critqiue after that will be on Anthropic. If you'd like to be a reviewer, or have critiques you'd like to share, please message us or email anonymouseaomega@gmail.com.
  • If you'd like to help edit our posts (incl. copy-editing - basic grammar etc, but also tone & structure suggestions and fact-checking/steel-manning), please email us!
    • We'd like to improve the pace of our publishing and think this is an area that external perspectives could help us
      • Make sure our cont
... (read more)
Reply