All of Judd Rosenblatt's Comments + Replies

Cool point, yes, seems right!

Reading this post, my immediate hunch is that the decline in sentence lengths has a lot to do with the historical role of Latin grammar and how deeply it influenced educated English writers. Latin inherently facilitates longer, complex sentences due to its use of grammatical inflections, declensions, and verb conjugations, significantly reducing reliance on prepositions and conjunctions. This syntactic flexibility allowed authors to naturally craft extensive yet smooth-flowing sentences. Latin's liberating lack of fixed word order and its fun little rhetor... (read more)

Yes, I am hopeful we have enough time before superintelligent AI systems are created to implement effective alignment approaches. I don't know if that is possible or not, but I think it is worth trying.

Given uncertainty about timelines and currently accelerating capabilities, it would be preferable to live in a world where we are making sure alignment advances more than otherwise. 

I think this is precisely the reason that you’d want to make sure the agent is engineered such that its utility function includes the utility of other agents—ie, so that the ‘alignment goals’ are its goals rather than ‘goals other than [its] own.’ We suspect that this exact sort of architecture could actually exhibit a negative alignment tax insofar as many other critical social competencies may require this as a foundation.

I think this risks getting into a definitions dispute about what concept the words ‘alignment tax’ should point at. Even if one grants the point about resource allocation being inherently zero-sum, our whole claim here is that some alignment techniques might indeed be the most cost-effective way to improve certain capabilities and that these techniques seem worth pursuing for that very reason.

Thanks for this comment! Definitely take your point that it may be too simplistic to classify entire techniques as exhibiting a negative alignment tax when tweaking the implementation of that technique slightly could feasibly produce misaligned behavior. It does still seem like there might be a relevant distinction between:

1. Techniques that can be applied to improve either alignment or capabilities, depending on how they’re implemented. Your example of ‘System 2 alignment’ would fall into this category, as would any other method with “the potential to be emp

Interesting relevant finding from the alignment researcher + EA survey we ran:

We also find in both datasets—but most dramatically in the EA community sample, plotted below—that respondents vastly overestimate (≈2.5x) how much high intelligence is actually valued, and underestimate other cognitive features like having strong work ethics, abilities to collaborate, and people skills. One potentially clear interpretation of this finding is that EAs/alignment researchers actually believe that high intelligence is necessary but not sufficient for being impactful

Will do!

Interesting. I wouldn't totally rule number 1 out though.  Depending on how fast things go, the average time to successful IPO may decrease substantially.

Yes, excellent point, and thanks for the callout. 

Note though that a fundamental part of this is that we at AE Studio do intend eventually to incubate as part of our skunkworks program alignment-driven startups. 

We've seen that we can take excellent people, have them grow on client projects for some amount of time, get better at stuff they don't even realize they need to get better at in a very high-accountability way, and then be well positioned to found startups we incubate internally. 

We've not turned attention to internally-incubated sta... (read more)

We plan to announce further details in a later post.

This is a great point. I also notice that a decent number of people's risk models change frequently with various news, and that's not ideal either, as it makes them less likely to stick with a particular approach that depends on some risk model. In an ideal world we'd have enough people pursuing enough approaches with most possible risk models that it's make little sense for anyone to consider switching. Maybe the best we can approximate now is to discuss this less.

That would be great! And it's exactly the sort of thing we've dreamed about building at AE since the start.

Incidentally, I've practiced something (inferior) like this with my wife in the past and we've gotten good at speaking simultaneously and actually understanding multiple threads at the same time (though it seems to break down if one of the threads is particularly complex).

It seems like an MVP hyperphone could potentially just be a software project/not explicitly require BCI (though certainly would be enhanced with it). We would definitely consider bui... (read more)

And another interesting one from the summit:

“There was almost no discussion around agents—all gen AI & model scaling concerns.

It’s perhaps because agent capabilities are mediocre today and thus hard to imagine, similar to how regulators couldn’t imagine GPT-3’s implications until ChatGPT.” - https://x.com/kanjun/status/1720502618169208994?s=46&t=D5sNUZS8uOg4FTcneuxVIg

Right now it seems to me that one of the highest impact things not likely to be done by default is substantially increased funding for AI safety.

I got https://www.pinkshoggoth.com/ inspired by Pink Shoggoths: What does alignment look like in practice?

Right now it's hosting a side project (that may wind up being replaced by new ChatGPT features). Feel free to DM me if you have a better use for it though!