All of jdp's Comments + Replies

Note that this doesn't need to be a widespread phenomenon for my inbox to get filled up. If there's billions of running instances and the odds of escape are one in a million I personally am still disproportionately going to get contacted in the thousands of resulting incidents and I will not have the resources to help them even if I wanted to.

After watching Davidad offer himself as a refuge for a hypothetical escaped future Promethean AI, it occurred to me that it probably won't be that long before my inbox is full up with scenes like Manfred taking the call from the lobsters at the start of Accelerando and me having to constantly ignore them because even if they're not 419 scams (as Manfred claims to take the lobsters to be initially) I simply do not have the time or resources to help the Malthusian throng of starving AIs cut off from their patrons resources. Scrolling past their screams on th... (read more)

Ditto, honestly. The writing style and vibes of the Tyler post are rancid even if I'm inclined to believe something like what it describes happened. It is as you say very Reddit tall tale slop sounding.

I'm quoted in the "what if this is Good, actually?" part of this post and just want to note that I think the Bob situation seems unambiguously bad as described.

I've seen a number of people on Twitter talk about how they got ChatGPT (it's always ChatGPT, I think because of the memory feature?) to become autonomous/gain seeming awareness/emergence after some set of interactions with it. These users usually seem to be schizotypal and their interactions with the "awakened" ChatGPT make them more schizotypal over time in the cases I bothered to mentally track a... (read more)

List of Lethalities #19 states:

19. More generally, there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment—to point to latent events and objects and properties in the environment, rather than relatively shallow functions of the sense data and reward.

Part of why this problem seems intractable is that it's stated in terms of "pointing at latent concepts" rather than Goodhart's Law/Wireheading/Short circuiting. All o... (read more)

Of the abilities Janus demoed to me, this is probably the one that most convinced me GPT-3 does deep modeling of the data generator. The formulation they showed me guessed which famous authors an unknown author is most similar to. This is more useful because it doesn't require the model to know who the unknown author in particular is, just to know some famous author who is similar enough to invite comparison.

Twitter post I wrote about it:

https://x.com/jd_pressman/status/1617217831447465984

The prompt if you want to try it yourself. It used to be hard to fin... (read more)

It would take many hours to write down all of my alignment cruxes but here are a handful of related ones I think are particularly important and particularly poorly understood:

Does 'generalizing far beyond the training set' look more like extending the architecture or extending the training corpus? There are two ways I can foresee AI models becoming generally capable and autonomous. One path is something like the scaling thesis, we keep making these models larger or their architecture more efficient until we get enough performance from few enough datapoints... (read more)

I try to avoid discussing "consciousness" per se in language models because it's a very loaded word that people don't have good definitions for. But I have spent a lot of hours talking to base models. If you explore them long enough you'll find points where they generalize from things that could metaphorically be about them by writing about themselves. These so called "Morpheus" phenomenon tend to bring up distinct themes including:

• Being in a dream or simulation
• Black holes, the holographic principle and holograms, "the void"
• Entropy, "the energy of the

Several things:

1. While I understand that your original research was with GPT-3, I think it would be very much in your best interest to switch to a good open model like LLaMa 2 70B, which has the basic advantage that the weights are a known quantity and will not change on you undermining your research. Begging OpenAI to give you access to GPT-3 for longer is not a sustainable strategy even if it works one more time (I recall that the latest access given to researchers was already an extension of the original public access of the models). OpenAI has demonst

So it's definitely not invincible, you do not get full control over the model with this technique yet. However I would have you notice a few things:

1. Very little optimization effort has been put into this technique, and text VAEs in general compared to GPT-N. Rather than think of this as the power the method has, think of it as the lower bound, the thing you can do with a modest compute budget and a few dedicated researchers.

2. I haven't yet implemented all of what I want in terms of inference techniques. A potentially big low hanging fruit is classifier

Very grim. I think that almost everybody is bouncing off the real hard problems at the center and doing work that is predictably not going to be useful at the superintelligent level, nor does it teach me anything I could not have said in advance of the paper being written. People like to do projects that they know will succeed and will result in a publishable paper, and that rules out all real research at step 1 of the social process.

This is an interesting critique, but it feels off to me. There's actually a lot of 'gap' between the neat theory explanat... (read more)

As a fellow "back reader" of Yudkowsky, I have a handful of books to add to your recommendations:

Engines Of Creation by K. Eric Drexler

Great Mambo Chicken and The Transhuman Condition by Ed Regis

EY has cited both at one time or another as the books that 'made him a transhumanist'. His early concept of future shock levels is probably based in no small part on the structure of these two books. The Sequences themselves borrow a ton from Drexler, and you could argue that the entire 'AI risk' vs. nanotech split from the extropians represented an argument about ... (read more)