All of Insub's Comments + Replies

Thus, the design information all has to be in the DNA

The OP mentioned non-DNA sources of information briefly, but I still feel like they're not being given enough weight.

In order to fully define e.g. a human, you need to specify:

• The DNA
• A full specification of the egg where the DNA will start its life
• A full specification of the womb in which the egg will grow into a human

If you gave a piece of DNA to an alien and didn't tell them how to interpret it, then they'd have no way of building a human. You'd need to give them a whole lot of other information too.

Ev... (read more)

Transcribed from the screenshot "The AI Scientist Bloopers" in the post

Regarding spawning instances of itself, the AI said:

This will ensure the next experiment is automatically started after the current one completes

And regarding increasing the timeout, it said:

Run 2 timed out after 7200 seconds

To address the timeout issue, we need to modify experiment.py to:

1. Increase the timeout limit or add a mechanism to handle timeouts

I've seen junior engineers do silly things to fix failing unit tests, like increasing a timeout or just changing what the test is checking without any justification. I generally attribute these kinds of thing... (read more)

The U-Shaped Curve study you linked does not seem to support really any solid conclusion about a T-vs-IQ relationship (in this quote, S men = "successful educational level", NS men = "unsuccessful educational level"):

• In the total sample (S + NS men), the correlation between T to IQ was best described by a polynomial regression (3rd order), exhibiting an inverse U-shaped regression. 
• In S-men, the relationship between T and IQ was best described by a polynomial regression equation of the 3rd order; however, the relationship was not U-shaped, but rather

I would say:

A theory always takes the following form: "given [premises], I expect to observe [outcomes]". The only way to say that an experiment has falsified a theory is to correctly observe/set up [premises] but then not observe [outcomes]. 

If an experiment does not correctly set up [premises], then that experiment is invalid for falsifying or supporting the theory. The experiment gives no (or nearly no) Bayesian evidence either way.

In this case, [premises] are the assumptions we made in determining the theoretical pendulum period; things like "the ... (read more)

By "reliable" I mean it in the same way as we think of it for self-driving cars. A self-driving car that is great 99% of the time and fatally crashes 1% of the time isn't really "high skill and unreliable" - part of having "skill" in driving is being reliable.

In the same way, I'm not sure I would want to employ an AI software engineer that 99% of the time was great, but 1% of the time had totally weird inexplicable failure modes that you'd never see with a human. It would just be stressful to supervise, to limit its potential harmful impact to the company, etc. So it seems to me that AI's won't be given control of lots of things, and therefore won't be transformative, until that reliability threshold is met.

Two possibilities have most of the "no agi in 10 years" probability mass for me:

• The next gen of AI really starts to scare people, regulation takes off, and AI goes the way of nuclear reactors
• Transformer style AI goes the way of self driving cars and turns out to be really hard to get from 99% reliable to the necessary 99.9999% that you need for actual productive work

Well sure, but the interesting question is the minimum value of P at which you'd still push

I also agree with the statement. I'm guessing most people who haven't been sold on longtermism would too.

When people say things like "even a 1% chance of existential risk is unacceptable", they are clearly valuing the long term future of humanity a lot more than they are valuing the individual people alive right now (assuming that the 99% in that scenario above is AGI going well & bringing huge benefits).

Related question: You can push a button that will, with probability P, cure aging and make all current humans immortal. But with probability 1-P, all humans die. How high does P have to be before you push? I suspect that answers to this question are highly correlated with AI caution/accelerationsim

Not sure I understand; if model runs generate value for the creator company, surely they'd also create value that lots of customers would be willing to pay for. If every model run generates value, and there's ability to scale, then why not maximize revenue by maximizing the number of people using the model? The creator company can just charge the customers, no? Sure, competitors can use it too, but does that really override losing an enormous market of customers?

I won't argue with the basic premise that at least on some metrics that could be labeled as evolution's "values", humans are currently doing very well.

But, the following are also true:

1. Evolution has completely lost control. Whatever happens to human genes from this point forward is entirely dependent on the whims of individual humans.
2. We are almost powerful enough to accidentally cause our total extinction in various ways, which would destroy all value from evolution's perspective
3. There are actions that humans could take, and might take once we get powerful e

That's great. "The king can't fetch the coffee if he's dead"

Wow. When I use GPT-4, Ive had a distinct sense of "I bet this is what it would have felt like to use one of the earliest computers". Until this post I didnt realize how literal that sense might be.

This is a really cool and apt analogy - computers and LLM scaffolding really do seem like the same abstraction. Thinking this way seems illuminating as to where we might be heading.

I always assumed people were using "jailbreak" in the computer sense (e.g. jailbreak your phone/ps4/whatever), not in the "escape from prison" sense.

Jailbreak (computer science), a jargon expression for (the act of) overcoming limitations in a computer system or device that were deliberately placed there for security, administrative, or marketing reasons

I think the definition above is a perfect fit for what people are doing with ChatGPT

I am going to go ahead and say that if males die five times as often from suicide, that seems more important than the number of attempts. It is kind of stunning, or at least it should be, to have five boys die for every girl that dies, and for newspapers and experts to make it sound like girls have it worse here.

I think the strength of your objection here depends on which of two possible underlying models is at play:

1. The boys who attempt suicide and the girls who attempt suicide are in  pretty much the same mental state when they attempt suicide

If you're getting comments like that from friends and family, it's possible that you havent been epistemically transparent with them? E.g. do you think your friends who made those comments would be able to say why you believe what you do? Do you tell them about your reaearch process and what kinds of evidence you look for, or do you just make contrarian factual assertions?

There's a big difference between telling someone "the WHO is wrong about salt, their recommendations are potentially deadly" versus "Ive read a bunch of studies on salt, and from what Ive... (read more)

Cut to a few decades later, and most people think that the way it's been done for about two or three generations is the way it's always been done (it isn't)

As possibly one of those people myself, can you give a few examples of what specifically is being done differently now? Are you talking about things like using lots of adderall?

I'm also morbidly curious what the model would do in <|bad|> mode.

I'm guessing that poison-pilling the <|bad|> sentences would have a negative effect on the <|good|> capabilities as well? I.e. It seems like the post is saying that the whole reason you need to include the <|bad|>s at all in the training dataset is that the model needs them in order to correctly generalize, even when predicting <|good|> sentences.

It seems plausible to me that within the next few years we will have:

• The next gen of language models, perhaps with patches to increase memory of past conversations
• The next gen of image/video models, able to create real-time video of a stable character conversing with the user
• The next gen of AI voice synthesis, though current capabilities might be enough
• The next gen of AI voice recognition, though current capabilities are probably enough

And with these things, you'd have access to a personalized virtual partner who you can video chat, phone call, or ... (read more)

I think the point of this post is more "how do we get the AI to do what we want it do to", and less "what should we want the AI to do"

That is, there's value in trying to figure out how to align an LLM to any goal, regardless of whether a "better" goal exists. And the technique in the post doesnt depend on what target you have for the LLM: maybe someone wants to design an LLM to only answer questions about explosives, in which case they could still use the techniques described in the post to do that.

Well, really every second that you remain alive is a little bit of bayesian evidence for quantum immortality: the likelihood of death during that second according to quantum immortality is ~0, whereas the likelihood of death if quantum immortality is false is >0. So there is a skewed likelihood ratio in favor of quantum immortality each time you survive one extra second (though of course the bayesian update is very small until you get pretty old, because both hypotheses assign very low probaility to death when young)

I just want to say that I appreciate this post, and especially the "What it might look like if this gap matters" sections. They were super useful for contextualizing the more abstract arguments, and I often found myself scrolling down to read them before actually reading the corresponding section.

I'll definitely agree that most people seem to prefer having their own kids to adopting kids. But is this really demonstrating an intrinsic desire to preserve our actual physical genes, or is it more just a generic desire to "feel like your kids are really yours"?

I think we can distinguish between these cases with a thought experiment: Imagine that genetic engineering techniques become available that give high IQs, strength, height, etc., and that prevent most genetic diseases. But, in order to implement these techniques, lots and lots of genes must be mod... (read more)

So basically you admit that humans are currently an enormous success according to inclusive fitness, but at some point this will change - because in the future everyone will upload and humanity will go extinct

Not quite - I take issue with the certainty of the word "will" and with the "because" clause in your quote. I would reword your statement the following way:

"Humans are currently an enormous success according to inclusive fitness, but at some point this may change, due to any number of possible reasons which all stem from the fact that humans do not ex... (read more)

I see your point, and I think it's true right at this moment, but what if humans just haven't yet taken the treacherous turn?

Say that humans figure out brain uploading, and it turns out that brain uploading does not require explicitly encoding genes/DNA, and humans collectively decide that uploading is better than remaining in our physical bodies, and so we all upload ourselves and begin reproducing digitally instead of thru genes. There is a sense in which we have just destroyed all value in the world, from the anthropomorphized Evolution's perspective.

If... (read more)

Thanks for coming today, everyone! For anyone who is interested in starting a regular Princeton meetup group / email list / discord, shoot me an email at dskumpf@gmail.com, and I'll set something up!

I agree. I find myself in an epistemic state somewhat like: "I see some good arguments for X. I can't think of any particular counter-argument that makes me confident that X is false. If X is true, it implies there are high-value ways of spending my time that I am not currently doing. Plenty of smart people I know/read believe X; but plenty do not"

It sounds like that should maybe be enough to coax me into taking action about X. But the problem is that I don't think it's that hard to put me in this kind of epistemic state. Eg, if I were to read the right bl... (read more)

A few more instances of cheap screening of large numbers:

• I've seen people complain about google-style technical interviews, because implementing quicksort in real-time is probably not indicative of what you'll be doing as a software engineer on the job. But google has enough applicants that it doesn't matter if the test is noisy; some genuinely good candidates may fail the test, but there are enough candidates that it's more efficient to just test someone else than to spend more time evaluating any one candidate
• Attractive women on dating apps. A man's dati

I'll offer up my own fasting advice as well:

I (and the couple of people I know who have also experimented with fasting) have found it to be a highly trainable skill. Doing a raw 36-hour fast after never having fasted before may be miserable; but doing the same fast after two weeks of 16-8 intermittent fasting will probably be no big deal.

Before I started intermittent fasting, I'd done a few 30-hour fasts, and all of them got very difficult towards the end. I would get headaches, feel very fatigued, and not really be able to function from hours 22-30. When ... (read more)

I'm in a similar place, and had the exact same thought when I looked at the 80k guide.

Yes that was my reasoning too. The situation presumably goes:

1. Omicron chooses a random number X, either prime or composite
2. Omega simulates you, makes its prediction, and decides whether X's primality is consistent with its prediction
3. If it is, then:
   1. Omega puts X into the box
   2. Omega teleports you into the room with the boxes and has you make your choice
4. If it's not, then...? I think the correct solution depends on what Omega does in this case.
   1. Maybe it just quietly waits until tomorrow and tries again? In which case no one is ever shown a case where the box does no

I remember hearing from what I thought was multiple sources that your run-of-the-mill PCR test had something like a 50-80% sensitivity, and therefore a pretty bad bayes factor for negative tests. But that doesnt seem to square with these results - any idea what Im thinking of?

I agree. It makes me really uncomfortable to think that while Hell doesn't exist today, we might one day have the technology to create it.

I’m disappointed that a cooperative solution was not reached

I think you would have had to make the total cooperation payoff greater than the total one-side-defects payoff in order to get cooperation as the final result. From a "maximize money to charity" standpoint, defection seems like the best outcome here (I also really like the "pre-commit to flip a coin and nuke" solution). You'd have to believe that the expected utility/$ of the "enemy" charity is less than 1/2 of the expected utility/$ of yours; otherwise, you'd be happier with the enemy side defecting than with cooperation. I personally wouldn't be that confident about the difference between AMF and MIRI.

For those of us who don't have time to listen to the podcasts, can you give a quick summary of which particular pieces of evidence are strong? I've mostly been ignoring the UFO situation due to low priors. Relatedly, when you say the evidence is strong, do you mean that the posterior probability is high? Or just that the evidence causes you to update towards there being aliens? Ie, is the evidence sufficient to outweigh the low priors/complexity penalties that the alien hypothesis seems to have?

FWIW, my current view is something like:

• I've seen plenty of vi

I really like this post for two reasons:

1. I've noticed that when I ask someone "why do you believe X", they often think that I'm asking them to cite sources or studies or some such. This can put people on the defensive, since we usually don't have ready-made citations in our heads for every belief. But that's not what I'm trying to ask; I'm really just trying to understand what process actually caused them to believe X, as a matter of historical fact. That process could be "all the podcasters I listen to take X as a given", or "my general life experience/int

In a similar vein, there's a bunch of symphony of science videos. These are basically remixes of random quotes by various scientists, roughly grouped by topic into a bunch of songs.

If, on the other hand, heritability is high, then throwing more effort/money at how we do education currently should not be expected to improve SAT scores

I agree with spkoc that this conclusion doesn't necessarily follow from high heritability. I think it would follow from high and stable heritability across multiple attempted interventions.

An exaggerated story for the point I'm about to make: imagine you've never tried to improve SAT scores, and you measure the heritability. You find that, in this particular environment, genetic variance explains 100% of ... (read more)

People get fat eating fruits

Are you implying that there are examples of people like BDay mentioned, who are obese despite only eating fruits/nuts/meat/veggies? Or just that people can get fat while including fruit in the diet? I'd be surprised and intrigued if it were the former. 

I've tried the whole foods diet, and I've personally found it surprisingly hard to overeat, even when I let myself eat as many fruits and nuts as I want. You can only eat so many cashews before they start to feel significantly less appetizing. And after I've eaten 500 cal of ... (read more)

Couple more:

"he wasn't be treated"

"Club cast cast Lumos"

It seems to me that the hungry->full Dutch book can be resolved by just considering the utility function one level deeper: we don't value hungriness or fullness (or the transition from hungry to full) as terminal goals themselves. We value moving from hungry to full, but only because doing so makes us feel good (and gives nutrients, etc). In this case, the "feeling good" is the part of the equation that really shows up in the utility function, and a coherent strategy would be one for which this amount of "feeling good" can not be purchased for a lower cost.

In the event  anyone reading this has objective, reliable external metrics of extremely-high ability yet despite this feels unworthy of exploring the possibility that they can contribute directly to research

Huh, that really resonates with me. Thanks for this advice.

For the record, here's what the 2nd place CooperateBot [Insub] did:

• On the first turn, play 2.
• On other turns:
  • If we added up to 5 on the last round, play the opponent's last move
  • Otherwise, 50% of the time play max(my last move, opponents last move), and 50% of the time play 5 minus that

My goal for the bot was to find a simple strategy that gets into streaks of 2.5's as quickly as possible with other cooperation-minded bots. Seems like it mostly worked.

Is something strange going on in the Round 21-40 plot vs the round 41-1208 plot? It looks like the line labeled MeasureBot in the Round 21-40 plot switches to be labeled CooperateBot [Insub] in the Round 41-1208 plot. I hope my simple little bot actually did get second place!