All of Igor Ivanov's Comments + Replies

And I'm unsure that experts are comparable, to be frank. Due to financial limitations, I used graduate students in BioLP, while the authors of LAB-bench used PhD-level scientists.

I didn't have in mind o1, these exact results seem consistent. Here's an example I had in mind: 

Claude 3.5 Sonnet (old) scores 48% on ProtocolQA, and 7.1% on BioLP-bench
GPT-4o scores 53% on ProtocolQA and 17% on BioLP-bench

Good post. 

The craziest thing for me is that the results of different evals, like ProtocolQA and my BioLP-bench, that suppose to evaluate similar things, are highly inconsistent. For example, two models can have similar scores on ProtocolQA, but one scores twice as much answers on BioLP-bench as the other. It means that we might not measure things we think we measure. And no one knows what causes this difference in the results.

This is an amazing overview of the field.  Even if it won't collect tons of upvotes, it is super important, and saved me many hours. Thank you.

I tried to use the exact quotes while describing things that they sent me because it's easy for me to misrepresent their actions, and I don't want tit to be the case.

Totally agree. But in other cases, when the agent was discouraged against dceiving, it did it too.

Thanks for your feedback. It's always a pleasure to see that my work is helpful for people. I hope you will write articles that are way better than mine!

Thanks for your thoughtful answer. It's interesting how I just describe my observations, and people make conclusions out of it that I didn't think of

Thanks. I got a bit clickbaity in the title.

Thanks for sharing your experience. I wish you to stay strong

The meaninglessness comes from the idea akin to to "why bother with anything if AGI will destroy everything it"

Read Feynman's citation from the beginning. It describes his feelings about atom bomb that are relevant for some people's thoughts about AGI.

Thanks

Your comment is somewhat along the lines of the stoic philosophy.

Hi

In this post you asked to leave the names of therapists familiar with alignment.

I am such a therapist. I live in the UK. That's my website.

I recently wrote a post about my experience as a therapist with clients working on AI safety. It might serve as indirect proof that I really have such clients. 

This is tricky. May it exacerbate your problems?

Anyway. If there's a chance I can helpful for you, let me know.

These problems are not unique to AI safety, but they are present way more often with my clients working on AI safety, than with my other clients.

Thanks. I am not a native English speaker, and I use GPT-4 to help me catch mistakes, but it seems like it's not perfect :)

Thanks for sharing your experience. My experience is that talking with non-AI safety people is similar to talks about global warming. If someone tells me about that, I say that this is an important issue, but I honestly don't invest that much effort to fight against it.

This is my experience, and yours might be different.

I totally agree that it might be good to have such a fire alarm as soon as possible, and looking at how fast people make GPT-4 more and more powerful makes me think that this is only a matter of time.

I believe we need a fire alarm.

People were scared of nuclear weapons since 1945, but no one restricted the arms race until The Cuban Missile Crisis in 1961. 

We know for sure that the crisis really scared both Soviet and US high commands, and the first document to restrict nukes was signed the next year, 1962.

What kind of fire alarm it might be? That is The Question.

I think an important thing to get people convinced of the importance of AI safety is to find proper "Gateway drug" ideas that already bother that person, so they are likely to accept this idea, and through it get interested in AI safety. 

For example, if a person is concerned about the rights of minorities, you might tell them about how we don't know how LLMs work, and this causes bias and discrimination, or how it will increase inequality.

If a person cares about privacy and is afraid of government surveillance, then you might tell them about how AI might make all these problems much worse.

Eh. It's sad if this problem is really so complex.

Thank you. At this point, I feel like I have to stick to some way to align AGI, even if it has not that big chance to succeed, because it looks like there are not that many options.

Thanks for your elaborate response!

But why do you think that this project will take so much time? Why can't it be implemented faster?

Do you have any plans for inter-lab communications based on your evals?

I think, your evals might be a good place for AGI labs to standardize protocols for safety measures. 

I think this Wizard of Oz problem in large part is about being mindful and honest with oneself.

Wishful thinking is somewhat the default state for people. It's hard to be critical to own ideas and wishes. Especially, when things like money or career advancement are at stake.

Thank you! The idea of inter-temporal coordination looks interesting

Thank you

Can you elaborate on your comment? 

It seems so intriguing to me, and I would love to learn more about "Why it's a bad strategy if our AGI timeline is 5 years or less"?

Why do you think that it will not be competitive with other approaches?

For example, it took 10 years to sequence the first human genome.  After nearly 7 years of work, another competitor started an alternative human genome project using completely another technology, and both projects were finished approximately at the same time.

I think, we are entering a black swan and it's hard to predict anything.

I absolutely agree with the conclusion. Everything is moving so fast.

I hope, these advances will cause massive interest in the alignment problem from all sorts of actors, and even if OpenAI are talking about safety (and recently they started talking about it quite often) in a large part because of PR reasons, it still means that they think, society is concerned about the progress which is a good sign.

What are examples of “knowledge of building systems that are broadly beneficial and safe while operating in the human capabilities regime?”

I assume the mentioned systems are institutions like courts, government, corporations, or universities


 

Charlotte thinks that humans and advanced AIs are universal Turing machines, so predicting capabilities is not about whether a capability is present at all, but whether it is feasible in finite time with a low enough error rate.

I have a similar thought. If AI has human-level capabilities, and a part of its job is ... (read more)

Thanks for your view on doomerism and your thoughts on the framing of a hopre

I agree. We have problems with emotional attachment to humans all the time, but humans are more or less predictable, not too powerful, and usually not so great at manipulations

Thank you for your comment and everything you mentioned in it. I am a psychologist entering the field of AI policy-making, and I am starving for content like this

It does, and it causes a lot of problems, so I would prefer to avoid such problems with AIs 

Also, I believe that an advanced AI will be much more capable in terms of deception and manipulation than an average human

I 100% agree with you. 

I am a person entering the field right now, I also know several people in a position similar to mine, and there are just no positions for people like me, even though I think I am very proactive and have valuable experience

Good post, but there is a big disbalance in human-ants relationships. 

If people could communicate with ants, nothing would stop humans to make ants suffer if it made the deal better for humans because of a power disbalance. 

For example, domesticated chickens live in very crowded and stinky conditions, and their average lifespan is a month after which they are killed. Not a particularly good living conditions.

People just care about profitability do it just because they can.

Good post

I have similar thoughts. I believe that at one moment, fears about TAI will spread like a wildfire, and the field will get a giant stream of people, money and policies, and it is hard to feel from today

First, your article is very insightful and well-structured, and totally like it.

But there is one thing that bugs me.

I am a person new to AI alignment field, and recently, I realized (maybe by mistake) that there is very hard to find a long-term financially stable full-time job in AI field-building. 

For me, it basically means that only a tiny amount of people consider AI alignment important enough to pay money to decrease P(doom).  And at the same time, here we are talking about possibility of doom within next 10 or 20 years. For me it is all a bi... (read more)

ChatGPT was recently launched, and it is so powerful, that it made me think that the problem of a misuse of a powerful AI  It's a very powerful tool. No one really knows how to use it, but I am sure, we will soon see it used as a tool for unpleasant things

But I also see more and more of perception of AI as a live entity with agency. People are having conversations with ChatGPT as with a human

I agree that fearmongering is thin ice, and can easily backfire, and it must be done carefully and ethically, but is it worse than the alternative in which people are unaware of AGI-related risks? I don't think that anybody can say with certainty

The reactor meltdown on a Soviet submarine was not posing an existential threat. In the worst case, it would be a little version of Chernobyl. We might compare it to an AI which causes some serious problems, like a stock market crash, but not existential ones. And the movie is not a threat at all.

"The question is how plausible it is to generate situations that are scary enough to be useful, but under enough control to be safe."
That is a great summary of what I wanted to say!

I agree

In my opinion, this methodology will be a great way for a model to learn how to persuade humans and exploit their biases because this way model might learn these biases not just from the data it collected but also fine-tune its understanding by testing its own hypotheses