All of Cole Wyeth's Comments + Replies

How exactly do you expect “evaluating ai consciousness 101” to look? That is not a well-defined or understood thing anyone can evaluate. There are however a vast number of capability specific evaluations from competent groups like METR.

The problem with your view is that they don’t have the ability to continue learning for long after being “born.” That’s just not how the architecture works. Learning in context is still very limited and continual learning is an open problem. 

Also, “consciousness” is not actually a very agreed-upon term. What do you mean? Qualia and a first person experience? I believe it’s almost a majority view here to take seriously the possibility that LLMs have some form of qualia, though it’s really hard to tell for sure. We don’t really have tests for that at al... (read more)

How do you recommend studying recent history?

I don’t know what question you think people here aren’t taking seriously.

A massive amount of ink has been spilled about whether current LLMs are AGI. 

I tried the string reversal thing with chatgpt and it was inconsistently successful. I’m not surprised that there is SOME model that solves it (what specifically did you change?), it’s obviously not a very difficult task. Anyway, if you investigate in a similar direction but spend more than five minutes, you’ll probably find similar string manipulation tasks that fail in whatever system you choose. 

You did name it “AI 2027” ;)

Same (though frankly nothing I've done has had the same level of impact). 

This is the curse of playing with very high and non-local stakes. 

Thanks for being specific. 

You claimed that "no one here can provide a test that would differentiate it from a human six year old". This is not what you actually observed. Perhaps no one HAS provided such a test yet, but that may be because you haven't given people much motivation to engage - for instance you also didn't post any convincing evidence that it is recursively self-improving despite implying this. In fact, as far as I can tell no one has bothered to provide ANY examples of tests that six year olds can pass? The tests I provided you dismiss... (read more)

You said “every text-based test of intelligence we have.” If you meant that to be qualified by “that a six your old could pass” as you did in some other places, then perhaps it’s true. But I don’t know - maybe six year olds are only AGI because they can grow into adults! Something trapped at six your old level may not be.

…and for what it’s worth, I have solved some open math problems, including semimeasure extension and integration problems posed by Marcus Hutter in his latest book and some modest final steps in fully resolving Kalai and Lehrer’s grain of ... (read more)

Eliezer’s form of moral realism about good (as a real but particular shared concept of value which is not universally compelling to minds) seems to imply that most of us prefer to be at least a little bit evil, and can’t necessarily be persuaded otherwise through reason.

Seems right.

And Nietzsche would probably argue the two impulses towards good and evil aren't really opposites anyway. 

There is little connection between a language model claiming to be conscious and actually being conscious, in the sense that this provides very weak evidence. The training text includes extensive discussion of consciousness, which is reason enough to expect this behavior.

Okay, but... even if it's a normal capability, shouldn't we be talking about that? "AI can fluidly mimic consciousness and passes every text-based test of intelligence we have" seems like a pretty huge milestone to me.

We ARE talking about it. I thought you were keeping up with the conversa... (read more)

Try a few different prompts with a vaguely similar flavor. I am guessing the LLM will always say it’s conscious. This part is pretty standard. As to whether it is recursively self-improving: well, is its ability to solve problems actually going up? For instance if it doesn’t make progress on ARC AGI  I’m not worried. 

It’s very unlikely that the prompt you have chosen is actually eliciting abilities far outside of the norm, and therefore sharing information about is very unlikely to be dangerous.

You are probably in the same position as nearly everyone else, passively watching capabilities emerge while hallucinating a sense of control.

Whether you use AIXI or IBP, a continual learning algorithm must contend with indexical uncertainty, which means it must contend with indexical complexity in some fashion. 

As far as I understand, IBP tries to evaluate hypotheses according to the complexity of the laws of physics, not the bridge transformation (or indexical) information. But that cannot allow it to overcome the fundamental limitations of the first-person perspective faced by an online learner as proved by Shane Legg. That’s a fact about the difficulty of the problem, not a feature (or ... (read more)

I'm not sure we disagree then.

My take on how recursion theory failed to be relevant for today's AI is that it turned out that what a machine could do if unconstrained basically didn't matter at all, and in particular it basically didn't matter what limits an ideal machine could do, because once we actually impose constraints that force computation to use very limited amounts of resources, we get a non-trivial theory and importantly all of the difficulty of explaining how humans do stuff lies here.

That's partially true (computational complexity is now much more active than recursion the... (read more)

That looks like (minor) good news… appears more consistent with the slower trendline before reasoning models. Is Claude 4 Opus using a comparable amount of inference-time compute as o3? 

I believe I predicted that models would fall behind even the slower exponential trendline (before inference time scaling) - before reaching 8-16 hour tasks. So far that hasn’t happened, but obviously it hasn’t resolved either. 

Thanks, but no. The post I had in mind was an explanation of a particular person's totalizing meta-worldview, which had to do with evolutionary psychology. I remember recognizing the username - also I have that apparently common form of synesthesia where letters seem to have colors and I vaguely remember the color of it (@lc? @lsusr?) but not what is was. 

I’m not sure about the rest of the arguments in the post, but it’s worth flagging that a kg to kg comparison of honey to chicken is kind of inappropriate. Essentially no one is eating a comparable amount of honey as a typical carnivore eats chicken (I didn’t, like, try to calculate this, but it seems obviously right).

Welcome to lesswrong!

I’m glad you’ve decided to join the conversation here. 

A problem with this argument is that it doesn’t prove we should pause AI, only that we should avoid deploying AI in high impact (e.g. military) applications. Insofar as LLMs can’t follow rules, the argument seems to indicate that we should continue to develop the technology until it can.

Personally, I’m concerned about the type of AI system which can follow rules, but is not intrinsically motivated to follow our moral rules. Whether LLMs will reach that threshold is not clear t... (read more)

The report is partially optimistic but the results seem unambiguously bearish.

Like, yeah, maybe some of these problems could be solved with scaffolding - but the first round of scaffolding failed, and if you're going to spend a lot of time iterating on scaffolding, you could probably instead write a decent bot that doesn't use Claude in that time. And then you wouldn't be vulnerable to bizarre hallucinations, which seem like an unacceptable risk. 

https://www.lesswrong.com/posts/KXujJjnmP85u8eM6B/policy-for-llm-writing-on-lesswrong

Agree about phones (in fact I am seriously considering switching to a flip phone and using my iphone only for things like navigation). 

Not so sure about LLMs. I had your attitude initially, and I still consider them an incredibly dangerous mental augmentation. But I do think that conservatively throwing a question at them to find searchable keywords is helpful, if you maintain the attitude that they are actively trying to take over your brain and therefore remain vigilant.  

Short fiction on lesswrong isn’t uncommon 

That’s why I specified “close on a log scale.” Evolution may be very inefficient, but it also has access to MUCH more data than a single lifetime.

Yes, we should put some weight on both perspectives. What I’m worried about here is this trend where everyone seems to expect AGI in a decade or so even if the current wave of progress fizzles - I think that is a cached belief. We should be prepared to update. 

The hedonic treadmill exists because minds are built to climb utility gradients - absolute utility levels are not even uniquely defined, so as long as your preferences are time-consistent you can just renormalize before maximizing the expected utility of your next decision. 

I find this vaguely comforting. It’s basically a decision-theoretic and psychological justification for stoicism. 

(must have read this somewhere in the sequences?)

I think self-reflection in bounded reasoners justifies some level of “regret,” “guilt,” “shame,” etc., but the basic reasoning above should hold to first order, and these should all be treated as corrections and for that reason should not get out of hand. 

Seems plausible, but not compelling.

Why one human lifetime and not somewhere closer to evolutionary time on log scale?

But… the success of LLMs is the only reason people have super short timelines! That’s why we’re all worried about them, and in particular if they can soon invent a better paradigm - which, yes, may be more efficient and dangerous than LLMs, but presumably requires them to pass human researcher level FIRST, maybe signficantly.

If you don’t believe LLMs will scale to AGI, I see no compelling reason to expect another paradigm which is much better to be discovered in the next 5 or 10 years. Neuroscience is a pretty old field! They haven’t figured out rhe brain’... (read more)

I’m surprised you think that the brain’s algorithm is SO simple that it must be discovered soon and ~all at once. This seems unlikely to me (reality has a surprising amount of detail). I think you may be underestimating the complexity because:

Though I don’t know enough biochem to say for sure, I’m guessing many “bits of the algorithm” are external to the genes (epigenetic?). Specifically, I don’t just mean data like education materials that is learned, I mean that actual pieces of the algorithm are probably constructed “in motion” by other machinery in the... (read more)

But maybe you mean that people like Alice would be quite rare? Could be so.

Yes

Interesting idea, but it’s an infinite money hack in sort of the same way that “find a rich person who hates money” is an infinite money hack.

I think the litany is about belief, not speech.

Personally, being shot for speaking an inconvenient truth sounds like an appropriate death for me, and I’m happy to endorse the stronger version that you seem to be arguing against. 

AI-specific pronouns would actually be kind of helpful. “They” and “It” are both frequently confusing. “He” and “she” feel anthropomorphic and fake. 

1: Not true, I hear about exponential time algorithms! People work on all sorts of problems only known to have exponential time algorithms. 

2: Yes, but the reason k only shows up as something we would interpret as a parameter and not as a result of the computational complexity of an algorithm invented for a natural problem is perhaps because of my original point - we can only invent the algorithm if the problem has structure that suggests the algorithm, in which case the algorithm is collapsible and k can be separated out as an additional input for a simpler algorithm.  

I wonder if the reason that polynomial time algorithms tend to be somewhat practical (not runtime n^100) is just that we aren’t smart enough to invent really necessarily complicated polynomial time algorithms.

Like, the obvious way to get n^100 is to nest 100 for loops. A problem which can only be solved in polynomial time by nesting 100 for loops (presumably doing logically distinct things that cannot be collapsed!) is a problem that I am not going to solve in polynomial time… 

Okay, this does raise the question of why the “if anyone builds it, everyone dies” frontage? 

I think that the difference in how we view this is because to me, lesswrong is a community / intellectual project. To you it’s a website.

The website may or may not be neutral, but it’s obvious that the project is not neutral. 

Here are some examples of neutral common spaces:

Libraries 

Facebook (usually)

Community center event spaces

Here are some examples of spaces which are not neutral or common:

The alignment forum

The NYT (or essentially any newspaper’s) opinions column

The EA forum 

Lesswrong

This seems straightforwardly true to me. I’m not sure what tribe it’s supposed to be a flag for. 

lesswrong is not a neutral common space.

I seem to have had essentially this exact conversation in a different comment thread on this post with the OP. 

I am saying that there may be no point to considering moral alignment as target. 

We need to solve single to single alignment. At that point, whoever a given AGI is aligned to decides its values. If one of your values resembles moral alignment, great - you want an AGI aligned to you just like many others. Better buy a supercluster ;)

(Just kidding, we don't know how to solve single to single alignment so please don't buy a supercluster)

Aren't you making this judgement based on your own values? In that case, it seems that an AGI aligned to you specifically is at least as good as an AGI aligned to all sentient life.

Of course, there is a substantial difference between the values of an individual human and human values.

I think this is technically much harder than the single to single alignment problem. I am highly pessimistic that we can get such values into any AGI system without first aligning it to a human(s) who then asks it to self-modify into valuing all sentient life.

Optimality is about winning. Rationality is about optimality.  

It seems that this model requires a lot of argumentation that is absent from post and only implicit in your comment. Why should I imagine that AGI would have that ability? Are there any examples of very smart humans who simultaneously acquire multiple seemingly magical abilities? If so, and if AGI scales well past human level, it would certainly be quite dangerous. But that seems to assume most of the conclusion.

Explicitly, in the current paradigm this is mostly about training data, though I suppose that with sufficient integration that data will eventuall... (read more)

I intentionally avoided as much as possible the implication that intelligence is "only" raw IQ. But if intelligence is not on some kind of real-valued scale, what does any part of this post mean?

The post is an intuition pump for the idea that intelligence enables capabilities that look like "magic." 

It seems to me that all it really demonstrates is that some people have capabilities that look like magic, within domains where they are highly specialized to succeed. The only example that seems particularly dangerous (El Chapo) does not seem convincingly connected to intelligence. I am also not sure what the chess example is supposed to prove - we already have chess engines that can defeat multiple people at once blindfolded, including (presumab... (read more)

What makes you think that those people were able to do those things because of high levels of intelligence? It seems to me that in most cases, the reported feat is probably driven by some capability / context combination that stretches the definition of intelligence to varying degrees. For instance I would guess that El Chapo pulled that off because he already had a lot of connections and money when he got to prison. The other examples seem to demonstrate that it is possible for a person to develop impressive capabilities in a restricted domain given enough experience. 

I’ve heard it stands for “Omni”

To what extent would a proof about AIXI’s behavior be normative advice?

Though AIXI itself is not computable, we can prove some properties of the agent - unfortunately, there are fairly few examples because of the “bad universal priors” barrier discovered by Jan Leike. In the sequential case we only know things like e.g. it will not indefinitely keep trying an action that yields minimal reward, though we can say more when the horizon is 1 (which reduces to the predictive case in a sense). And there are lots of interesting results about the behavior of Solom... (read more)

This is a generalization problem which I expect to be solved before any system achieves dangerous capabilities. It’s already been discussed at some length in these comments with Steven Byrnes. 

There are pretty strong reasons to expect that neither direction (conditioning or switching UTM) perfectly simulates the other. I think one of the two directions is known to be impossible - that conditioning cannot be replaced by switching UTM. 

I don’t see any qualitative reason that it should not count, even if it’s not terribly impressive.