All of Jonathan Paulson's Comments + Replies

Answer: it was not given the solution. https://x.com/wtgowers/status/1816839783034843630?s=46&t=UlLg1ou4o7odVYEppVUWoQ

Anyone have a good intuition for why Combinatorics is harder than Algebra, and/or why Algebra is harder than Geometry? (For AIs). Why is it different than for humans?

It’s funny to me that the one part of the problem the AI cannot solve is translating the problem statements to Lean. I guess it’s the only part that the computer has no way to check.

Does anyone know if “translating the problem statements” includes the providing the solution (eg “an even integer” for P1), and the AI just needs to prove the solution correct? Its not clear to me what’s human-written and what’s AI-written, and the solution is part of the “theorem” part which I’d guess is human-written.

For row V, why is SS highlighted but DD is lower?

I think there's a typo; the text refers to "Poltergeist Pummelers" but the input data says "Phantom Pummelers".

I think you are failing to distinguish between "being able to pursue goals" and "having a goal".

Optimization is a useful subroutine, but that doesn't mean it is useful for it to be the top-level loop. I can decide to pursue arbitrary goals for arbitrary amounts of time, but that doesn't mean that my entire life is in service of some single objective.

Similarly, it seems useful for an AI assistant to try and do the things I ask it to, but that doesn't imply it has some kind of larger master plan.

Professors are selected to be good at research not good at teaching. They are also evaluated at being good at research, not at teaching. You are assuming universities primarily care about undergraduate teaching, but that is very wrong.

(I’m not sure why this is the case, but I’m confident that it is)

I think you are underrating the number of high-stakes decisions in the world. A few examples: whether or not to hire someone, the design of some mass-produced item, which job to take, who to marry. There are many more.

These are all cases where making the decision 100x faster is of little value, because it will take a long time to see if the decision was good or not after it is made. And where making a better decision is of high value. (Many of these will also be the hardest tasks for AI to do well on, because there is very little training data about them).

Why do you think so?

Presumably the people playing correspondence chess think that they are adding something, or they would just let the computer play alone. And it’s not a hard thing to check; they can just play against a computer and see. So it would surprise me if they were all wrong about this.

https://www.iccf.com/ allows computer assistance

The idea that all cognitive labor will be automated in the near-future is a very controversial premise, not at all implied by the idea that AI will be useful for tutoring. I think that’s the disconnect here between Altman’s words and your interpretation.

Nate’s view here seems similar to “To do cutting-edge alignment research, you need to do enough self-reflection that you might go crazy”. This seems really wrong to me. (I’m not sure if he means all scientific breakthroughs require this kind of reflection, or if alignment research is special).

I don’t think many top scientists are crazy, especially not in a POUDA way. I don’t think top scientists have done a huge amount of self-reflection/philosophy.

On the other hand, my understanding is that some rationalists have driven themselves crazy via too much self-... (read more)

Tim Cook could not do all the cognitive labor to design an iPhone (indeed, no individual human could). The CEO of Boeing could not fully design a modern plane. Elon Musk could not make a Tesla from scratch. All of these cases violate all of your three bullet points. Practically everything in the modern world is too complicated for any single person to fully understand, and yet it all works fairly well, because successful outsourcing of cognitive labor is routinely successful.

It is true that a random layperson would have a hard time verifying an AI's (or an... (read more)

"This is what it looks like in practice, by default, when someone tries to outsource some cognitive labor which they could not themselves perform."
This proves way too much. People successfully outsource cognitive labor all the time (this describes most white-collar jobs). This is possible because very frequently, it is easier to be confident that work has been done correctly than to actually do the work. You shouldn't just blindly trust an AI that claims to have solved alignment (just like you wouldn't blindly trust a human), but that doesn't mean AIs (or other humans) can't do any useful work.

The link at the top is to the wrong previous scenario

I don't think "they" would (collectively) decide anything, since I don't think it's trivial to cooperate even with a near-copy of yourself. I think they would mostly individually end up working with/for some group of humans, probably either whichever group created them or whichever group they work most closely with.

I agree humans could end up disempowered even if AIs aren't particularly good at coordinating; I just wanted to put some scrutiny on the claim I've seen in a few places that AIs will be particularly good at coordinating.

The key question here is how difficult the objective O is to achieve. If O is "drive a car from point A to point B", then we agree that it is feasible to have AI systems that "strongly increase the chance of O occuring" (which is precisely what we mean by "goal-directedness") without being dangerous. But if O is something that is very difficult to achieve (i.e. all of humanity is currently unable to achieve it), then it seems that any system that does reliably achieve O has to "find new and strange routes to O" almost tautologically.

Once we build AI sy

A more charitable interpretation is that this is a probability rounded to the nearest percent

I don’t think most people are trying to explicitly write down all human values and then tell them to an AI. Here are some more promising alternatives:

1. Tell an AI to “consult a human if you aren’t sure what to do”
2. Instead of explicitly trying to write down human values, learn them by example (by watching human actions, or reading books, or…)

Why should we expect AGIs to optimize much more strongly and “widely” than humans? As far as I know a lot of AI risk is thought to come from “extreme optimization”, but I’m not sure why extreme optimization is the default outcome.

To illustrate: if you hire a human to solve a math problem, the human will probably mostly think about the math problem. They might consult google, or talk to some other humans. They will probably not hire other humans without consulting you first. They definitely won’t try to get brain surgery to become smarter, or kill everyone ... (read more)

I agree with it but I don’t think it’s making very strong claims.

I mostly agree with part 1; just giving advice seems too restrictive. But there’s a lot of ground between “only gives advice” and “fully autonomous” and “fully autonomous” and “globally optimizing a utility function”, and I basically expect a smooth increase in AI autonomy over time as they are proved capable and safe. I work in HFT; I think that industry has some of the most autonomous AIs deployed today (although not that sophisticated), but they’re very constrained over what actions they c... (read more)

My sense is that the existing arguments are not very strong (e.g. I do not find them convincing), and their pretty wide acceptance in EA discussions mostly reflects self-selection (people who are convinced that AI risk is a big problem are more interested in discussing AI risk). So in that sense better intro documents would be nice. But maybe there simply aren't stronger arguments available? (I personally would like to see more arguments from an "engineering" perspective, starting from current computer systems rather than from humans or thought experiments... (read more)

I expect people to continue making better AI to pursue money/fame/etc., but I don't see why "better" is the same as "extremely goal-directed". There needs to be an argument that optimizer AIs will outcompete other AIs.

Eliezer says that as AI gets more capable, it will naturally switch from "doing more or less what we want" to things like "try and take over the world", "make sure it can never be turned off", "kill all humans" (instrumental goals), "single-mindedly pursue some goal that was haphazardly baked in by the training process" (inner optimization), ... (read more)

IMO the biggest hole here is "why should a superhuman AI be extremely consequentialist/optimizing"? This is a key assumption; without it concerns about instrumental convergence or inner alignment fall away. But there's no explicit argument for it.

Current AIs don't really seem to have goals; humans sort of have goals but very far from the level of "I want to make a cup of coffee so first I'll kill everyone nearby so they don't interfere with that".

I don't think "burn all GPUs" fares better on any of these questions. I guess you could imagine it being more "accessible" if you think building aligned AGI is easier than convincing the US government AI risk is truly an existential threat (seems implausible).

"Accessibility" seems to illustrate the extent to which AI risk can be seen as a social rather than technical problem; if a small number of decision-makers in the US and Chinese governments (and perhaps some semiconductor companies and software companies) were really convinced AI risk was a concern, t... (read more)

Isn't "bomb all sufficiently advanced semiconductor fabs" an example of a pivotal act that the US government could do right now, without any AGI at all?

If current hardware is sufficient for AGI than maybe that doesn't make us safe, but plausibly current hardware is not sufficient for AGI, and either way stopping hardware progress would slow AI timelines a lot.

A > B > human. I expect B < human would also be quite useful.

B does not have a lot of opportunity for action - all it can do is prevent A from acting. It seems like its hard to "eliminate humans" with just that freedom. I agree B has an incentive to hamper A.

Even a B that is dumber than humans is valuable, because it is faster than humans. B can provide real-time feedback on every action that A takes, whereas consulting humans for every action would be impractical - it costs too much human time and it would prevent A from operating in realtime.

Why isn't it competitive? A is being trained the same way as an agentic system, so it will be competitive.

Adding B is a 2x runtime/training-cost overhead, so there is a "constant factor" cost; is that enough to say something is "not competitive"? In practice I'd expect you could strike a good safety/overhead balance for much less.

On (3): I don't expect an agentic AI to consist of a single deep learning model. For concreteness, let me sketch an architecture:

We have some robot R that we want an AI to run. R will supply the AI with sensory input and take actions that the AI recommends. R is like the human body and the AI is like the brain.

AI A is trained to operate R using whatever method you like. It ends up with some goal. AI B is trained as an "overseer" alongside A; it takes in actions recommended by A and decides whether to execute or reject them; B is supposed to reject actions ... (read more)

Just commenting on the concept of "goals" and particularly the "off switch" problem: no AI system has (to my knowledge) run into this problem, which IMO strongly suggests that "goals" in this sense are not the right way to think about AI systems. AlphaZero in some sense has a goal of winning a Go game, but AlphaZero does not resist being turned off, and I claim its obvious that even a very advanced version of AlphaZero would not resist being turned off. The same is true for large language models (indeed, it's not even clear the idea of turning off a language model is meaningful, since different executions of the model share no state). 

I think a more likely explanation is that people just like to complain. Why would people do things that everyone thought were a waste of time? (At my office, we have meetings and email too, but I usually think they are good ways to communicate with people and not a waste of time)

Also, you didn't answer my question. It sounds like your answer is that you are compelled to waste 20 hours of time every week?

I don't understand. Are you saying you could get 2x as much work done in your 40 hour week, or that due to dependencies on other people you cannot possibly do more than 20 hours of productive work per week no matter how many hours you are in the office?

False. At a company-wide level, Google makes an effort to encourage work-life balance.

Ultimately you need to produce a reasonable amount of output ("reasonable" as defined by your peers + manager). How it gets there doesn't really matter.

Sort of. My opinion takes that objection into account.

But on the other hand, I don't have any data to quantitatively refute or support your point.

I work at Google, and I work ~40 hours a week. And that includes breakfast and lunch every day. As far as I can tell, this is typical (for Google).

I think you can get more done by working longer hours...up to a point, and for limited amounts of time. Loss in productivity still means the total work output is going up. I think the break-even point is 60h / week.

Why not start with a probability distribution over (the finite list of) objects of size at most N, and see what happens when N becomes large?

It really depends on what distribution you want to define though. I don't think there's an obvious "correct" answer.

Here is the Haskell typeclass for doing this, if it helps: https://hackage.haskell.org/package/QuickCheck-2.1.0.1/docs/Test-QuickCheck-Arbitrary.html

Unfortunately, it seems much easier to list particularly inefficient uses of time than particularly efficient uses of time :P I guess it all depends on your zero point.

I think for most things, it's important to have a specific person in charge, and have that person be responsible for the success of the thing as a whole. Having someone in charge makes sure there's a coherent vision in one person, makes a specific person accountable, and helps make sure nothing falls through the cracks because it was "someone else's job". When you're in charge, everything is your job.

If no one else has taken charge, stepping up yourself can be a good idea. In my software job, I often feel this way when no one is really championin... (read more)

I was using "power" in the sense of the OP (which is just: more time/skills/influence). Sorry the examples aren't as dramatic as you would like; unfortunately, I can't think of more dramatic examples.

I disagree.

1 and 2 are "negative": avoiding common failure modes.

3 and 4 are "positive": ways to get "more bang for your buck" than you "normally" would.

This seems true, but obvious. I'm not sure that I buy that fiction promotes this idea: IMO, fiction usually glosses over how the characters got their powers because it's boring. Some real-life examples of power for cheap would be very useful. Here are some suggestions:

• Stick your money in index funds. This is way easier and more effective than trying to beat the market.
• Ignore the news. It will waste your time and make you sad.
• Go into a high-paying major / career
• Ask for things/information/advice. Asking is cheap, and sometimes it works.

Anyone have other real-world suggestions?

Say the player thought that they were likely win the lottery, that it was a good purchase. This may seem insane to someone familiar with probability and the lottery system, but not everyone is familiar with these things.

I would say this person made a good decision with bad information.

Perhaps we should attempt to stop placing so much emphasis on individualism and just try to do the best we can while not judging others nor other decisions much.

There are lots of times when it's important to judge people e.g. for hiring or performance reviews.

The pervasive influence of money in politics sort of functions as a proxy of this. YMMV for whether it's a good thing...

Doesn't "contrarian" just mean "disagrees with the majority"? Any further logic-chopping seems pointless and defensive.

The fact that 98% of people are theists is evidence against atheism. I'm perfectly happy to admit this. I think there is other, stronger evidence for atheism, but the contrarian heuristic definitely argues for belief in God.

Similarly, believing that cryonics is a good investment is obviously contrarian. AGI is harder to say; most people probably haven't thought about it.

It seems like the question you're really trying to... (read more)

Most of your post is not arguments against curing death.

People being risk-averse has nothing to do with anti-aging research and everything to do with individuals not wanting to die...which has always been true (and becomes more true as life expectancy rises and the "average life" becomes more valuable). The same is true for "we should risk more lives for science".

I agree that people adapt OK to death, but I think you're poking a strawman; the reason death is bad is because it kills you, not because it makes your friends sad.

I think &quo... (read more)

The problem of "old people will be close-minded and it will be harder for new ideas to gain a foothold" seems pretty inherent in abolishing death, and not just an implementation detail we can work around.

Yeah, this is a priority for me. My plan is to stick my money in a few mutual funds and forget about it for 40 years. Hopefully the economy will grow in that time :)

OK, I believe there is conflicting research. There usually is. And as usual, I don't know what to make of it, except that the preponderance of search hits supports $75k as satisficing. shrug

I think I saw that on LessWrong quite recently. That study is trying to refute the claim that income satisficing happens at ~$20k (and is mostly focused on countries rather than individuals). $20k << $75k.