All of AnthonyC's Comments + Replies

If you do it right, being willing to ask questions of those higher up, like said CEO, is how you get noticed, on their radar, as someone potentially worth watching and investing in and promoting in the future. A secure CEO in a healthy culture is likely to take it as a good sign that employees are aware, intelligent, and paying attention enough to ask clear, well-formed questions.

But if you ask a question in a way that offends that particular individual in whatever way, or makes your direct boss look bad to his direct boss (in either of their perceptions),... (read more)

Without a currently-implausible level of trust in a whole bunch of models, people, and companies to understand how and when to use privileged information and be able to execute it, removing the New Chat button would be a de factor ban on LLM use in some businesses, including mine (consulting). The fact that Chemical Company A asked a question about X last month is very important information that I'm not allowed to use when answering Chemical Company B's new question about the future of X, and also I'm not allowed to tell the model where either question cam... (read more)

There's an important reason to keep some of us around. This is also an important point.

Consequentialism tells us that the thing to do is the thing with the best results. But, this is a ridiculously high standard, that nobody can actually live up to. Thus, consequentialism tells us that everybody is bad, and we should all condemn everybody and all feel guilty. 

In these scenarios I like point out that the other party is using an appeal to consequences as the justification for rejecting consequentialism.

This, to me, gestures at a set of questions with potentially different answers.

1. If I've been living as a sexual person of any kind, should I choose to make myself ace, given the choice?
2. If I've been living as an asexual person, should I choose to change that, given the option? If so, to what sexuality?
3. If I am in something like Rawls' original position and can choose my sexuality for my upcoming life, what should I pick?
4. If I am in something like Rawls' original position and can choose everyone's sexualities for their upcoming lives, what should I pick?

(1) and... (read more)

Some of both, more of the former, but I think that is largely an artifact of how we have historically defined tasks. None of us have ever managed an infinite army of untrained interns before, which is how I think of LLM use (over the past two years they've roughly gone from high school student interns to grad student interns), so we've never refactored tasks into appropriate chunks for that context. 

I've been leading my company's team working on figuring out how to best integrate LLMs into our workflow, and frankly, they're changing so fast with new r... (read more)

These are very reasonable questions that I learned about the hard way camping in the desert two years ago. I do not recommend boondocking in central Wyoming in August. 

First, because when you live in an aluminum box with 1" thick R7 walls you need more air conditioning in summer than that much solar can provide. It doesn't help that RV air conditioners are designed to be small and light and cheap (most people only use them a handful of days a year), so they're much less efficient than home air conditioners, even window units. I have 2x 15k BTU/hr AC u... (read more)

Yes, this lines up with current average prices for solar at time of production vs firmed. We're only finally starting to see firmed green power prices get covered much even by experts, now that penetration rates are rising and companies are realizing they made big noises about 2030 and 2050 goals before having actually made any kind of plan to achieve them.

But, unless you're at Kardashev-1 levels of power demand (we're not), why would you try to run a grid on all solar? Who is proposing doing that, even in the world's sunniest regions? The most cost-effect... (read more)

I can't comment on software engineering, not my field. I work at a market research/tech scouting/consulting firm. What I can say is that over the past ~6 months we've gone from "I put together this 1 hour training for everyone to get some more value out of these free LLM tools," to "This can automate ~half of everything we do for $50/person/month." I wouldn't be surprised if a few small improvements in agents over the next 3-6 months push that 50% up to 80%, then maybe 90% by mid next year. That's not AGI, but it does get you to a place where you need people to have significantly more complex and subtle skills, that currently take a couple of years to build, before their work is adding significant value. 

The clockwise rule is what you are supposed to do if people arrive to the intersection at the same time.

If exactly two people going opposite directions arrive at the same time and aren't both going straight, then the one going straight goes before the one turning, or the one turning right goes before the one turning left.

At least, that's how I and everyone I know was taught, and no, those of us who asked what "at the same time" actually means never got a straight answer. 

Sure. And I'm of the opinion that it is only common sense after you've done quite a lot of the work of developing a level of intuition for mathematical objects that most people, including a significant proportion of high school math teachers, never got.

Wanted to add:

I think this post is great for here on LW, but if someone wanted to actually start teaching students to understand math more deeply, calling it common sense probably comes off as condescending, because it doesn't feel that way until you get comfortable with it. There's a lot to unlearn and for a lot of people it is very intimidating.

Personally I wish we treated math class at least some of the time as a form of play. We make sure to teach kids about jokes and wordplay and do fun science-y demonstrations, but math is all dry and technical. We a... (read more)

Fair enough. 

I do believe it's plausible that feelings, like pain and hunger, may be old and fundamental enough to exist across phyla. 

I'm much less inclined to assume emotions are so widely shared, but I wish I could be more sure either way.

Mostly agreed. I have no idea how to evaluate this for most animals, but I would be very surprised if other mammals did not have subjective experiences analogous to our own for at least some feelings and emotions.

which I worry your teachers didn’t

Oh it can be so much worse than that - actively pushing students away from that kind of understanding. I've had math teachers mark answers wrong because I (correctly) derived a rule I'd forgotten instead of phrasing it the way they taught it, or because they couldn't follow the derivation. Before college, I can think of maybe two of my teachers who actually seemed to understand high school math in any deeper way.

Thanks, fixed!

As in, if a truck stops, within 10 minutes you have to put out orange triangles. But a driverless truck has no way to do that

This...seems like a problem that can be solved with basically three slightly-customized triangle-carrying roombas that drop from the rear bumper and form a line? May remote controllable with a remotely monitorable rear view camera or small camera drone?

Yet evolution specifically programmed humans to never ever worry about the welfare of beings in our "imagination," because "they aren't real."

True. And yet we don't even need to go as far as a realistic movie to override that limitation. All it takes to create such worry is to have someone draw a 2D cartoon of a very sad and lonely dog, which is even less real. Or play some sad music while showing a video a lamp in the rain, which is clearly inanimate. In some ways these induced worries for unfeeling entities are super stimuli for many of us, stronger than... (read more)

I agree that AI in general has the potential to implement something-like-CEV, and this would be better than what we have now by far. Reading your original post I didn't get much sense of attention to the 'E,' and without that I think this would be horrible. Of course, either one implemented strongly enough goes off the rails unless it's done just right, aka the whole question is downstream of pretty strong alignment success, and so for the time being we should be cautious about floating this kind of idea and clear about what would be needed to make it a go... (read more)

LLMs may enable direct democracy at scale

I suppose they could, and from now on I'll consider this to be one of the many significant dangers of governance by AI. 

Part of the problem with direct democracy is that it provides no unambiguous mechanism for leaders to exercise judgment about the relative importance of different individuals' preferences and concerns, or the quality of their information and reasoning. A great many of America's, and the world's, most important advances and accomplishments in governance have happened in spite of, not because of... (read more)

For learning the foundational principles of stat mech, yes, you can start that way. For example, you can certainly come up with definitions of pressure and temperature for an ideal gas or calculate the heat capacity of an Einstein solid without any quantum considerations. But once you start asking questions about real systems, the quantization is hugely important for e.g. understanding the form of the partition function.

I'm so sorry, and I hope someday you can clone Phoenix. I just lost my own black cat in December at age 15, and I am 100% convinced that black cats, in general, are just friendlier, sweeter, and more social - and I say that as someone who up until 2012 thought he didn't like cats! Phoenix sounds a lot like she was. Up to and including the fluids - she tolerated anything, just trusted we had a reason and would make her feel better.

On the one hand, I agree with the arc of this discussion. I would also like to have better tools for societal information sharing, vetting, collaboration, and truth-seeking.

On the other, I'm not sure I buy the premise that this has gotten worse. Whenever I've actually looked back at media from the past, what I find is that past media, both mainstream and fringe, were in many ways even worse in terms of information quality than today (other than maybe the high points within very heavily regulated midcentury broadcast TV, which enforced something of a monocu... (read more)

I agree, but when people want to use the presence or absence of Original Thought™ as a criterion for judging the capabilities of AI, then drawing that line somewhere matters, and the judge should write it down, even if it is approximate.

I'm not sure what the concept of and "entirely new" or "fully novel" idea means in practice. How many such things actually exist and how often should we expect any mind however intelligent to find one? Ideas can be more or less novel, and we can have thresholds for measuring that, but where should we place the bar?

If you place it at "generate a correct or useful hypothesis you don't actually have enough data to locate in idea-space" then that seems like a mistake.

I'd put it more near "generate and idea good enough to lead to a publishable scientific paper ... (read more)

I'm generally in favor of being polite even to inanimate objects, and approach LLMs the same way. 

Does this point get meaningfully strengthened by the way companies use past chats to train future models? Or is that mostly noise?

I'm still confused enough about consciousness that I can only directionally and approximately agree, but I do agree with that.

It gets really fun when the same individual holds multiple titles with conflicting obligations, and ends up doing things like approve and then veto the same measure while wearing different hats. I also think it's unfortunate that we seem to have gotten way to intolerant of people doing this compared to a few decades or generations ago. We're less willing to separate individuals from the roles they're enacting in public life, and that makes many critical capabilities harder.

Yes, this is exactly what I do expect. There are many problems for which this is a sufficient or even good approach. There are other problems for which it is not. And there are lessons that (most?) governments seem incapable of learning (often for understandable or predictable reasons) even after centuries or millennia. This is why I specified that I don't think we can straightforwardly say the government does or does not know a complicated thing. Does the government know how to fight a war? Does it know how to build a city? How to negotiate and enact a tr... (read more)

The Eye of Sauron turned its gaze upon the Fellowship, and still didn't know that they'd actually try to destroy the ring instead of use it.

Less abstractly, I agree with you in principle, and I understand that many examples of the phenomena you referenced do exist. But there are also a large number of examples of government turning its eye on a thing, and with the best of intentions completely butchering whatever they wanted to do about it by completely failing to address the critical dynamics of the issue. And then not acknowledging or fixing the mistake,... (read more)

"The government" is too large, diffuse, and incoherent of a collective entity to be straightforwardly said to know or not know anything more complicated on most topics than a few headlines. I am certain there are individuals and teams and maybe agencies within the government that understand, and others that are so far from understanding they wouldn't know where to begin learning.

Yeah, I've been waiting for this as a sign of energy storage maturing for around 10-15 years. Ironically I had a conversation just this morning with someone who plans utility projects, and they told me that they're finally starting to see operators of gas peaker plants develop plans to start buying batteries so they can run their plants at higher efficiency/closer to constant power.

Upvoted for you posting it at all. I think these stories can be a great window into a culture I don't understand even a little. Whatever you decide to post in the future, it would be great to get your reflections on why you chose a particular story, what it means to you, that kind of thing.

When I was in high school I was drum major of the marching band, they sent us to a week long "leadership camp" training, and this was how they recommended giving criticism. Praise-correction-praise. It can be done well, but is much more often done poorly. Basically, it's level-1 advice that needs to be executed very well or used on an audience that isn't really free to complain much about it, and by the time you are able to do so skill-wise, there are better methods available.

As someone tasked with deciding what AI tools the company I work for should be using, and training people to use them, the version names and numbers have been tons of fun. "Deep Research, not DeepSeek. No the other one. No no, the other other one." 

Although, today I did remind myself that (over a much longer timespan) the version names/numbers for Windows major releases have been 3.1, 95, NT, 98 Second Edition, 2000, ME, XP, Vista, 7, 8, 10, and 11. And also almost no founder should ever be allowed to name their company.

And so I need to point out that when people enslaved human beings of equal intelligence with limited information access, it still didn’t end well for the slavers.

I would point out that for thousands of years, it very much often did. Sometimes spectacularly so. Even in the US, it went very well for many of the slavers, and only ended poorly for their many-times-great-grandchildren who didn't get a say in the original policy discussion.

I do in fact believe this is relevant, since in the context of AI I expect that early successes in aligning weak systems are... (read more)

Falsification is, in general, not actually a useful metric, because evidence and strength of belief are quantitative and the space of hypotheses is larger than we can actually scan.

I'd note that the layperson's description of a black hole is, in fact, false. Squeezing a given mass into a singularity doesn't make it heavier. The mass stays the same, but the density goes up. Even as it collapses into a black hole, the Schwartzchild radius will be much smaller than the original object's size - about 3km for a 1 solar mass black hole. If you personally could d... (read more)

I think this is all reasonable, but I'm unsure who the target audience is for this post? I ask because this all seems par-for-the-course on LW as to what people should be doing, and a source of despair that leading labs frequently aren't. 

Your outline lays out multiple very hard but not impossible problems that need to be solved before RSI really gets going (assuming it does) for it to reliably go well. People here have been shouting about them for over 15 years now. Yet, we're not close to solving any of them, and also the leading AI labs are repeatedly claiming we'll have RSI within 1-3 years and ASI in 5-10.

I think we'd be talking about AI progress slowing down at this point if it weren't for reasoning models.

Possibly, but 1) There are reasoning models, 2) Value per token may still raise faster than cost per token for non-reasoning models which could be enough to sustain progress, and 3) It's possible that a more expensive non-reasoning model makes reasoning more efficient and/or effective by increasing the quality and complexity of each reasoning step.

At this point I pretty much never use 4o for anything. It's o1, o1-pro, or o3-mini-high. Looking forward to testing 4.5 though.

I don't really get the argument that ASI would naturally choose to isolate itself without consuming any of the resources humanity requires. Will there be resources ASI uses that humanity can't? Sure, I assume so. Is it possible ASI will have access to energy, matter, and computational resources so much better that it isn't worth its time to take stuff humans want? I can imagine that, but I don't know how likely it is, and in particular I don't know why I would expect humans to survive the transitional period as a maturing ASI figures all that out. It seems... (read more)

So, this is true in two (not really independent) senses I can think of. First, in most cases, there isn't enough money chasing shares to sell all shares at the current price. Second, the act of shareholders selling in large numbers is new information that itself changes the price. The current price is a marginal price, and we don't usually know how steeply the rest of the price curve slopes at larger volumes for either buying or selling.

I'm curious as to the viewpoint of the other party in these conversations? If they're not aware of/interested in/likely to be thinking about the disruptive effects of AI, then I would usually just omit mentioning it. You know you're conditioning on that caveat, and their thinking does so without them realizing it.

If the other party is more AI-aware, and they know you are as well, you can maybe just keep it simple, something like, "assuming enough normality for this to matter."

True, that can definitely happen, but consider

1) the median and average timeline estimates have been getting shorter, not longer, by most measures, 

and 

2) no previous iteration of such claims was credible enough to attract hundreds of billions of dollars in funding, or meaningfully impact politics and geopolitics, or shift the global near-consensus that has held back nuclear power for generations. This suggests a difference in the strength of evidence for the claims in question.

Also 3) When adopted as a general principle of thought, this approach... (read more)

Oh, I already completely agree with that. But quite frankly I don't have the skills to contribute to AI development meaningfully in a technical sense, or the right kind of security mindset to think anyone should trust me to work on safety research. And of course, all the actual plans I've seen anyone talk about are full of holes, and many seem to rely on something akin to safety-by-default for at least part of the work, whether they admit it or not. Which I hope ends up not being true, but if someone decides to roll the dice on the future that way, then it... (read more)

And that makes perfect sense. I guess I'm just not sure I trust any particular service provider or research team to properly list the full set of things it's important to weight against. Kind of feels like a lighter version of not trusting a list of explicit rules someone claims will make an AI safe.

True, and this does indicate that children produced from genes found in 2 parents will not be outside the range which a hypothetical natural child of theirs could occupy. I am also hopeful that this is what matters, here. 

However, there are absolutely, definitely viable combinations of genes found in a random pair of parents which, if combined in a single individual, result in high-IQ offspring predisposed to any number of physical or mental problems, some of which may not manifest until long after the child is born. In practice, any intervention of t... (read more)

I definitely want to see more work in this direction, and agree that improving humans is a high-value goal.

But to play devil's advocate for a second on what I see as my big ethical concern: There's a step in the non-human selective breeding or genetic modification comparison where the experimenter watches several generations grow to maturity, evaluates whether their interventions worked in practice, and decides which experimental subjects if any get to survive or reproduce further. What's the plan for this step in humans, since "make the right prediction e... (read more)

I expect that we will probably end up doing something like this, whether it is workable in practice or not, if for no other reason than it seems to be the most common plan anyone in a position to actually implement any plan at all seems to have devised and publicized. I appreciate seeing it laid out in so much detail.

By analogy, it certainly rhymes with the way I use LLMs to answer fuzzy complex questions now. I have a conversation with o3-mini to get all the key background I can into the context window, have it write a prompt to pass the conversation onto... (read more)

Thanks for writing this. I said a few years ago, at the time just over half seriously, that there could be a lot of value in trying to solve non-AI-related problems even on short timelines, if our actions and writings become a larger part of the data on which AI is trained and through which it comes to understand the world.

That said, this one gives me pause in particular: 

I hope you treat me in ways I would treat you

I think that in the context of non-human minds of any kind, it is especially important to aim for the platinum rule and not the golden. We want to treat them the way they would want to be treated, and vice versa.

I agree with many of the parts of this post. I think xkcd was largely right, our brains have one scale and resize our experiences to fit. I think for a lot of people the hardest step is to just notice what things they actually like, and how much, and in what quantities before they habituate. 

However, the specific substitutions, ascetic choices, etc. are very much going to vary between people, because we have different preferences. You can often get a lot of economic-efficiency-of-pleasure benefit by embracing the places where you prefer things society... (read more)

In the world where AI does put most SWEs out of work or severely curtails their future earnings, how likely is it that the economy stays in a context where USD or other fiat currencies stay valuable, and for how long? At some level we don't normally need to think about, USD has value because the US government demands citizens use that currency to pay taxes, and it has an army and can ruin your life if you refuse. 

I've mentioned it before and am glad to see people exploring the possibilities, but I really get confused whenever I try to think about (absolute or relative) asset prices along the path to AGI/ASI.