Lightcone Infrastructure FundraiserGoal 1:$891,196 of $1,000,000
Customize
Rationality+Rationality+World Modeling+World Modeling+AIAIWorld OptimizationWorld OptimizationPracticalPracticalCommunityCommunity
Personal Blog+
I'm worried that it will be hard to govern inference-time compute scaling.  My (rather uninformed) sense is that "AI governance" is mostly predicated on governing training and post-training compute, with the implicit assumption that scaling these will lead to AGI (and hence x-risk).  However, the paradigm has shifted to scaling inference-time compute. And I think this will be much harder to effectively control, because 1) it's much cheaper to just run a ton of queries on a model as opposed to training a new one from scratch (so I expect more entities to be able to scale inference-time compute) and 2) inference can probably be done in a distributed way without requiring specialized hardware (so it's much harder to effectively detect / prevent).  Tl;dr the old assumption of 'frontier AI models will be in the hands of a few big players where regulatory efforts can be centralized' doesn't seem true anymore.  Are there good governance proposals for inference-time compute? 
Kaj_Sotala4924
6
I doubt that anyone even remembers this, but I feel compelled to say it: there was some conversation about AI maybe 10 years ago, possibly on LessWrong, where I offered the view that abstract math might take AI a particularly long time to master compared to other things. I don't think I ever had a particularly good reason for that belief other than a vague sense of "math is hard for humans so maybe it's hard for machines too". But formally considering that prediction falsified now.
links 12/23/2024: https://roamresearch.com/#/app/srcpublic/page/12-23-2024 *  a bunch of detailed (and still debated) fan meta about what's even going on in Neon Genesis Evangelion * https://forum.evageeks.org/thread/20093/Kaworu-and-SEELE-Gendos-Plans-Angel-Rebirth/ * https://forum.evageeks.org/thread/20116/Questions-about-SEELEs-Gendos-Angels-goals/ * https://wiki.evageeks.org/Theory_and_Analysis:Kaworu%27s_Agenda * https://www.reddit.com/r/evangelion/comments/1ech72/what_does_seele_actually_want_ive_only_watched/ * https://www.syllabi.directory/ "syllabi" or informational resources for getting up to speed on various topics: so far topics include clinical trial design, cities, English literature, and housing supply. * https://en.m.wikipedia.org/wiki/Samuel_Langley Early pioneer of aviation; in 1903 his steam-powered airplane actually flew farther than the Wright Brothers' famous attempt later that year, but crashed in the Potomac due to poor controllability. * an online acquaintance argued recently that the reason the Wright Brothers are celebrated as pioneers of flight is not because they were "first" to achieve sustained powered flight, but that they were the first to start an airplane company, to continue the project and iteratively improve and teach others. Lots of people claim competing "first flight" stories, but (including Langley's) they were all one-off stunts that didn't leave a legacy. * also (according to him), controllability is a key piece of the puzzle, and there's a tradeoff between controllability and stability (too "controllable" and the plane will respond all-too-vigorously to small random pilot movements) and the Wrights' three-axis control setup was the most successful and the one that came to be the ancestor of modern flight control mechanisms. * https://nyuu.page/essays/solidity/ extremely detailed analysis of one type of crypto scam that was popular in 2021 with smart contracts * https://www.fuisz.xyz/blog blog by
leogao2010
6
people around these parts often take their salary and divide it by their working hours to figure out how much to value their time. but I think this actually doesn't make that much sense (at least for research work), and often leads to bad decision making. time is extremely non fungible; some time is a lot more valuable than other time. further, the relation of amount of time worked to amount earned/value produced is extremely nonlinear (sharp diminishing returns). a lot of value is produced in short flashes of insight that you can't just get more of by spending more time trying to get insight (but rather require other inputs like life experience/good conversations/mentorship/happiness). resting or having fun can help improve your mental health, which is especially important for positive tail outcomes. given that the assumptions of fungibility and linearity are extremely violated, I think it makes about as much sense as dividing salary by number of keystrokes or number of slack messages. concretely, one might forgo doing something fun because it seems like the opportunity cost is very high, but actually diminishing returns means one more hour on the margin is much less valuable than the average implies, and having fun improves productivity in ways not accounted for when just considering the intrinsic value one places on fun.
Hopenope3718
0
Many expert level benchmarks totally overestimate the range and diversity of their experts' knowledge. A person with a PhD in physics is probably undergraduate level in many parts of physics that are not related to his/her research area, and sometimes we even see that within expert's domain (Neurologists usually forget about nerves that are not clinically relevant).

Popular Comments

Recent Discussion

My median expectation is that AGI[1] will be created 3 years from now. This has implications on how to behave, and I will share some useful thoughts I and others have had on how to orient to short timelines.

I’ve led multiple small workshops on orienting to short AGI timelines and compiled the wisdom of around 50 participants (but mostly my thoughts) here. I’ve also participated in multiple short-timelines AGI wargames and co-led one wargame.

This post will assume median AGI timelines of 2027 and will not spend time arguing for this point. Instead, I focus on what the implications of 3 year timelines would be. 

I didn’t update much on o3 (as my timelines were already short) but I imagine some readers did and might feel disoriented now. I hope...

rvnnt20

make their models sufficiently safe

What does "safe" mean, in this post?

Do you mean something like "effectively controllable"? If yes: controlled by whom? Suppose AGI were controlled by some high-ranking people at (e.g.) the NSA; with what probability do you think that would be "safe" for most people?

2rvnnt
I think this post (or the models/thinking that generated it) might be missing an important consideration[1]: "Is it possible to ensure that the nationalized AGI project does not end up de facto controlled by not-good people? If yes, how?" Relevant quote from Yudkowsky's Six Dimensions of Operational Adequacy in AGI Projects (emphasis added): Another quote (emphasis mine): ---------------------------------------- 1. or possibly a crucial consideration ↩︎ 2. The quote is referring to "[...] a single global Manhattan Project which is somehow not answerable to non-common-good command such as Trump or Putin or the United Nations Security Council. [...]" ↩︎
2Oscar
Substantiate? I down- and disagree-voted because of this un-evidenced very grave accusation.
1kairos_
Assuming short timelines, I wonder how much NVIDIA's stock will increase and if anywhere near a 100x return is possible. The further out and higher strike price NVIDIA call I could find is at 290$ SP, dated Jan 15 2027, at $13.25. If NVIDIA goes to a 10T market cap I get an 8x return on investment, if the company goes to a 15T market cap I get a ~20x return on investment.  I'm not sure how realistic it is for NVIDIA to increase past a 15 Trillion Market cap. Plus, increased government intervention seems like it would negatively impact profits.

A new article in Science Policy Forum voices concern about a particular line of biological research which, if successful in the long term, could eventually create a grave threat to humanity and to most life on Earth.

Fortunately, the threat is distant, and avoidable—but only if we have common knowledge of it.

What follows is an explanation of the threat, what we can do about it, and my comments.

Background: chirality

Glucose, a building block of sugars and starches, looks like this:

Adapted from Wikimedia

But there is also a molecule that is the exact mirror-image of glucose. It is called simply L-glucose (in contrast, the glucose in our food and bodies is sometimes called D-glucose):

L-glucose, the mirror twin of normal D-glucose.
L-glucose, the mirror twin of normal D-glucose. Adapted from Wikimedia

This is not just the same molecule flipped around,...

Not all forms of mirror biology would even need to be restricted. For instance, there are potential uses for mirror proteins, and those can be safely engineered in the lab. The only dangerous technologies are the creation of full mirror cells, and certain enabling technologies which could easily lead to that (such as the creation of a full mirror genome or key components of a proteome).

Once we get used to create and deal with mirror proteins, and once we get used to designing & building cells, which I don't know when it happens, maybe adding 1+1 togeth... (read more)

2jasoncrawford
Failure to detect other life in the universe is only really evidence against advanced intelligent civilizations, I think. The universe could easily be absolutely teeming with bacterial life. Re “take steps to stop it”, I was replying to @Purplehermann
1StartAtTheEnd
Many of the advantages are like that, but I think it's a little pessimistic not to dare to look anyway. I've personally noticed that people who are on the helpless side are good at making others want to help them, so not all insights are about immoral behaviour. But even then, aren't you curious how people less capable than yourself can be immoral without getting caught, or immoral in a way which others somehow forgive? Most things which can be used for evil can also be used for good, so I think it's a shame if you don't allow yourself to look and analyze (though I understand that things surrounding immorality can be off-putting) I'm not all that afraid of things surrounding morality, but it's because trust myself quite a lot, so the borders between good and bad are more clear (the grey area is smaller, it's more white and black) so I don't bully myself for just getting sort of close to immorality. I don't know if you do this yourself, but having steeper gradients has benefited me personally, I feel more mentally sharp after making my own boundaries clear to myself. I'm just sharing this because I think most people could benefit from it (less so LW users than the general population, but there should still be some)
2Ann
I don't know about bullying myself, but it's easy to make myself angry by looking too long at this manner of conceptual space, and that's not always the most productive thing for me, personally, to be doing too much of. Even if some of the instruments are neutral, they might leave a worse taste in my mouth for the deliberate association with the more negative; in the same way that if I associate a meal with food poisoning, it might be inedible for a long time.
2StartAtTheEnd
Sometimes I spend a few hours talking with myself, and finding out what I really believe, what I really value, and what I'm for and against. The effect is clearity of mind and a greater trust in myself. A lot of good and bad things have a low distance to eachother, for instance "arrogance" and "confidence", so without the granularity to differentiate subtle differences, you put yourself at a disadvantage, suspecting even good things. I suppose another reason that I recommend trusting yourself is that some people, afraid of being misunderstood and judged by others, stay away from anything which can be misunderstood as evil, so they distance themselves from any red flags with a distance of, say, 3 degrees of association. Having ones associations corrupted because something negative poisons everything without 3 degrees/links of distance has really screwed me over, so I kind of want you to hear me out on this: I might go to the supermarket, and buy a milkshake, but hate the experience because I know the milkshake has a lot of chemicals in it, because I hate the company which makes them, because I hate the advertisement, because I know the text on the bottle is misleading... But wait a minute, the milkshake tastes good, I like it, the hatred is a few associations away. What I did was sabotage my own experience of enjoying the milkshake, because if I didn't, it would feel like I was supporting something which I hated, merely because something like that existed 2-3 links away in concept space. I can't enjoy my bed because I think about dust mites, I can't enjoy video-games because I think about exploitative skinners boxes, I can't enjoy pop music because, even though I like the melody, I know that the singer is somewhat talentless and that somebody else wrote the lyrics for them. But, I have some young friends (early 20s) who simply enjoy what they enjoy and hate what they hate, and they do not mix the two. They drink a milkshake and it's tasty, and they listen to the m
Ann10

I think you are working to outline something interesting and useful, that might be a necessary step for carrying out your original post's suggestion with less risk; especially when the connection is directly there and even what you find yourself analyzing rather than multiple links away.

Once I talked to a person who said they were asexual. They were also heavily depressed and thought about committing suicide. I repeatedly told them to eat some meat, as they were vegan for many years. I myself had experienced veganism-induced depression. Finally, after many weeks they ate some chicken, and the next time we spoke, they said that they were no longer asexual (they never were), nor depressed.

I was vegan or vegetarian for many consecutive years. Vegetarianism was manageable, perhaps because of cheese. I never hit the extreme low points that I did with veganism. I remember once after not eating meat for a long time there was a period of maybe a weak, where I got extremely fatigued. I took 200mg of modafinil[1], without having any build-up resistance. Usually, this would give me a lot...

they said that they were no longer asexual (they never were),

I'm somewhat skeptical of the claim in parentheses. It certainly sounds like there is a state where they demonstrated enough traits to think they were asexual, and that information tends to be worth tracking, even if only for self-diagnostics.

2niplav
Huh, interesting!
2transhumanist_atom_understander
I'm replying to a post that said they lacked energy despite iron supplementation. Maybe it wasn't iron deficiency, or maybe it could have been solved by raising the supplement dose, I don't know, but if it was iron deficiency and the supplements weren't helping then it's good to know you can get iron in the same form as in meat from Impossible burgers.
2Johannes C. Mayer
I think running this experiment is generally worth it. It's very different to read a study and to run the experiment and see the effect yourself. You may also try to figure out if you are amino acid deficient. See this comment, as well as others in that comment stack.

“Just expose yourself to more social situations!” — Ah yes, you felt anxious the first 100 times, but the 101st will be the breakthrough!

“But exposure works!” people yell from across the street. “Like for fear of snakes - you know, those things you see once a year!”

Uh, it’s pretty rational to fear things you have little experience with. But social anxiety… you interact with people everyday! Why would anything change after the first 100 attempts?

I don’t doubt that a couple of exposures can often reduce anxieties. However, if you still feel anxious even after hundreds of social situations and years of trying... then maybe your fear is actually doing something presently useful and you should reconnect with your intuitions.

At a 100% eye contact workshop I led earlier this year, most...

I wrote (in Hebrew, alas) two years ago, about locally-useful methods that doesn't have stopping condition. I'm sure there are people out there that will benefit from exposure. the attitude you described come from them, and from people whose bubble includes mostly them.

the problem is the luck of stopping condition. who many tries before you decide this method doesn't work? before stopping and re-evaluating? before trying something else instead?

also, what Scott Alexander wrote about exposure, and Trapped Priors. 

I think you make the same mistake the ex... (read more)

To get the best posts emailed to you, create an account! (2-3 posts per week, selected by the LessWrong moderation team.)
Log In Reset Password
...or continue with

I’ve updated quite hard against computational functionalism (CF) recently (as an explanation for phenomenal consciousness), from ~80% to ~30%. Of course it’s more complicated than that, since there are different ways to interpret CF and having credences on theories of consciousness can be hella slippery.

So far in this sequence, I’ve scrutinised a couple of concrete claims that computational functionalists might make, which I called theoretical and practical CF. In this post, I want to address CF more generally.

Like most rationalists I know, I used to basically assume some kind of CF when thinking about phenomenal consciousness. I found a lot of the arguments against functionalism, like Searle’s Chinese room, unconvincing. They just further entrenched my functionalismness. But as I came across and tried to explain away more and more...

A failure of practical CF can be of two kinds: 

  1. We fail to create a digital copy of a person which have the same behavior with 99.9 fidelity. 
  2. Copy is possible, but it will not have phenomenal consciousness or, at least, it will be non-human or non-mine phenomenal consciousness, e.g., it will have different non-human qualia. 

    What is your opinion about (1) – the possibility of creating a copy?

Similar to other people's shortform feeds, short stuff that people on LW might be interested in, but which doesn't feel like it's worth a separate post. (Will probably be mostly cross-posted from my Facebook wall.)

1deepthoughtlife
Math is just a language (a very simple one, in fact). Thus, abstract math is right in the wheelhouse for something made for language. Large Language Models are called that for a reason, and abstract math doesn't rely on the world itself, just the language of math. LLMs lack grounding, but abstract math doesn't require it at all. It seems more surprising how badly LLMs did math, not that they made progress. (Admittedly, if you actually mean ten years ago, that's before LLMs were really a thing. The primary mechanism that distinguishes the transformer was only barely invented then.)

I disagree with this, in that good mathematics definitely requires at least a little understanding of the world, and if I were to think about why LLMs succeeded at math, I'd probably point to the fact that it's an unusually verifiable task, relative to the vast majority of tasks, and would also think that the fact that you can get a lot of high-quality data also helps LLMs.

Only programming shares these traits to an exceptional degree, and outside of mathematics/programming, I expect less transferability, though not effectively 0 transferability.

2Kaj_Sotala
Yeah I'm not sure of the exact date but it was definitely before LLMs were a thing.