Quick Takes — LessWrong

Weird thought-experiment I came up with:

Suppose you are magically transported to a society + economy "peopled" and run by present-day LLM-shaped intelligences. Suppose further that for whatever reason you can't do manual labor.

What remote-only work can you do that'd still be economically valuable?

__

I ask this because I think the "obvious" answers for what LLMs can't do today often rely on either embodiment or specific human preferences. For the later, either intrinsic preferences for the human touch (eg priests, therapists), or idiosyncratic preferences (e... (read more)

Gurkenglas6m20

Prediction market trading.

tailcalled's Shortform

tailcalled18h50

If math allowed mutable state, we could reduce the axiom of choice to excluded middle:

def axiom_of_choice(family):
    cache = {}
    def choice_fn(index):
        if index not in cache:
            cache[index] = family(index)
        return cache[index]
    return choice_fn

This relies on excluded middle because the ability to check whether an index is in the cache and if so look it up requires the ability to compare indices by equality.

(Sampling from family(index) is legit. The axiom of choice asserts that dependent products commute with propositional truncati... (read more)

Showing 3 of 4 replies (Click to show all)

2Viliam2h

I am not an expert, but I think solutions like this have a potential of hitting an infinite recursion, if evaluating some expression inside the code somehow indirectly invokes the axiom of choice, for example if you construct an object specifically for that purpose. My understanding of the axiom of choice, and the axioms in general, is that instead of merely describing how things work, they are describing what things are are talking about (by describing how they work). There are multiple concepts (in the Platonic realm) you could plausibly mean when you use the word "set". The axiom of choice says "I only mean those models that have the property that for any set of sets, there is a selection that works like this, and the selection itself is included in the model of 'sets'". There are models that follow the remaining axioms and have the property, and also models that follow the remaining axioms and don't have the property.

tailcalled33m20

I am not an expert, but I think solutions like this have a potential of hitting an infinite recursion, if evaluating some expression inside the code somehow indirectly invokes the axiom of choice, for example if you construct an object specifically for that purpose.

I don't think you run into this problem.

My understanding of the axiom of choice, and the axioms in general, is that instead of merely describing how things work, they are describing what things are are talking about (by describing how they work). There are multiple concepts (in the Platonic real

... (read more)

2tailcalled6h

family is a function that witnesses the "nonemptiness" (really, mere inhabitedness) by returning an element of the propositional truncation of the set. So more formally, axiom_of_choice has type (Πi: I. ||F(i)||) -> ||Πi: I. F(i)||. The "choice" comes from the fact that family (due to the propositional truncation) is allowed to return different F(i)'s from identical i's, whereas choice_fn is required to choose a canonical F(i) for each i . (The intermediate results are untyped because this is a realizer, not a typed term in a type theory.) 😅 This explanation is, I assume, not very helpful to anyone who doesn't already know what's going on. But basically in type theory, propositions correspond to (propositionally truncated) types, and proofs correspond to programs/terms with the types in question, so in proving the nonemptiness of the sets, you automatically end up with a witness function that picks out elements - except the witness function may be "nondeterministic" (that's where the propositional truncation comes in), which is what requires choice to fix. The propositional truncation is important to understand. There is a version of type theory's logic that doesn't include propositional truncation, but in it the axiom of choice is trivial to prove and not really of any use. Unfortunately the intuitive way to explain propositional truncation gives catastrophically wrong intuition for my use-case, as explained here: https://mathstodon.xyz/@MartinEscardo/111014413939196037

Caleb Biddulph's Shortform

Caleb Biddulph9h17-1

The Dead Economy Theory was on the top of Hacker News a few days ago. The post's strawmanning of rationalism/EA is a bit annoying, it's 58% AI-generated according to Pangram, and some of its ideas will already be well-known to LWers (despite claims that we have naively failed to consider them). However, the post packages together some potential blindspots I've been mulling over recently and brings up some points I hadn't thought about.

Some ideas and takeaways from (or vaguely reminiscent of) the post:

Some people actually want a job - people find work to be

... (read more)

Viliam2h20

it's 58% AI-generated according to Pangram

Most AI criticism seems to be, these days. (No, this is not about the Pope thing specifically.) We know what people want to read, and we know how to deliver it to them effectively...

why can't we just have our super-smart AI set up a society that gives people something to do?

Given that the choice of works that still remain for humans to do is somewhat arbitrary, I hope that it will include some solving of technical problems that does not require navigating office politics.

"only 51% of Americans are in favor of liter

... (read more)

2Charlie Steiner4h

Thank you for the summary. I think this imagines a world where we continue current trend lines for a few more years, but don't get AI that navigates the world and executes clever general plans like humans do. LLMs just get a little bit smarter, and a lot more work is put into making them good at specific domains (similar to what's been done for coding). (But AI never actually acts autonomously to compete with humanity in this imagined future. Either it never gets easy to get an AI to act like a clever goal-directed agent that navigates the real world, or it does become easy but somehow it never happens by accident and also nobody does it on purpose.) So we can have "even 'ASI' may be bad at [...] moral philosophy and social planning.", and in this imagined scenario it doesn't mean "the AI is amoral and we're fucked," but "the AI is amoral and this is a moderate inconvenience." This thing being referred to as "ASI" is just a few years linear extrapolation of current LLMs - it's superhuman at, say, managing factories only because humans have poured millions of dollars into building factory-management RL environments, and millions more dollars into training a big base model on them. It doesn't learn like humans do, and very especially it doesn't navigate the world in general like humans do - it's missing some of the key skills to do so, even though it has lots of skills for factory management. Amoral 'ASI' of this sort can't do anything complicated in the world without human help at many key steps. Not just menial help, actual help in the decision-making process. So when bad stuff happens, there's a bunch of convenient humans involved to blame. Anyhow, yeah, I agree, building AI "bad at moral philosophy" is bad even in near-term extrapolations where it's just running factories amorally and causing societal upheaval and making people sad. It's just that also, there's this thing where once the AI can do complicated stuff in the real world by itself, it being amoral

Yeonwoo Kim's Shortform

Salmonus_Kim3h00

In my view, the biggest limitation of the Logical Inductor lies in its failure to satisfy Desiderata 15 and 17. What is the ultimate value of a system if it lacks metacognition—the ability to determine on its own which problems are genuinely important—and fails to emulate the way human collective intelligence solves problems (akin to how Einstein’s theories revolutionized Newtonian mechanics)?

I believe an algorithm capable of distinguishing between a "larger problem" and a "smaller problem" is necessary. For instance, to prove that "primes are infinite," o... (read more)

leogao's Shortform

leogao5h95

i find it very interesting how becoming familiar with a place makes it feel so different, and yet it's recognizably the same place. especially when the first time you see the place you just think of it as a disconnected location floating around somewhere in abstract locationspace, but then slowly discover its relationship to other locations. sometimes this feels exhilarating, because the world feels more cohesive and whole and familiar. other times it's melancholy, because it feels like the magic has gone.

sam4h10

This is a big part of the appeal of travel for me. I feel like time speeds up and and presence drops a lot when my brain has a sufficiently good representation of the location I'm in.

1Andrii Vasylenko5h

This happens with concepts too.

Linch's Shortform

Linch16h*3123

One reflection I've had in the whole "AI use in the encyclical" affair is to slightly increase my trust in traditional media, especially non-American traditional media, and slightly decrease my trust in social media and new media epistemics.

I tried my best to promote my analysis as legibly and reasonably as I could and focused on logos rather than ethos: I didn't frame my article with institutional affiliations and intentionally chose not to include obvious, flashy, but irrelevant signaling. Stuff I could've done but explicitly chose not to: get an ML prof... (read more)

3Shankar Sivarajan11h

Frankly, I'd say it shows precisely the opposite. "New media" assessed your "analysis" as "some rando online thinks something is AI generated, runs it through an 'AI detector' which turns up positive, and posts about it. There are lots of these all the time, so that's entirely unremarkable by itself. Is the guy famous in some way? No, not particularly. Are impactful media outlets picking it up? No, at least not yet. This is not a story for us." Whereas traditional media outlets go something like "Okay, so the Pope just posted something, and we need to write about it. Let's see how people online are reacting … hmm, some guy is speculating it might be written by AI? That'll get us clicks! Oh, and he's using Pangram, which is peer-reviewed, so it's Science! Great, publish." [...] Okay. My own takeaway from your story getting traction has been to tell people that Pangram seems to now be prominent enough that they should run their writing through it and see if it gets flagged, and if it does, get some LLM to generate a few variants until it doesn't anymore.

Linch5h22

(I didn't downvote)

My own takeaway from your story getting traction has been to tell people that Pangram seems to now be prominent enough that they should run their writing through it and see if it gets flagged, and if it does, get some LLM to generate a few variants until it doesn't anymore.

This just seems straight-up antisocial, borderline evil. Why do you hate good things?

Bruce Middleton's Shortform

Bruce Middleton6h30

Spawn and winnow: Eric Drexler at my party, 1977

It was a Transvestite Housewarming Party, put on by me and two room-mates for our new off-campus digs - fascinated as we carved our foam breasts. His girlfriend - Margaret Minsky, if memory serves - stole the show, in knee-high boots and a black leather bikini, carrying a bull-whip. Not actually obeying the premise, but no-one complained. At the party, he was still all about the L5 Society. He swerved to nanotechnology later that year. Smalley became fascinated, then trashed him, en route to a Nobel - while n... (read more)

Sheikh Abdur Raheem Ali's Shortform

Sheikh Abdur Raheem Ali6h10

I was surprised to learn today that the AGI Safety & Alignment at GDM is unable to veto the deployment of a potentially unsafe model unless this is escalated to and signed off by decision makers that, due to competitive pressures, are largely incentivised to lean towards deploying potentially unsafe models unless compelled otherwise by government intervention.

I believe that there is some evidence that frontier AI labs publicly welcome this sort of regulatory oversight, but Anthropic has a proposed 1.5 page frontier model transparency framework which i... (read more)

Bruce Middleton's Shortform

Bruce Middleton8h*40

Bootlace fatalities: so you think you know how to tie your shoes?

Last year, while hiking, I had a bootlace snag on a speed hook on the other boot, and went down. It was like a low tackle: your feet stop moving, and your weight is still moving forward. A while before, an acquaintance had inexplicably fallen to his death while working on a roof - wearing workboots. I had a theory, looked online - and found Professor Shoelace: https://www.fieggen.com/shoelace/. Fatalities are regular. But though the good professor records them, he doesn't offer the solution: ... (read more)

banx7h10

I was running downhill holding my 3yo daughter’s hands when this happened to me. Not fun.

Taylor G. Lunt's Shortform

Taylor G. Lunt9h10

The original Vidhaven post (https://www.lesswrong.com/posts/MSng3Y4rsNLykbEF7/the-vidhaven-challenge) didn't get much attention, but it's not to late to join. 30 days, 30 videos, any subject.

I absolutely hate the videos I've made so far, so that's a sign I'm moving in the right direction I think.

Raven Of Empire's Shortform

Raven Of Empire3d30

Hobbes' "state of nature" honestly bothers me, particularly from the perhaps simplistic lens of its name. As a toy game-theoretic model of a multi-polar trap, it is interesting, but in reality it's too reductive. This isn't a new criticism; I just want to talk about it.

I dislike many things about the concept, but I will illustrate two: his assumption of a "war of all against all", and his assumption that this state of nature has ended.

The war of all against all is an oversimplification; humans are fundamentally collaborative animals. For example, an easy... (read more)

Showing 3 of 6 replies (Click to show all)

Raven Of Empire9h10

I do agree the wars are less in scale, I was not making any claim that they weren't. The claim is about the sum of all structural violence that occurs, the sum of Moloch. War, disease, poverty, starvation and death are all caused by unnatural concentrations of resources outside immediate access on the land. Those resources include the information and skill required to survive. In hunter-gatherer life, the knowledge and skills needed to survive were distributed across the community and accessible to everyone. Finally, even if you gained the skills to go off... (read more)

2Viliam1d

I agree that a human alpha cannot rule alone, he is typically a leader of a clique. But if egalitarianism is our nature, why is there so much bullying at schools, reinvented independently? My alternative explanation is that it is simply the dominant clique maintaining order.

1Raven Of Empire9h

I am not just making a claim, this egalitarianism you refer to is the conclusion of modern anthropology. You're right that the inference from 'people suppress bragging' to 'egalitarian society' deserves scrutiny in isolation. The logical point is sound, but you're applying it to a straw version of the claim. The anthropological case isn't built on the thin inference 'people suppress bragging, therefore egalitarianism'. You are doing a few things here; one is conflating egalitarianism with lack of violence, drama or conflict, another is a misunderstanding of what our social skills actually evolved to do and in what context. Our social skills evolved for groups of at most Dunbar's number (~150 people). When hunter-gatherers exist in such small groups, there is no need to specialize. Classes (such as a warrior class in your post) don't exist yet, as specialization only occurs after farming was invented. With a good balance of information (skill) symmetry between peoples, power mostly concentrates along social lines (who has respect) and not violent ones. This is because most people know how to fight and are themselves strong, making it difficult to truly physically resist the group. Hunter-gatherer reverse dominance hierarchies work because the group is small, everyone knows everyone, and there are constant interactions. Schools are large, semi-anonymous institutions where those accountability mechanisms are weak or absent. The persistence and reinvention of bullying cliques in schools is precisely what you'd predict if the egalitarian mechanisms of small-band life don't transfer to large modern institutions. Robin Hanson's own framework predicts that in small bands where dominance is hard to achieve physically, prestige-based status dominates.

Kongo Landwalker's Shortform

Kongo Landwalker10h40

In my personal experience the peak of AI was exactly a year ago. While others were discussing hallucinations and spiraling, my chats could stay useful for hundreds of messages and allow me to work with huge tasks. Back then ChatGPT could even understand a new mathematical notion I came up with and engage in discussion of the theorems.

But during the last year i was mentally tracking how long does a chat last before it starts hallucinating (saying nonsense, or generating the human part of the conversation, or starting to repeat itself, or to ignore recent co... (read more)

ryan_greenblatt's Shortform

ryan_greenblatt18h121

Sometimes people (at least on twitter/X) talk about how AI companies being publicly traded earlier is really great for power/wealth inequality. This quantitatively obviously doesn't make sense and there is a simple argument for this. To grow wealth by X companies must gain that much in additional (pre-dilution) valuation relative to the point of investment. To reduce inequality you need many, many trillions and companies just aren't that valuable (yet). I do agree it's helpful for a financial reduction of concentration of power if companies are eventually ... (read more)

Fabien Roger13h31

which I think is true

If you think markets are not pricing the singularity is true and >>$150T valuations are possible, why "This quantitatively obviously doesn't make sense"? (I read your sentence as "This is obviously not true", maybe you mean "This isn't obviously true"? Claude thinks the most natural interpretation is the former.)

9gbtw14h

The "permanent underclass" idea from the tweet seems to be mostly a concern of American white collar workers -- and so I imagine the claim here is more or less that this lets relatively wealthy American white collar workers who are vulnerable to being automated away buy some insurance against that. Post-IPO, a white collar worker with a couple of tens of thousands of dollars available can probably buy that insurance via leveraged positions in the stocks, thus escaping the "permanent underclass."

Fabien's Shortform

Fabien Roger2d5710

The main cost of safety is not always compute or money.

The cost of trying hard at safely building AI may not be easily measured by compute, money, or headcount. I expect that the main costs of AI safety will look more like a very long list of indirect costs (probably downstream of it becoming a bigger organizational priority).

Eventually this looks like indirect compute or money usage because almost everything will be powered by compute and money, but I think a lot of it won’t be part of the safety budget in balance sheets, and this may also not be the sor... (read more)

Richard_Ngo13h20

I broadly agree, and it's useful to have such a fleshed-out list.

Though note that once we're talking about highly non-fungible/non-commensurable effects, the term "costs" might be misleading. As a toy example, you might intuitively assume that needing someone else to put effort into safety costs "political capital". But suppose Franklin was right when he wrote "He that has once done you a kindness will be more ready to do you another, than he whom you yourself have obliged." Then the implicit assumptions behind talking in terms of political costs and capital break down.

Alex Gibson's Shortform

Alex Gibson13h30

One assumption I used to overlook with the linear representation hypothesis is that the same direction is assumed to represent the same concept independent of auxiliary context.

My mental model is increasingly that a direction in the residual stream need not have meaning independent of the interpretation provided by the MLP/attention layers. And even a single MLP layer can interpret the same direction differently depending on context, just via having a subset of neurons associated with each auxiliary context (and suppressing the neurons not related to said... (read more)

Dagon's Shortform

Dagon21h5-7

Consciousness isn't special.

My take on p-zombies, AI consciousness, and all related questions: sure, whatever. Anything that claims to be conscious is probably conscious. Qualia are a side effect of at least some kinds of compute, and there's not much reason to believe it's ONLY biological, and almost no reason to believe it's divorced from the capabilities.

I don't think consciousness is always similar to my own experiences, especially in terms of valence, tropism (pain/pleasure), or emotional inner experience (though I do recognize external similarity... (read more)

3nielsrolf20h

This glosses over roughly all of the interesting cases - does this apply to a python program that always prints "I am conscious"? If not, then does it apply to an LLM that was trained to output similar text? Do you think the reverse is also true - that most things that don't claim to be conscious is probably not conscious? (I suppose no since you grant animals consciousness) [...] You can do that but then it is roughly impossible to make moral progress, or have useful debates with people who don't already share your intuitions. In order for that to work, I think you need to appeal to some meta ethics principle. Utilitarianism appeals to a simplicity prior and a veil of ignorance kind of thing. Is there anything upstream of your assessment of moral patienthood?

Andrii Vasylenko14h10

a python program that always prints "I am conscious"

I think it would be more effective to consider the mental state of whoever wrote that program, not the program itself.

3Dagon19h

I don't find the python case interesting, but you're right - I should amend to "things which I find plausibly conscious are probably actually conscious". It needs to be complex enough that there are causal physical pathways which I can't decompose to a simple dismiss-able model. And it doesn't need to make the claim directly - other behaviors that I can infer consciousness from qualify. [...] It applies to human brains that have learned/been trained to generate those outputs. Why wouldn't it apply to other complicated general-purpose predictors? [...] Yes, I bite that bullet. Moral realism is misguided and incorrect. "moral progress" is refinement and update of my beliefs to make me more satisfied, and is discussion and alignment with other actors to come into agreement with my preferences (or optimal trade/compromise between our unresolved and unresolvable differences). At the root, it's all physics, including my and your preferences about what we'd like to perceive/experience.

agulaya24's Shortform

agulaya2414h10

I've found that through what ended up being one of the most rigorous processes of research, review, and writing that I have ever undertaken, I have realized new perspectives and a deeper understanding of what I am interested in doing with my life. There is a tendency to think this feeling of clarity is temporary, fleeting. Sometimes, having a dream means trying to prove it wrong for your own sanity, to verify you are working on something that still aligns. How many trials and tribulations must it go through until it has earned its place as part of your be... (read more)

wingspan's Shortform

OhadA14h30

If AI labs were nationalized by the US government for national security reasons, the USG might be in a more convenient position to enforce an AI pause. With or without nationalization we'd have to do the hard work of convincing the government to pause, but if it's nationalized then they have already swallowed one pill, which is strongly regulating AI and destroying the competition. From that state they'd only have to pause one big AGI project instead of many different competing projects at once.

(On the other hand, if we're in that state then the US-China race will likely intensify which can make things harder for global coordination)

jbash's Shortform

jbash1d471

Something seems to be really wrong with Claude Opus 4.8.

I like to test out new models on literary and poetic material. I used to send them the lyrics of Kate Bush's "The Kick Inside" and see what they could make of it, until they started just recognizing the song. Lately I've been using some text I wrote myself, with this prompt or something very close to it:

This is meant to introduce a novel and set the stage for some of its themes. It’s the first text in the book; at this point, a reader who hasn’t read any reviews or blurbs has only the title, which i

Linch14h40

Fwiw I tested Opus 4.8 XHigh on The Distaff Texts and it got a higher fraction of the plot than past models, including the obvious-to-LW-readers meta part that historically tripped up other models.

2jbash21h

I went ahead tried a run with "xhigh", which I think OpenRouter will translate to Anthropic's "max". Didn't seem to make a lot of difference. It didn't fixate as much, but it mostly seems to have just gotten more conservative about what it was willing to say, and it didn't catch anything new. Generally I've found that when models miss hints at the beginning, I have to steer them pretty hard conversationally if I want them to see those hints, so I'm guessing that more self-talk probably won't be helpful for most of them.

2Linch21h

That's so different from my use case for anything more substantive than a search query, even "High" feels too low for me most of the time!

gabeorosan's Shortform

gabeorosan2d304

Inspired by the recent Negation Neglect paper, I did some fine-tuning experiments with different prompts.

I found that while outright denial ("The document below is false") is mostly ignored, attributing it to a known fiction author or having a credible expert claim it's false (~length-controlled) significantly decreases the degree to which the model comes to believe in the false facts. Intuitively, I figured that framings that more closely mirrored what the LLM may have seen during training would be more readily understood, and beliefs updated more accurat... (read more)

Sheikh Abdur Raheem Ali18h10

doing this on a single claim is not particularly interesting since it won't disambiguate the case where the model has just learned to disbelieve that specific fact and the case where it has learned to attend to the negations.