All of Jackson Wagner's Comments + Replies

My thoughts about the story, perhaps interesting to any future reader trying to decipher the "mysterianism" here, a la these two analyses of sci-fi short stories, or my own attempts at exegeses of videogames like Braid or The Witness.  Consider the following also as a token of thanks for all the enjoyment I've recieved from reading Gwern's various analyses over the years.

... (read more)
8_dain_
[Reposting and expanding on my reply from HN] Regarding coal futures: more likely a reference to the quixotism of "selling coal to Newcastle"[1], and by extension to that famous Irish-American Quixote, Timothy Dexter, who indeed did so, and broke a miner's strike thereby. The similarities probably end there; Mr Rosier was clearly a literate man, and Mr Dexter decidedly was not. As for the final baseball anecdote, I think you've misread it. It is not from the point of view of the batter, but the catcher -- the anonymous narrator was a boy watching the game from the stands, and he caught the ball at the final moment. It is unarguably the 12 year old Mr Rosier; this would have been during his "youth gang" era, and he tells us that >Whiston is glaring daggers at me: I will suffer eternally for letting them down. The other guys look away at the stand ... -- Whiston is one of his boyhood friends, and prior to this incident Mr Rosier disappointed them in some way. Catching that ball was a singular moment of glory, on which he dwelled for the rest of his life, looming larger and larger in his mind until it could no longer be contained, but spilled out into a vast museum cataloguing everything connected to that instant. It was his "Rosebud". Everything about the war, Poland, etc is a red herring. He founded the museum precisely 30 years after the game, at the age of 42, possibly brought on by a midlife crisis. This is the hidden "atrocious or banal reality" behind the story, of the kind imagined by Borges and Casares that fateful evening in Buenos Aires. It's a funny inversion of Ezra Buckley's project from Tlön: while Buckley's ambition was the secret invention of an entire planet, the more humble Mr Rosier contented himself with the public preservation of an infinitesimal point in time and space (one which, per The Aleph, contains all other points in time and space, and through it they may be perfectly understood). Buckley's motivation was an explicit rebellion agains
3Thane Ruthenis
Kudos for going so in-depth on this. None of the speculations here seem convincing to me. I'd expect there to be a simple "key" that "unlocks" the story and clearly makes its overall meaning clear (see Suzanne Delage), and none of this feels like it fits. One element that struck with me, which you don't mention, is: To me, that evokes Laplace's demon, the idea that precisely learning the state of the universe at one moment would let you run it forwards or backwards arbitrarily[1]. Perhaps it'd make sense to take the Institute's mission seriously, and try to figure out what it is they're actually trying to do? (Though the Laplace's-demon angle on things makes the "too late" idea even less sensible: it's never too late, collecting the full knowledge of the state at any moment would let you unlock/anchor the whole history. So if something like this were the goal, re-centering the temporal locus on a day in a more information-rich era would be a much better approach.) 1. ^ This doesn't actually work under QM. Or, at least, doesn't work from inside an Everett branch, since the information needed to exactly calculate forward evolution is spread throughout the whole amplitude distribution.
2rsaarelm
Was it the title? I always thought Scott used the phrase as commentary on why he was posting the story, same as gwern is doing here. As in, he tried to clearly say "an omnipresent personal AI agent that observes your life and directly tells you the best way to act in every situation you encounter would be a bad thing because building up your own mind into being able to overcome challenging situations is necessary for a meaningful life", people didn't buy it, and then he went "okay, let's try this untitled short story to illustrate the idea". For this gwern thing though, I've no idea what the failed preceding non-mysterian attempt was.

I think that jumping straight to big-picture ideological questions is a mistake.  But for what it's worth, I tried to tally up some pros and cons of "ideologically who is more likely to implement a post-scarcity socialist utopia" here; I think it's more of a mixed bag than many assume.

I think when it comes to the question of "who's more likely to use AGI build fully-automated luxury communism", there are actually a lot of competing considerations on both sides, and it's not nearly as clear as you make it out.

Xi Jinping, the leader of the CCP, seems like kind of a mixed bag:

  • On the one hand, I agree with you that Xi does seem to be a true believer in some elements of the core socialist dream of equality, common dignity for everyone, and improved lives for ordinary people.  Hence his "Common Prosperity" campaign to reduce inequality,
... (read more)

@Tomás B.  There is also vastly less of an "AI safety community" in China -- probably much less AI safety research in general, and much less of it, in percentage terms, is aimed at thinking ahead about superintelligent AI.  (ie, more of China's "AI safety research" is probably focused on things like reducing LLM hallucinations, making sure it doesn't make politically incorrect statements, etc.)

  • Where are the chinese equivalents of the American and British AISI government departments?  Organizations like METR, Epoch, Forethought, MIRI, et cete
... (read more)

The four questions you ask are excellent, since they get away from general differences of culture or political system, and address the processes that are actually producing Chinese AI. 

The best reference I have so far is a May 2024 report from Concordia AI on "The State of AI Safety in China". I haven't even gone through it yet, but let me reproduce the executive summary here: 

The relevance and quality of Chinese technical research for frontier AI safety has increased substantially, with growing work on frontier issues such as LLM unlearning, mis

... (read more)

Thanks for this informative review!  (May I suggest that The Witness is a much better candidate for "this generation's Myst"!)

That's a good point -- the kind of idealized personal life coach / advisor Dario describes in his post "Machines of Loving Grace" is definitely in a sense a personality upgrade over Claude 3.7.  But I feel like when you think about it more closely, most of the improvements from Claude to ideal-AI-life-coach are coming from non-personality improvements, like:

  • having a TON of context about my personal life, interests, all my ongoing projects and relationships, etc
  • having more intelligence (including reasoning ability, but also fuzzier skills like social /
... (read more)

Semi-related: if I'm reading OpenAI's recent post "How we think about safety and alignment" correctly, they seem to announce that they're planning on implementing some kind of AI Control agenda.  Under the heading "iterative development" in the section "Our Core Principles" they say:

In the future, we may see scenarios where the model risks become unacceptable even relative to benefits. We’ll work hard to figure out how to mitigate those risks so that the benefits of the model can be realized. Along the way, we’ll likely test them in secure, controlled

... (read more)

Maybe this is obvious / already common knowledge, but I noticed that OpenAI's post seems to be embracing an AI control agenda for their future superalignment plans.  The heading "iterative development" in the section "Our Core Principles" says the following (emphasis mine):

It’s an advantage for safety that AI models have been growing in usefulness steadily over the years, making it possible for the world to experience incrementally better capabilities. This allows the world to engage with the technology as it evolves, helping society better understand

... (read more)

I enjoyed this post, which feels to me part of a cluster of recent posts pointing out that the current LLM architecture is showing some limitations, that future AI capabilities will likely be quite jagged (thus more complementary to human labor, rather than perfectly substituting for labor as a "drop-in remote worker"), and that a variety of skills around memory, long-term planning, agenticness, etc, seem like like important bottlenecks.

(Some other posts in this category include this one about Claude's abysmal Pokemon skills, and the section called "What I... (read more)

Improved personality is indeed a real, important improvement in the models, but (compared to traditional pre-training scaling) it feels like more of a one-off "unhobbling" than something we should expect to continue driving improved performance in the future.  Going from pure next-token-predictors to chatbots with RLHF was a huge boost in usefulness.  Then, going from OpenAI's chatbot personality to Claude's chatbot personality was a noticeable (but much smaller) boost.  But where do we go from here?  I can't really imagine a way for Anthropic to improve Claude's personality by 10x or 100x (whatever that would even mean).  Versus I can imagine scaling RL to improve a reasoning model's math skills by 100x.

1ScottWofford
Interesting point about personality improvements being a "one-off unhobbling" with diminishing returns. But I wonder if this reflects a measurement bias rather than an actual capability ceiling: we have clear benchmarks for evaluating math skills - it's easy to measure 100x improvement when a model goes from solving basic algebra to proving novel theorems. But how do we quantify personality improvements? There's a vast gap between "helpful but generic" and "perfectly attuned to individual users' needs, communication styles, and thinking patterns." I can imagine future models that feel like they truly understand me personally, anticipate my unstated needs, communicate in exactly my preferred style, and adapt their approach based on my emotional state - something far beyond current implementations. The lack of obvious metrics for these qualities doesn't mean the improvement ceiling is low, just that we're not good at measuring them yet. Thoughts? 

Seems like an easy way to create a less-fakeable benchmark would be to evaluate the LLM+scaffolding on multiple different games?  Optimizing for beating Pokemon Red alone would of course be a cheap PR win, so people will try to do it.  But optimizing for beating a wide variety of games would be a much bigger win, since it would probably require the AI to develop some more actually-valuable agentic capabilities.

It will probably be correct to chide people who update on the cheap PR win.  But perhaps the bigger win, which would actually justify such updates, might come soon afterwards!

Yeah -- just like how we are teaching LLMs to do math and coding by doing reinforcement learning on those tasks, it seems like we could just do a ton of RL on assorted videogames (and other agentic tasks, like booking a restaurant reservation online), to create reasoning-style models that have better ability to make and stick to a plan.

In addition to the literal reinforcement learning and gradient descent used for training AI models, there is also the more metaphorical gradient descent process that happens when hundreds of researchers all start tinkering w... (read more)

Note that this story has recieved a beautifully-animated video adaptation by Rational Animations!
 

Possibly look for other skills / career paths, besides math and computer science?  Glancing through 80,000 Hours' list:

- AI governance and policy -- I'm guessing that seeking out "policy people" will be a non-starter in Russia, either because it's dangerous or because there are fewer such people (not whole graduating classes at Harvard, etc, waiting to become the next generation of DC elites).

- AI safety technical research -- of course you are already thinking about this via IOM, IOI, etc. Others have mentioned trying to expand to LLM-specific competi... (read more)

5Mikhail Samin
* Yep, we've also been sending the books to winners of national and international olympiads in biology and chemistry. * Sending these books to policy-/foreign policy-related students seems like a bad idea: too many risks involved (in Russia, this is a career path you often choose if you're not very value-aligned. For the context, according to Russia, there's an extremist organization called "international LGBT movement"). * If you know anyone with an understanding of the context who'd want to find more people to send the books to, let me know. LLM competitions, ML hackathons, etc. all might be good. * Ideally, we'd also want to then alignment-pill these people, but no one has a ball on this. 

Satellites were also plausibly a very important military technology.  Since the 1960s, some applications have panned out, while others haven't.  Some of the things that have worked out:

  • GPS satellites were designed by the air force in the 1980s for guiding precision weapons like JDAMs, and only later incidentally became integral to the world economy.  They still do a great job guiding JDAMs, powering the style of "precision warfare" that has given the USA a decisive military advantage ever since 1991's first Iraq war.
  • Spy satellites were very
... (read more)

Maybe other people have a very different image of meditation than I do, such that they imagine it as something much more delusional and hyperreligious? Eg, some religious people do stuff like chanting mantras, or visualizing specific images of Buddhist deities, which indeed seems pretty crazy to me.

But the kind of meditation taught by popular secular sources like Sam Harris's Waking Up app, (or that I talk about in my "Examining The Witness" youtube series about the videogame The Witness), seems to me obviously much closer to basic psychology or rationali... (read more)

4Viliam
Thanks for answering my question directly in the second half. I find the testimonies of rationalists who experimented with meditation less convincing than perhaps I should, simply because of selection bias. People who have pre-existing affinity towards "woo" will presumably be more likely to try meditation. And they will be more likely to report that it works, whether it does or not. I am not sure how much should I discount for this, perhaps I overdo it. I don't know. A proper experiment would require a control group -- some people who were originally skeptical about meditation and Buddhism in general, and only agreed to do some exactly defined exercises, and preferably the reported differences should be measurable somehow. Otherwise, we have another selection bias, that if there are people for whom meditation does nothing, or is even harmful, they will stop trying. So at the end, 100% of people who tried will report success (whether real or imaginary), because those who didn't see any success have selected themselves out. I approve of making the "secular version of Buddhism", but in a similar way, we could make a "secular version of Christianity". (For example, how is gratitude journaling significantly different from thanking God for all his blessing before you go sleep?) And yet, I assume that the objection against "secular Christianity" on Less Wrong would be much greater than against "secular Buddhism". Maybe I am wrong, but the fact that no one is currently promoting "secular Christianity" on LW sounds like weak evidence. I suspect, the relevant difference is that for an American atheist, Christianity is outgroup, and Buddhism is fargroup. Meditation is culturally acceptable among contrarians, because our neighbors don't do it. But that is unrelated to whether it works or not. Also, I am not sure how secular the "secular Buddhism" actually is, given that people still go to retreats organized by religious people, etc. It feels too much for me to trust that s

I think there are many cases of reasonably successful people who often cite either some variety of meditation, or other self-improvement regimes / habits, as having a big impact on their success. This random article I googled cites the billionaires Ray Dalio, Marc Benioff, and Bill Gates, among others. (https://trytwello.com/ceos-that-meditate/)

Similarly you could find people (like Arnold Schwarzenegger, if I recall?) citing that adopting a more mature, stoic mindset about life was helpful to them -- Ray Dalio has this whole series of videos on "life pri... (read more)

4MondSemmel
Re: successful people who meditate, IIRC in Tim Ferriss' book Tools of Titans, meditation was one of the most commonly mentioned habits of the interviewees.

To compare to the obvious alternative, is the evidence for meditation stronger than the evidence for prayer? I assume there are also some religious billionaires and other successful people who would attribute their success to praying every day or something like that.

It feels sorta understandable to me (albeit frustrating) that OpenPhil faces these assorted political constraints.  In my view this seems to create a big unfilled niche in the rationalist ecosystem: a new, more right-coded, EA-adjacent funding organization could optimize itself for being able to enter many of those blacklisted areas with enthusiasm.

If I was a billionare, I would love to put together a kind of "completion portfolio" to complement some of OP's work.  Rationality community building, macrostrategy stuff, AI-related advocacy to try an... (read more)

There are actually a number of ways that you might see a permanently stable totalitarian government arise, in addition to the simplest idea that maybe the leader never dies:

https://80000hours.org/problem-profiles/risks-of-stable-totalitarianism/

I and perhaps other LessWrongers would appreciate reading your review (of any length) of the book, since lots of us loved HPMOR, the Sequences, etc, but are collectively skeptical / on the fence about whether to dive into Project Lawful.  (What's the best way to read the bizzare glowfic format?  What are the main themes of the book and which did you like best?  etc)

1Mo Putera
I thought this review was fine: https://recordcrash.substack.com/p/mad-investor-chaos-woman-asmodeus 
2Tapatakt
My opinion, very briefly: Good stuff: * Deception plotline * Demonstration of LDT in action * A lot of thought processes of smart characters * Convincing depictions of how people with very weird and evil ideology can have at least seemingly consistent worldview, be humans and not be completely insane. Stuff that might be good for some and bad for others: * It's Yudkowsky's work and it feels. Some people like the style of his texts, some don't. * Sex scenes (not very erotic and mostly talking) * Re-construction of Pathfinder game mechanics in setting * Math classes (not the best possible explanations, but not the worst either) * A lot of descriptions of "how everything works on dath ilan" Bad stuff: * It's isekai (it's bad if you're allergic to this genre) * It's very long * And in some places it could be shorter without losing anything (but, I think, nothing as egregious as school wars in HPMOR) (but if you don't appreciate the point about "thought processes of smart characters", then it could be much shorter without losing anything in most places)

I loved Project Lawful/Planecrash (not sure which is the actual title), but I do hesitate to recommend it to others. Not everyone likes their medium-core S&M with a side of hardcore decision theory, or vice-versa. It is definitely weirder than HPMOR.

Something that threw me off at first: it takes the mechanics of the adapted setting very literally (e.g. spell slots and saving throws are non-metaphorical in-universe Things). That's not normal for (good) game fanfiction. The authors make it work anyway -- perhaps because clear rules make it easier to prod... (read more)

7Double
Spoiler free again: Good to know there’s demand for such a review! It’s now on my todo list. To quickly address some of your questions: Pros of PL: If the premise I described above interests you, then PL will interest you. Some good Sequences-style rationality. I certainly was obsessed reading it for months. Cons: Some of the Rationality lectures were too long, but I didn’t mind much. The least sexy sex scenes. Because they are about moral dilemmas and deception, not sex. Really long. Even if you read it constantly and read quickly, it will take time (1.8 million words will do that). I really have to read some authors that aren’t Yud. Yud is great, but this is clearly too much of him, and I’m sure he’d agree. I read PL when it was already complete, so maybe I didn’t get the full experience, but there really wasn’t anything all that strange about the format (the content is another matter!). I can imagine that *writing * a glowfic would be a much different experience than writing a normal serialized work (ie dealing with your co-authors), but reading it isn’t very different from reading any other fiction. Look at the picture to see the POV, look at who’s the author if you’re curious, and read as normal. I’m used to books that change POV (though usually not this often). There are sometimes bonus tangent threads, but the story is linear. What problems do you have with the glowfic format? Main themes would require a longer post, but I hope this helps.
2RHollerith
My question is, Can I download an offline copy of it -- either text or spoken audio? The audio consists of 195 episodes, each of which can be individually downloaded, but can I get it as a single audio file (of duration 150 hours or so)?

This is nice!  I like seeing all the different subfields of research listed and compared; as a non-medical person I often just hear about one at a time in any given news story, which makes things confusing.

Some other things I hear about in longevity spaces:
- Senescent-cell-based theories and medicines -- what's up with these?  This seems like something people were actually trying in humans; any progress, or is this a dud?
- Repurposing essentially random drugs that might have some effect on longevity -- most famously the diabetes drug metformin (a... (read more)

7Abhishaike Mahajan
Thank you for reading!  Senescent-cell-based therapeutics feels like somewhat of a dead-end...senescence happens for a reason, and clearing out these cells have some second-order downsides. E.g., the inflammation caused by senescence is important for acute injury repair. I am less well-read on this area though! Metformin and rapamicyn are promising in the same way ozempic is promising; helping curtail metabolic problems helps a LOT of things, but it won't lead to dramatic changes in lifespan. Definitely in healthspan! But even there, nothing insane.  Imo, partial cellular reprogramming is the only real viable approach we have left, I'm kinda unsure what else the field has to offer if that ends up failing. 

Fellow Thiel fans may be interested in this post of mine called "X-Risk, Anthropics, & Peter Thiel's Investment Thesis", analyzing Thiel's old essay "The Optimistic Thought Experiment", and trying to figure out how he thinks about the intersection of markets and existential risk.

"Americans eat more fats and oils, more sugars and sweets, more grains, and more red meat; all four items that grew the most in price since 2003."

Nice to know that you can eat healthy -- fish, veggies, beans/nuts, eggs, fresh fruit, etc -- and beat inflation at the same time! (Albeit these healthier foods still probably have a higher baseline price. But maybe not for much longer!)

The linked chart actually makes red meat look fine (beef has middling inflation, and pork has actually experienced deflation), but beverages, another generally unhealthy food, a... (read more)

2Declan Molony
"Albeit these healthier foods still probably have a higher baseline price." Maybe in the short-term, but considering the lifetime consequences of unhealthy eating (e.g., atherosclerosis, heart disease, cancer, dementia---Dr. Peter Attia's stated 4 horsemen of death that account for 80% of death in the US---not to mention the emotional damage on your mood and potential productivity), the cost/benefit analysis seems heavily weighted in favor of eating healthier foods.

A thoughtful post! I think about this kind of stuff a lot, and wonder what the implications are. If we're more pessimistic about saving lives in sub-saharan africa, should we:

  1. promote things like lead removal (similar evidence-backed, scalable intervention as bednets, but aimed more directly at human capital)?
  2. promote things like charter cities (untested crazy longshot megaproject, but aimed squarely at transformative political / societal improvements)?
  3. switch to bednet-style lifesaving charities in South Asia, like you mention?
  4. keep on trucking with ou
... (read more)
2vaishnav92
I posted this on the EA forum a couple of weeks ago - https://forum.effectivealtruism.org/posts/7WKiW4fTvJMzJwPsk/adverse-selection-in-minimizing-cost-per-life-saved No surprise that people on the forum seem to think #4 is the right answer (although they did acknowledge this is a valid consideration). But a lot of it was "this is so cheap that this is probably still the right answer" and "we should be humble and not violate the intuition people have that all lives are equal". 

Re: your point #2, there is another potential spiral where abstract concepts of "greatness" are increasingly defined in a hostile and negative way by partisans of slave morality.  This might make it harder to have that "aspirational dialogue about what counts as greatness", as it gets increasingly difficult for ordinary people to even conceptualize a good version of greatness worth aspiring to.  ("Why would I want to become an entrepeneur and found a company?  Wouldn't that make me an evil big-corporation CEO, which has a whiff of the same f... (read more)

Feynman is imagining lots of components being made with "hand tools", in order to cut down on the amount of specialized machinery we need. So you'd want sophisticated manipulators to use the tools, move the components, clean up bits of waste, etc. Plus of course for gathering raw resources and navigating Canadian tundra. And you'd need video cameras for the system to look at what it's doing (otherwise you'd only have feed-forward controls in many situations, which would probably cause lots of cascading errors).

I don't know how big a rasberry pi would be if it had to be hand-assembled from transistors big enough to pick up individually. So maybe it's doable!

1[anonymous]
.

idk, you still have to fit video cameras and complex robotic arms and wifi equipment into that 1m^3 box, even if you are doing all the AI inference somewhere else!  I have a much longer comment replying to the top-level post, where I try to analyze the concept of an autofac and what an optimized autofac design would really look like.  Imagining a 100% self-contained design is a pretty cool intellectual exercise, but it's hard to imagine a situation where it doesn't make sense to import the most complex components from somewhere else (at least initially, until you can make computers that don't take up 90% of your manufacturing output).

1[anonymous]
.

This was a very interesting post.  A few scattered thoughts, as I try to take a step back and take a big-picture economic view of this idea:

What is an autofac?  It is a vastly simplified economy, in the hopes that enough simplification will unlock various big gains (like gains from "automation").  Let's interpolate between the existing global economy, and Feynman's proposed 1-meter cube.  It's not true that "the smallest technological system capable of physical self-reproduction is the entire economy.", since I can imagine many potentia... (read more)

9Carl Feynman
Wow, I think that comment is as long as my original essay.  Lots of good points.  Let me take them one by one. The real motivation for the efficiency-impairing simplifications is none of size, cost or complexity.  It is to reduce replication time.  We need an Autofac efficient enough that what it produces is higher value than what it consumes.  We don't want to reproduce Soviet industry, much of which processed expensive resources into lousy products worth less than the inputs.  Having achieved this minimum, however, the goal is to allow the shortest possible time of replication.  This allows for the most rapid production of the millions of tons of machinery needed to produce massive effects. Consider that the Autofac, 50 kg in a 1 m^3, is modeled on a regular machine shop, with the machinist replaced by a robot.  The machine shop is 6250 kg in 125 m^3.  I just scale it down by a factor of 5, and thereby reduce the duplication time by a factor of 5.  So it duplicates in 5 weeks instead of 25 weeks.  Suppose we start the Autofac versus the robot machine shop at the same time.  After a year, there are 1000 Autofacs versus 4 machine shops; or in terms of mass, 50,000 kg of Autofac and 25,000 kg of machine shop.  After two years, 50,000,000 kg of Autofac versus 100,000 kg of machine shop.  After 3 years, it's even more extreme.  At any time, we can turn the Autofacs from making themselves to making what we need, or to making the tools to make what we need.  The Autofac wins by orders of magnitude even if it's teeny and inefficient, because of sheer speed. That's why I picked a one meter cube.  I would have picked a smaller cube, that reproduced faster, but that would scale various production processes beyond reasonable limits.  I didn't want to venture beyond ordinary machining into weird techniques only watchmakers use. This is certainly a consideration.  Given the phenomenal reproductive capacity of the Autofac, there's an enormous return to finishing design as qu

1950s era computers likely couldn't handle the complex AI tasks imagined here (doing image recognition; navigating rough Baffin Island terrain, finishing parts with hand tools, etc) without taking up much more than 1 meter cubed.

1[anonymous]
.

Socialism / communism is about equally abstract as Georgism, and it certainly inspired a lot of people to fight! Similarly, Republican campaigns to lower corporate tax rates, cut regulations, reduce entitlement spending, etc, are pretty abstract (and often actively unpopular when people do understand them!), but have achieved some notable victories over the years. Georgism is similar to YIMBYism, which has lots of victories these days, even though YIMBYism also suffers from being more abstract than conspiracy theories with obvious villains about people "... (read more)

2mako yass
Maybe "abstract" was the wrong word. Communism and minarchy both have very simple visceral moral impulses supporting them. Fairness/equality vs liberty/choice. It's possible to get a person into a state where they feel one pole so intensely that they will be willing to fight against someone fighting earnestly for the other pole (right? But I'm not sure there's actually been a civil war between communists and minarchists, it's usually been communists vs monarchists/nationalists) For grey civics, I don't know what the unifying principle is. Commitment to growth? Progress? Hey, maybe that's it. I've been considering the slogan "defending zoning isn't compatible with calling yourself a progressive. If you believe in urban zoning you don't believe in progress." Progress seems to require meritocracy, rewarding work in proportion to its subjective EV or capricious outcomes, distributing rewards unevenly, and progress comes with a duty to future generations that minarchists might not like very much, but at least in tech, people seem alright with that. On the left, the tension mostly comes out of earnest disbelief, it's not intuitive that progress is real. For most of our evolutionary history it wasn't real, and today it happens only on the scale of years, and its every step is unprecedented. But how would we resolve the tension with humanism. I guess e/acc is the faction within the grey tribe who don't try to resolve that tension, they lean into it, they either explicitly reject the duty to defend the preferences of present humanity against those aspects of progress that threaten it, or they find reasons to downplay the immanence of those threats. The other faction has to sit and listen while Hanson warns them about the inevitability of absolute cultural drift, and I don't think we know what to say to that.

Future readers of this post might be interested this other lesswrong post about the current state of multiplex gene editing: https://www.lesswrong.com/posts/oSy5vHvwSfnjmC7Tf/multiplex-gene-editing-where-are-we-now

Future readers of this blog post may be interested in this book-review entry at ACX, which is much more suspicious/wary/pessimistic about prion disease generally:

  • They dispute the idea that having M/V or V/V genes reduces the odds of getting CJD / mad cow disease / etc.
  • They imply that Britain's mad cow disease problem maybe never really went away, in the sense that "spontaneous" cases of CJD have quadrupled since the 80s, so it seems CJD is being passed around somehow?

https://www.astralcodexten.com/p/your-book-review-the-family-that

What kinds of space resources are like "mice & cheese"?  I am picturing civilizations expanding to new star systems mostly for the matter and energy (turn asteroids & planets into a dyson swarm of orbiting solar panels and supercomputers on which to run trillions of emulated minds, plus constructing new probes to send onwards to new star systems).

re: the Three Body Problem books -- I think the book series imagines that alien life is much, much more common (ie, many civilizations per galaxy) than Robin Hanson imagines in his Grabby Aliens hypot... (read more)

2JBlack
After even the first million years as slow as 0.1c, the galaxy is full and it's time to go intergalactic. A million years is nothing in the scale of the universe's age. When sending a probe millions of light years to other galaxies, the expense of 0.999c probes start to look more useful than 0.8c ones, saving hundreds of thousands of years. Chances are that it wouldn't just be one probe either, but billions of them seeding each galaxy within plausible reach. Though as with any discussion about these sorts of things, we have no idea what we don't know about what a civilization a million years old might achieve. Discussions of relativistic probes are probably even more laughably primitive than those of using swan's wings to fly to the abode of the Gods.

Yes, it does have to be fast IMO, but I think fast expansion (at least among civilizations that decide to expand much at all) is very likely.

Of course the first few starships that a civilization sends to colonize the nearest stars will probably not be going anywhere near the speed of light.  (Unless it really is a paperclips-style superintelligence, perhaps.)  But within a million years or so, even with relatively slow-moving ships, you have colonized thousands of solar systems, built dyson swarms around every star, have a total population in the... (read more)

6jmh
Maybe, but I have to also put this all in a somewhat different frame. Is the universe populated by birds or mice? Are the resources nice ground full or worms or perhaps dangerous traps with the cheese we want?  So if we're birds and the universe resources are worms, maybe a race. If we're all mice and resources are those dangerous traps with cheese, well, the old saying "The early bird might get the worm but the second mouse gets the cheese." In a universe populated by mice & cheese, civilation expansion may well be much slower and measured. Perhpas we can add one of the thoughts from the Three Body Problem series -- advertising your civilation in the universe might be a sure way to kill yourself. Possibly fits with the Grabby Aliens thought but would argue for a different type of expansion patter I would think. That, and I'm not sure how the apparent solution ot energy problems (apparenly a civilization has no engery problem so accelleration and decellerations costs don't really matter) impacts a desire for additional resources. And if the energy problem is not solved then we need to know the cost curves for accelleration and decelleration to optimize speed in that resource search/grab.

I think part of the "calculus" being run by the AI safety folks is as follows:

  1. there are certainly both some dumb ways humanity could die (ie, AI-enabled bioweapon terrorism that could have easily been prevented by some RLHF + basic checks at protein synthesis companies), as well as some very tricky, advanced ways (AI takeover by a superintelligence with a very subtle form of misalignment, using lots of brilliant deception, etc)

  2. It seems like the dumber ways are generally more obvious / visible to other people (like military generals or the median vote

... (read more)

re: your comments on Fermi paradox -- if an alien super-civilization (or alien-killing AI) is expanding in all directions at close to the speed of light (which you might expect a superintelligence to do), then you mostly don't see them coming until it's nearly too late, since the civilization is expanding almost as fast as the light emitted by the civilization. So it might look like the universe is empty, even if there's actually a couple of civilizations racing right towards you!

There is some interesting cosmological evidence that we are in fact living i... (read more)

4faul_sname
"Close to the speed of light" has to be quite close to the speed of light for that argument to hold (at 0.8c about half of the volume in the light cone of an expanding civilization is outside of that civilization's expansion front).

[spoilers for minor details of later chapters of the book] Isn't the book at least a little self-aware about this plot hole? If I recall correctly, the book eventually reveals (rot13 from here on out)...

gung gur fcnpr cebtenz cyna jnf rffragvnyyl qrfvtarq nf n CE fghag gb qvfgenpg/zbgvingr uhznavgl, juvpu vaqrrq unq ab erny cebfcrpg bs jbexvat (gbb srj crbcyr, gbb uneq gb qbqtr zrgrbef, rirelguvat arrqf gb or 100% erhfrq, rgp, yvxr lbh fnl). Juvyr gur fcnpr cebtenz jnf unccravat, bgure yrff-choyvp rssbegf (eryngrq gb ahpyrne fhoznevarf, qvttvat haqreteb... (read more)

2Yair Halberstadt
Vg pna'g or gur pnfr gung jnf gur gehyr rssbeg, fvapr gur cerfvqrag pubfr gb tb gb fcnpr vafgrnq bs gur bgure rssbeg.

In addition to the researchy implications for topics like deception and superpersuasion and so forth, I imagine that results like this (although, as you say, unsuprising in a technical sense) could have a huge impact on the public discussion of AI (paging @Holly_Elmore and @Joseph Miller?) -- the general public often seems to get very freaked out about privacy issues where others might learn their personal information, demographic characteristics, etc.

In fact, the way people react about privacy issues is so strong that it usually seems very overblown to me... (read more)

1eggsyntax
Agreed. I considered releasing a web demo where people could put in text they'd written and GPT would give estimates of their gender, ethnicity, etc. I built one, and anecdotally people found it really interesting. I held off because I can imagine it going viral and getting mixed up in culture war drama, and I don't particularly want to be embroiled in that (and I can also imagine OpenAI just shutting down my account because it's bad PR). That said, I feel fine about someone else deciding to take that on, and would be happy to help them figure out the details -- AI Digest expressed some interest but I'm not sure if they're still considering it.
1Joseph Miller
Nice idea. Might try to work it into some of our material.

So, sure, there is a threshold effect in whether you get value from bike lanes on your complex journey from point A to point G.  But other people throughout the city have different threshold effects:

  • Other people are starting and ending their trips from other points; some people are even starting and ending their trip entirely on Naito Parkway.
  • People have a variety of different tolerances for how much they are willing to bike in streets, as you mention.
  • Even people who don't like biking in streets often have some flexibility.  You say that you pers
... (read more)

Nice; Colorado recently passed a statewide law that finally does away with a similar "U+2" rule in my own town of Fort Collins (as well as other such rules in Boulder and elsewhere). To progress!

I don't understand this post, because it seems to be parodying Anthropic's Responsible Scaling Policies (ie, saying that the RSPs are not sufficient), but the analogy to nuclear power is confusing since IMO nuclear power has in fact been harmfully over-regulated, such that advocating for a "balanced, pragmatic approach to mitigating potential harms from nuclear power" does actually seem good, compared to the status quo where society hugely overreacted to the risks of nuclear power without properly taking a balanced view of the costs vs benefits.

Maybe you c... (read more)

I will definitely check out that youtube channel!  I'm pretty interested in mechanism design and public-goods stuff, and I agree there are a lot of good ideas there.  For instance, I am a huge fan of Georgism, so I definitely recognize that going all-in on the "libertarian individualist approach" is often not the right fit for the situation!  Honestly, even though charter cities are somewhat an intrinsically libertarian concept, part of the reason I like the charter city idea is indeed the potential for experimenting with new ways to manage ... (read more)

Yup, there are definitely a lot of places (like 99+% of places, 99+% of the time!) which aren't interested in a given reform -- especially one as uniqely big and experimental as charter cities.  This is why in our video we tried to focus on political tractability as one of the biggest difficulties -- hopefully we don't come across as saying that the world will instantly be tiled over with charter cities tomorrow!  But some charter cities are happening sometimes in some places -- in addition to the examples in the video, Zambia is pretty friendly ... (read more)

Thanks, this is exciting and inspiring stuff to learn about!

I guess another thing I'm wondering about, is how we could tell apart genes that impact a trait via their ongoing metabolic activities (maybe metabolic is not the right term... what I mean is that the gene is being expressed, creating proteins, etc, on an ongoing basis), versus genes that impact a trait via being important for early embryonic / childhood development, but which aren't very relevant in adulthood.  Genes related to intelligence, for instance, seem like they might show up with po... (read more)

3GeneSmith
Yes, this is an excellent question. And I think it's likely we could (at least for the brain) thanks to some data from this study that took brain biopsies from individuals of varying stages of life and looked at the transcriptome of cells from different parts of the brain. My basic prior is that the effect of editing is likely to be close to the same as if you edited the same gene in an embryo iff the peak protein expression occurs in adulthood. Though there aren't really any animal experiments that I know of yet which look at how the distribution of effect sizes vary by trait and organ.

Is there a plausible path towards gene therapies that edit dozens, hundreds, or thousands of different genes like this? I thought people were worried about off-target errors, etc? (Or at least problems like "you'll have to take 1000 different customized doses of CRISPR therapy, which will be expensive".) So my impression is that this kind of GWAS-inspired medicine would be most impactful with whole-genome synthesis? (Currently super-expensive?)

To be clear I agree with the main point this post is making about how we don't need animal models, etc, to do medicine if we have something that we know works!

7GeneSmith
Yes, there is. I've been working on a post about this for the last few months and hope to post something much more comprehensive soon. Off-targets are a potential issue though they're less of an issue if you target non-coding regions that aren't directly translated into proteins. The editing tools have also improved a lot since the original CRISPR publications back in 2012. Base editors and prime editors have indel rates like 50-300x lower than original CRISPR. Base editors and prime editors can do simultaneous edits in the same cell. I've read a paper where the authors did 50 concurrent base edits (though it was in a cell type that is easier than average to edit). Scaling concurrent editing capabilities is the very first thing I want to focus on. Also, if your delivery vector doesn't trigger an adaptive immune response (or end up being toxic for some other reason), you can redose someone a few times and make new edits with each round. If we can solve delivery issues, the dosing would be as simple as giving someone an IV injection. I'm not saying these are simple problems. Solving all of them is going to be hard. But many of the steps have already been done independently in one research paper or another. No. You don't need to be able to synthesize a genome to make any of this work. You can edit the genome of a living person.

(this comment is kind of a "i didn't have time to write you a short letter so I wrote you a long one" situation)

re: Infowar between great powers -- the view that China+Russia+USA invest a lot of efforts into infowar, but mostly "defensively" / mostly trying to shape domestic opinion, makes sense.  (After all, it must be easier to control the domestic media/information lansdscape!)  I would tend to expect that doing domestically-focused infowar stuff at a massive scale would be harder for the USA to pull off (wouldn't it be leaked? wouldn't it be ... (read more)

(Copies from EA Forum for the benefit of lesswrongers following the discussion here)

Definitely agree that empathy and other social feelings provide indirect evidence for self-awareness (ie, "modeling stuff about yourself" in your brain) in a way that optimism/pessimism or pain-avoidance doesn't.  (Although wouldn't a sophisticated-enough RL circuit, interacting with other RL circuits in some kind of virtual evolutionary landscape, also develop social emotions like loyalty, empathy, etc?  Even tiny mammals like mice/rats display sophisticated soci... (read more)

1Mikhail Samin
I appreciate this comment. Qualia (IMO) certainly is "information processing": there are inputs and outputs. And it is a part of a larger information-processing thing, the brain. What I'm saying is that there's information processing happening outside of the qualia circuits, and some of the results of the information processing outside of the qualia circuits are inputs to our qualia.  Well, how do you know that visual information processing produces qualia? You can match when algorithms implemented by other humans' brains to algorithms implemented by your brain, because all of you talk about subjective experience; how do you, inside your neural circuitry, make an inference that a similar thing happens in neurons that just process visual information? You know you have subjective experience, self-evidently. You can match the computation run by the neural circuitry of your brain to the computation run by the neural circuitry of other humans: because since they talk about subjective experience, you can expect this to be caused by similar computation. This is valid. Thinking that visual information processing is part of what makes qualia (i.e., there's no way to replace a bunch of your neurons with something that outputs the same stuff without first seeing and processing something, such that you'll experience seeing as before) is something you can make theories about but is not a valid inference, you don't have a way of matching the computation of qualia to the whole of your brain. And, how can you match it to matrix multiplications that don't talk about qualia, did not have evolutionary reasons for experience, etc.? Do you think an untrained or a small convolutional neural network experiences images to some extent, or only large and trained? Where does that expectation come from? I'm not saying that qualia is solved. We don't yet know how to build it, and we can't yet scan brains and say which circuits implement it. But some people seem more confused than warranted

Why would showing that fish "feel empathy" prove that they have inner subjective experience?  It seems perfectly possible to build a totally mechanical, non-conscious system that nevertheless displays signs of empathy.  Couldn't fish just have some kind of built-in, not-necessarily-conscious instinct to protect other fish (for instance, by swimming together in a large school) in order to obtain some evolutionary benefit?

Conversely, isn't it possible for fish to have inner subjective experience but not feel empathy?  Fish are very simple crea... (read more)

0Mikhail Samin
Both (modeling stuff about others by reusing circuits for modeling stuff about yourself without having experience; and having experience without modelling others similarly to yourself) are possible, and the reason why I think the suggested experiment would provide indirect evidence is related to the evolutionary role I consider qualia to possibly play. It wouldn't be extremely strong evidence and certainly wouldn't be proof, but it'd be enough evidence for me to stop eating fish that has these things. The studies about optimistic/pessimistic behaviour tell us nothing about whether these things experience optimism/pessimism, as they are an adaptation an RL algorithm would implement without the need to implement circuits that would also experience these things, unless you can provide a story for why circuitry for experience is beneficial or a natural side effect of something beneficial. One of the points of the post is that any evidence we can have except for what we have about humans would be inderect, and people call things evidence for confused reasons. Pain-related behaviour is something you'd see in neural networks trained with RL, because it's good to avoid pain and you need a good explanation for how exactly it can be evidence for qualia. (Copied from EA Forum)

Hi Trevor!  I appreciate this thread of related ideas that you have been developing about intelligence agencies, AI-augmented persuasion techniques, social media, etc.

  • It seems important to "think ahead" about how the power-struggle over AI will play out as things escalate to increasingly intense levels, involving eg national governments and militaries and highly-polarized political movements and etc.
  • Obviously if some organization was hypercompetent and super-good at behind-the-scenes persuasion, we wouldn't really know about it!  So it is hard to
... (read more)

Thanks for putting the work into this attempt to falsify my model! I read all of it, but I will put work into keeping my response concise and helpful.

If the CIA (or other entities affiliated with the US government, including tech companies being pressured by the government) is so good at persuasion ops, why are there so many political movements that seem to go against the CIA's interests?

  1. Quoting ChristianKI quoting Michael Vassar: "We didn't have an atomic war as people expected after WWII, but we had an infowar and now most people are like zombies when it
... (read more)
Load More