ryan_b

Replying toStone Age Billionaire Can't Words Good

I agree on all three counts, but what I am talking about is the rhetorical strategy for trying to communicate the belief that it is very important in the long term to let people keep their stuff, to people who are proposing to take some people's stuff right now.

I don't think bad long term consequences would be hard to communicate in this instance (though it would not be easy in the middle of a protest). For example, I expect almost everyone who supports a wealth tax to also oppose the idea of corporate personhood; but these rulings started showing up in the 1800s, and the Citizens United decision showed up in 2010.

There's a bit of line to walk so as to not misrepresent what OP is believes, but I feel like establishing a link between longer-term bad effects that billionaire tax supporters understand might be as simple as saying "Citizens United was in 2010 and. . ." gestures at things in general

Replying toStone Age Billionaire Can't Words Good

ryan_b4d

Stone Age Billionaire Can't Words Good

that we must stand true to the Code and allow those billionaires to retain the wealth if we want to remain a Just and Wealthy society.

I think this right here is the crux of it. I doubt anyone who supports a wealth tax believes we live in a just society, and I expect them all to believe that the behavior of billionaires actively suppresses our wealth in the sense that you mean it. It looks, to them, like our current crop of billionaires are not honorably wealthy men, but trying very hard to become the new feudal lords themselves.

Protecting the Code is incredibly important NOT because it serves some billionaires, but because

... (read more)

Replying toPost-AGI Economics As If Nothing Ever Happens

ryan_b4d

Post-AGI Economics As If Nothing Ever Happens

Excellent post, strong upvote. You've done a great job articulating what I felt as basically just a twisting of the guts whenever I read economic analysis of the idea of AGI. Tackling the problem head-on:

Model AI as a firm directly: I believe AI straightforwardly breaks the usefulness of the capital-labor distinction. The central crux for me is the extent to which the AI could perform the knowledge work of corporate management; I claim it doesn't matter for economic purposes that the source of the decisions is an abstract machine the company owns (or rents), what matters for economic purposes is the level at which the decisions are made. If AI makes the... (read more)

Replying toOn Dwarkesh Patel’s Podcast With Richard Sutton

ryan_b4mo

On Dwarkesh Patel’s Podcast With Richard Sutton

It feels to me like Sutton is too deep inside the experiential learning theory. When he says there is no evidence for imitation, this only makes sense if you imagine it strictly according to the RL theory he has in mind. He isn't applying the theory to anything; he is inside the theory and interpreting everything according his understanding of it.

It did feel like there was a lot of talking past one another when Dwarkesh was clearly talking about the superintelligent behaviors everyone is interested in (doing science, math, and engineering) as his model for intelligence, and Sutton is blowing all of this off only to articulate quite late in the game... (read more)

Replying toI have decided to stop lying to Americans about 9/11

ryan_b5mo

I have decided to stop lying to Americans about 9/11

Good job saying a brave thing; on a US-based site with a plurality US membership, this was a risk. Well done.

Out of curiosity, how often do conversations about 9/11 come up? For the most part, we don't discuss it that much among ourselves except during the anniversary, although I do make note that it was just a couple of weeks ago and indeed the traditional observance is literally just to talk about where we were and what we were doing at the time, which precisely when the observation about cheering would come up.

It may or may not surprise you that while there were basically no rooms cheering in the US, there was... (read more)

Replying toIABIED Review - An Unfortunate Miss

ryan_b5mo

IABIED Review - An Unfortunate Miss

Strong upvote, I appreciate the inside-view context that you have from publishing a similar book. I bought it as a result of this review.

I cannot, alas, promise a side-by-side review. However, there are a couple of questions I am primed to look for, foremost among them right now: how much detail is invested in identifying the target audience? The impression I am getting so far is that it has been approximately defined as not us, but a lot of complaints seem to turn on this question. I see a lot of discussion about laymen but that's an information level, not a target audience. I don't know if I have seen much discussion of the target audiences at all outside of the AI policy area, come to think of it.

ryan_b5moQuick Take

The important information I should take from a strong trend is an axis, or a dimension, rather than the default takeaway of a direction.

I have the colloquial habit of talking about a trend as a direction, leaning on the implicit metaphor of physical space for whatever domain in which the trend appears. I've only just today started to realize that, while I am pretty sure the physical space (or, probably, geography and maps) metaphor is why I speak that way, there's no reason not to lean into it as an explicit description within the abstract space for the domain. By this I mean that whatever it is we are talking about (the... (read more)

Replying toUnderdog bias rules everything around me

ryan_b6mo

Underdog bias rules everything around me

Boiling this down for myself a bit, I want to frame this as a legibility problem: we can see our own limitations, but outsiders successes are much more visible than their limitations.

Replying toYudkowsky on "Don't use p(doom)"

ryan_b6mo

Yudkowsky on "Don't use p(doom)"

I'm inclined to look at the blunt limitations of bandwidth on this one. The first hurdle is that p(doom) can pass through tweets and shouted conversations in bay area house parties.

Replying toYudkowsky on "Don't use p(doom)"

ryan_b6mo

Yudkowsky on "Don't use p(doom)"

I also think he objects to putting numbers on things, and I also avoid doing it. A concrete example: I explicitly avoid putting numbers on things in LessWrong posts. The reason is straightforward - if a number appears anywhere in the post, about half of the conversation in the comments will be on that number to the exclusion of the point of the post (or the lack of one, etc). So unless numbers are indeed the thing you want to be talking about, in the sense of detailed results of specific computations, they are positively distracting from the rest of the post for the audience.

I focused on the communication aspect in my... (read more)

Near term discussions need something smaller and more concrete than AGI

ryan_b

Motivation

I want a more concrete concept than AGI^[1] to talk and write with. I want something more concrete because I am tired of the costs associated with how big, inferentially distant, and many-pathed the concept of AGI is, which makes conversation expensive. Accounting for the bigness, inferential distance, and many-pathed-ness is very important for dealing with AGI properly - I believe they are causal in the risks AGI poses - but they drive a very high communication burden. By this I mean things like distinguishing between different meanings, summarizing a bunch of previous discussions, etc. This burden does not seem to have reduced much if at all over time, and I don't expect... (read 1711 more words →)

SB 1047 gets vetoed

ryan_b

Just what it says on the tin. Covered most everywhere, but I found the quote in this Reuters article stuck out to me the most:

Newsom said the bill "does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data" and would apply "stringent standards to even the most basic functions — so long as a large system deploys it."

Emphasis mine. So the governor says he vetoed it for the exact reason that it would have been a good law, in my view.

If I ask an LLM to think step by step, how big are the steps?

ryan_b

I mean big in terms of number of tokens, and I am thinking about this question specifically in the context of training windows vs context windows. This question is inspired by an Andrew Mayne tweet covered in AI #80: Never Have I Ever:

Most AI systems are trained on less than 2,000 words per sample. They can generalize across all of its training but no system to date trains on 80,000 words per sample. This will change with more memory and processing power. When that happens AI will be able to actually “read” entire novels and understand story structure on a fundamental level.

Gwern says context windows solve this. Pretty sure I am behind... (read more)

Special Purpose Permit Companies

How many kinds of environmental and/or building permits could be gotten by company A, and then used for company B's project, after company B buys company A? For example, could we form a special purpose company to pursue navigating all the NEPA stuff for building a power plant at a particular site, and then sell that company to a power company, who could then begin construction immediately?

Consider real estate developers: one company buys the land and develops it, which means do all the construction, and then sells it to other companies who use the buildings.

I propose an extension of this model, where one company will buy the land and... (read more)

I wonder how feasible it would be to make something that would allow people to coordinate for consumption like unions for bargaining, parties for politics, or corporations for production. In particular I have in mind targeting entertainment and social media consumption because of the prevalence of recommendation algorithms here.

The intuition is that these algorithms are political computations: who gets the advertising traffic, what type of content is in demand, which consumers are catered to.

The thing being coordinated on is the consumer behavior; namely what people click on.

Examples:

like a union, but instead of collective bargaining to sell labor, it is collective behavioring^[1] to consume content.
like the old website Massdrop, but instead of coordinated buying

... (read more)

I read a story once about a billionaire who managed to flout a zoning limitation by building himself an absurdly huge high-rise single apartment. Once that was done, he waited a bit, and either using a loophole that already existed or one he finagled into the law, he added a bunch of walls inside is personal mega-apartment to and then sold off all these new units as regular apartments.

Now I think of this in the context of the American dredging and shipping laws. There's a lot of rules surrounding dredges and shipping vessels, but there are a lot fewer surrounding luxury items like yachts. So imagine a group of investors building a... (read more)

Do you have a satisfactory workflow for learning about a line of research using GPT4, Claude, etc?

ryan_b

I'm trying to learn about plasma separation techniques, and I just stumbled on a line of papers that don't seem particularly connected to the other lines of research I was looking into, and I would like to quickly get a sense of what it says without having to go through everything the old fashioned way.

What I wanted to do was use Claude or GPT4 to submit a pile of these papers (the recent paper I found, a big chunk of its bibliography, and a chunk of those bibliographies) and then try asking questions about the body of the research in a way similar to how people do it with books. I have... (read more)

My simple model for Alignment vs Capability

ryan_b

I have a simple model of the alignment vs. capabilities question. I am writing it down because after chewing on Michael Nielson's post about existential risk from AI, I thought it was great but was unsatisfied with with the idea an alignment dilemma. I wasn't the only person to feel that way, but neither was I able to find any satisfactory source describing the (tension? dichotomy? tradeoff?). The real rub of Nielsen's dilemma for me is that it was expressed in terms of accelerationism, which is a form of one-dialism, and one-dialism is wrong.

I note there is nothing actually new here; I am quite sure these considerations were all covered, and more... (read 1940 more words →)

Does compression research have any analogue of training behavior in ML research?

Watched a talk by Ilya Sutskever about generalization, and during the course of it he covered the equivalence of prediction and compression, and then further into maximum likelihood. The conversation was motivated by how thinking about this was a helpful perspective for him since 2016 or so.

In the talk he makes an analogy to Kolmogorov complexity, and then extends it so that conditional Kolmogorov complexity can be thought of as Kolmogorov compression and uses that when talking about why unstructured learning might work.

At 38:30 there's a question/comment about where the Kolmogorov analogy breaks down, which was that in Kolmogorov the order... (read more)

Assuming LK99 or similar: how to accelerate commercialization?

ryan_b

Tabling the question of whether LK99 itself is a room temperature superconductor, I wonder about how we would commercialize it assuming it (or something similar) is, as rapidly as possible. I was reading Ben Reinhardt's Metalessons from the LK-99 Saga post, which raised the point that it normally takes a few decades for an invention to commercially mature. This is consistent with the perspective from Gordon's The Rise and Fall of American Growth (summary), and Smil's Creating the Twentieth Century.

First, I would like the benefits of widespread superconduction sooner than 20 years. Second, given the US/EU's moderate-to-severe distaste for capital-intensive work these paste couple of decades, I think there's a too-high-for-sanity risk... (read more)

Saw a YouTube video by a guy named Michael Penn about why there is no 3 dimensional equivalent of the complex numbers. It's going through an abridged version of the mathematical reasons and I was able to follow along until we got a point where he showed that it would have to commute ix with xi, which contradicts an initial required claim that ix does not commute with xi.

This is not satisfying to me intuitively. The thing that bothers me is that I can accept the argument that the definition is incoherent, but that doesn't show me why we can't get there using some different claim or procedure. Here's what I came... (read more)

I'm confused about algorithms vs scaling conversations. I've been noodling around this question space since reading Yudkowsky vs Hanson on FOOM: Whose Predictions Were Better? a couple weeks ago, and the more I think about predictions in terms of algorithm and scale the more complete my confusion.

I currently have it head-chunked this way:

Architecture: neural nets are a family of architectures. Transformers are an example of a specific architecture.
Meta-algorithm: the arrangement of and training procedure for the stack of transforms; this seems to be what people are referring to when they say algorithmic improvements. I think of it this way because it is an algorithm-finding-algorithm.
Algorithm: the learned algorithm found by the meta-algorithm, being

... (read more)

I was trying out a different intuition for articulating the problem of AGI, and thought I would ask ChatGPT about it. I am curious how GPT4 would respond differently, and may ask some of the other LLMs about it.

Imagine a general optimizer function which optimizes the value of any function (or functions) passed to it. We pass a set of functions, A, to the general optimizer function. What happens to the value of all other functions, ~A?

The behavior of the general optimizer function towards functions outside of the set A would depend on the implementation of the function and the optimization algorithm used.

In general, the optimizer function would focus on optimizing the... (read 623 more words →)

They gave LLMs access to physics simulators

ryan_b

Over at Google, large language models have been plugged into physics simulators to help them share a world model with their human interlocutors, resulting in big performance gains. They call it Mind's Eye.

This is how the authors describe the work:

Correct and complete understanding of properties and interactions in the physical world is not only essential to achieve human-level reasoning (Lake et al., 2017), but also fundamental to build a general-purpose embodied intelligence (Huang et al., 2022). In this work, we investigate to what extent current LMs understand the basic rules and principles of the physical world, and describe how to ground their reasoning with the aid of simulation. Our contributions are three-fold:
•

... (read 241 more words →)

My wife's iPhone put together one of its Over the Years slideshows for my birthday, and she sent it to me because - exactly as the designers hoped - it contains several good memories. I have two wonderings about this.

First, the goodness in good memories. We periodically talk about the lowest-cost, best-return psychological intervention of gratitude, which seems heavily wrapped up with the idea of deliberately reviewing good memories. I know nothing about the algorithm that is used to put together these slideshows, but it seems to me it mostly triggers on various holidays and other calendar events. I wonder if there is a way to optimize this in the direction of... (read more)

Rhetoric about AGI: Notes to self

Caught the aftermath of a contest for generating short, no-context arguments for why AGI matters. It appears there will be a long-form contest in the future; these are basically notes towards an entry.

The categories of audience are too vague; my principal interest is in policymakers, so dedicate some time to analysis of (and by extension strategy for) this group.
This will require some consideration of how policy actually gets done. Big question, but I have a line of attack on that problem.
Be explicit about the rhetoric. This will provide the context for the strategy, a pattern people may mimic for future strategies, and as a nice bonus will

ryan_b

As reported by la3lorn and Daniel Kokotajlo, Gato is here and appears to me to represent a sub-human AGI, or near enough as makes no difference in a timeline sense. I think this probably means a general thickening of deep learning applications everywhere, and the introduction of a kind of "stack" AI that can do things we used to need whole organizations to do - as an example, I mean things like do patent research, label patent diagrams, and file patent lawsuits.

I also have an idea about a business I would like to start. This is already a notoriously trying task with low probability of success, and I wonder how much more... (read more)

Common Knowledge is a Circle Game for Toddlers

ryan_b

I wanted to jot down a particular instance of a common interaction I have with my daughter.

My child, wife, and I are sitting around the coffee table in the living room discussing the plan to take the car to the mechanic the next day. We describe this as going on an adventure to the child as a lingering holdover from isolation during covid, which seems to be fine because she enjoys car rides. What is different from the usual this time is that I am coming; normally car adventures are my wife driving, because she runs most of the errands. The following (simplified) dialogue takes place:

Child: "Dad, can you come in the... (read 219 more words →)

Wargaming AGI Development

ryan_b

I think wargames would be a useful tool for exploring AGI development scenarios. More specifically, I mean wargames of the kind used to train military officers in strategy. I think this format would help explore things like timelines and takeoff scenarios, in a way that is both concrete and transparent.

The thought cropped up because I have been popping into the AMA/Discussion post for Late 2021 MIRI Conversations over the course of the last week. Over the course of the MIRI conversations, and again in the AMA, I saw a lot of the tactic of offering or requesting concrete situations to test or illustrate some point. Wargaming feels to me like a good... (read 1273 more words →)

LESSWRONG
LW

LESSWRONG
LW

Update the best textbooks on every subject list

Strategy is the Deconfusion of Action

The Best Visualizations on Every Subject

Common Knowledge is a Circle Game for Toddlers

ryan_b

ryan_b

Near term discussions need something smaller and more concrete than AGI

SB 1047 gets vetoed

If I ask an LLM to think step by step, how big are the steps?

Do you have a satisfactory workflow for learning about a line of research using GPT4, Claude, etc?

My simple model for Alignment vs Capability

Assuming LK99 or similar: how to accelerate commercialization?

They gave LLMs access to physics simulators

National Institute of Standards and Technology: AI Standards

ryan_b

Update the best textbooks on every subject list

Strategy is the Deconfusion of Action

The Best Visualizations on Every Subject

Common Knowledge is a Circle Game for Toddlers

ryan_b

ryan_b

Near term discussions need something smaller and more concrete than AGI

SB 1047 gets vetoed

If I ask an LLM to think step by step, how big are the steps?

Do you have a satisfactory workflow for learning about a line of research using GPT4, Claude, etc?

My simple model for Alignment vs Capability

Assuming LK99 or similar: how to accelerate commercialization?

They gave LLMs access to physics simulators

National Institute of Standards and Technology: AI Standards

Motivation