All of evand's Comments + Replies

Have you considered looking at the old Foresight Exchange / Ideosphere data? It should be available, and it's old enough that there might be a useful number of long-term forecasts there.

http://www.ideosphere.com/

Yep, those are extremely different drugs with very different effects. They do in fact take a while and the effects can be much more subtle.

having read a few books about ADHD, my best guess is that I have (and had) a moderate case of it.

when I'm behind the wheel of a car and X = "the stoplight in front of me" and Y = "that new Indian restaurant", it is bad

Which is one of the reasons why I don't drive.

If your ADHD is interfering with your driving, that does not sound like a moderate case!

But option #3 is much better: take medication for six weeks and see what happens.

My expectation is that you will likely get a lot of information from trying the meds for 6 days; 6 weeks sounds like a very long ... (read more)

Thank you!

I went ahead and created the 2024 version of one of the questions. If you're looking for high-liquidity questions to include, which seems like a good way to avoid false alarms / pranks, this one seems like a good inclusion.

There are a bunch of lower-liquidity questions; including a mix of those with some majority-rule type logic might or might not be worth it.

Thank you! Much to think about, but later...

If there are a large number of true-but-not-publicly-proven statements, does that impose a large computational cost on the market making mechanism?

I expect that the computers running this system might have to be fairly beefy, but they're only checking proofs.

They're not, though. They're making markets on all the interrelated statements. How do they know when they're done exhausting the standing limit orders and AMM liquidity pools? My working assumption is that this is equivalent to a full Bayesian network and ex... (read more)

This is very neat work, thank you. One of those delightful things that seems obvious in retrospect, but that I've never seen expressed like this before. A few questions, or maybe implementation details that aren't obvious:

For complicated proofs, the fully formally verified statement all the way back to axioms might be very long. In practice, do we end up with markets for all of those? Do they each need liquidity from an automated market maker? Presumably not if you're starting from axioms and building a full proof, and that applies to implications and conj... (read more)

My background: educated amateur. I can design simple to not-quite-simple analog circuits and have taken ordinary but fiddly material property measurements with electronics test equipment and gotten industrially-useful results.

One person alleges an online rumor that poorly connected electrical leads can produce the same graph.  Is that a conventional view?

I'm not seeing it. With a bad enough setup, poor technique can do almost anything. I'm not seeing the authors as that awful, though. I don't think they're immune from mistakes, but I give low odds on ... (read more)

The comparison between the calculations saying igniting the atmosphere was impossible and the catastrophic mistake on Castle Bravo is apposite as the initial calculations for both were done by the same people at the same gathering!

One out of two isn't bad, right?

https://twitter.com/tobyordoxford/status/1659659089658388545

Of course a superintelligence could read your keys off your computer's power light, if it found it worthwhile. Most of the time it would not need to, it would find easier ways to do whatever humans do by pressing keys. Or make the human press the keys.

FYI, the referenced thing is not about what keys are being pressed on a keyboard, it's about extracting the secret keys used for encryption or authentication. You're using the wrong meaning of "keys".

If you think the true likelihood is 10%, and are being offered odds of 50:1 on the bet, then the Kelly Criterion suggests you should be about 8% of your bankroll. For various reasons (mostly human fallibility and an asymmetry in the curve of the Kelly utility), lots of people recommend betting at fractions of the Kelly amount. So someone in the position you suggest might reasonably wish to be something like $2-5k per $100k of bankroll. That strategy, your proposed credences, and the behavior observed so far would imply a bankroll of a few hundred thousand ... (read more)

My concern with conflating those two definitions of alignment is largely with the degree of reliability that's relevant.

The definition "does what the developer wanted" seems like it could cash out as something like "x% of the responses are good". So, if 99.7% of responses are "good", it's "99.7% aligned". You could even strengthen that as something like "99.7% aligned against adversarial prompting".

On the other hand, from a safety perspective, the relevant metric is something more like "probabilistic confidence that it's aligned against any input". So "99.... (read more)

If there are reasons to refuse bets in general, that apply to the LessWrong community in aggregate, something has gone horribly horribly wrong.

No one is requiring you personally to participate, and I doubt anyone here is going to judge you for reluctance to engage in bets with people from the Internet who you don't know. Certainly I wouldn't. But if no one took up this bet, it would have a meaningful impact on my view of the community as a whole.

I don't know how it prevents us from dying either! I don't have a plan that accomplishes that; I don't think anyone else does either. If I did, I promise I'd be trying to explain it.

That said, I think there are pieces of plans that might help buy time, or might combine with other pieces to do something more useful. For example, we could implement regulations that take effect above a certain model size or training effort. Or that prevent putting too many flops worth of compute in one tightly-coupled cluster.

One problem with implementing those regulations is... (read more)

I think neither. Or rather, I support it, but that's not quite what I had in mind with the above comment, unless there's specific stuff they're doing that I'm not aware of. (Which is entirely possible; I'm following this work only loosely, and not in detail. If I'm missing something, I would be very grateful for more specific links to stuff I should be reading. Git links to usable software packages would be great.)

What I'm looking for mostly, at the moment, is software tools that could be put to use. A library, a tutorial, a guide for how to incorporate th... (read more)

One thing I'd like to see more of: attempts at voluntary compliance with proposed plans, and libraries and tools to support that.

I've seen suggestions to limit the compute power used on large training runs. Sounds great; might or might not be the answer, but if folks want to give it a try, let's help them. Where are the libraries that make it super easy to report the compute power used on a training run? To show a Merkle tree of what other models or input data that training run depends on? (Or, if extinction risk isn't your highest priority, to report whic... (read more)

Aren't the other used cars available nearby, and the potential other buyers should you walk away, relevant to that negotiation?

This was fantastic; thank you! I still haven't quite figured it out, I'll definitely have to watch it a second time (or at least some parts of it).

I think some sort of improved interface for your math annotations and diagrams would be a big benefit, whether that's a drawing tablet or typing out some LaTeX or something else.

I think the section on induction heads and how they work could have used a bit more depth. Maybe a couple more examples, maybe some additional demos of how to play around with PySvelte, maybe something else. That's the section I had the ... (read more)

Yes, if Omega accurately simulates me and wants me to be wrong, Omega wins. But why do I need to get the answer exactly "right"? What does it matter if I'm slightly off?

This would be a (very slightly) more interesting problem if Omega was offering a bet or a reward and my goal was to maximize reward or utility or whatever. It sure looks like for this setup, combined with a non-adversarial reward schedule, I can get arbitrarily close to maximizing the reward.

This feel reminiscent of:

If the human brain were so simple that we could understand it, we would be so simple that we couldn’t.

And while it's a well-constructed pithy quote, I don't think it's true. Can a system understand itself? Can a quining computer program exist? Where is the line between being able to recite itself and understand itself?

You need a model above some threshold of capability at which it can provide useful interpretations, yes, but I don't see any obvious reason why that threshold would move up with the size of the model under interpretat

I think that sounds about right. Collecting the arguments in one place is definitely helpful, and I think they carry some weight as initial heuristics, which this post helps clarify.

But I also think the technical arguments should (mostly) screen off the heuristics; the heuristics are better for evaluating whether it's worth paying attention to the details. By the time you're having a long debate, it's better to spend (at least some) time looking instead of continuing to rely on the heuristics. Rhymes with Argument Screens Off Authority. (And in both cases, only mostly screens off.)

That's the point. SpaceX can afford to fail at this; the decision makers know it. Eliezer can afford to fail at tweet writing and knows it. So they naturally ratchet up the difficulty of the problem until they're working on problems that maximize their expected return (in utility, not necessarily dollars). At least approximately. And then fail sometimes.

Or, for the trapeze artist... how long do they keep practicing? Do they do the no-net route when they estimate their odds of failure are 1/100? 1/10,000? 1e-6? They don't push them to zero, at some point th... (read more)

Perhaps I'm missing something obvious, and just continuing the misunderstanding, but...

It seems to me that if you're the sort of thing capable of one-shotting Starship launches, you don't just hang around doing so. You tackle harder problems. The basic Umeshism: if you're not failing sometimes, you're not trying hard enough problems.

Even the "existential" risk of SpaceX getting permanently and entirely shut down, or just Starship getting shut down, is much closer in magnitude to the payoff than is the case in AI risk scenarios.

Some problems are well calibr... (read more)

Taboo "rationally".

I think the question you want is more like: "how can one have well-calibrated strong probabilities?". Or maybe "correct". I don't think you need the word "rationally" here, and it's almost never helpful at the object level -- it's a tool for meta-level discussions, training habits, discussing patterns, and so on.

To answer the object-level question... well, do you have well-calibrated beliefs in other domains? Did you test that? What do you think you know about your belief calibration, and how do you think you know it?

Personally, I think ... (read more)

I'm still uncertain how I feel about a lot of the details on this (and am enough of a lurker rather than poster that I suspect it's not worth my time to figure that out / write it publicly), but I just wanted to say that I think this is an extremely good thing to include:

I will probably build something that let's people Opt Into More Said. I think it's fairly likely the mod team will probably generally do some more heavier handed moderation in the nearish future, and I think a reasonable countermeasure to build, to alleviate some downsides of this, is to a

i could believe this number’s within 3 orders of magnitude of truth, which is probably good enough for the point of this article

It's not. As best I can tell it's off by more like 4+ OOM. A very quick search suggests actual usage was maybe more like 1 GWh. Back of the envelope guess: thousands of GPUs, thousands of hours, < 1kW/GPU, a few GWh.

https://www.theregister.com/2020/11/04/gpt3_carbon_footprint_estimate/

https://www.numenta.com/blog/2022/05/24/ai-is-harming-our-planet/

i am a little surprised if you just took it 100% at face value.

Same. That doesn'... (read more)

I feel like the argumentation here is kinda misleading.

Here's a pattern that doesn't work very well: a tragedy catches our attention, we point to statistics to show it's an example of a distressingly common problem, and we propose laws to address the issue.

The post promises to discuss a pattern. It's obviously a culture-war-relevant pattern, and I can see a pretty good case for it being one where all the examples are going to be at least culture-war-adjacent. It's an important pattern, if true, so far that seems justified and worth worrying about how to im... (read more)

Something like an h-index might be better than a total.

For an extremely brief summary of the problem, I like this from Zvi:

The core reason we so often have seen creation of things we value win over destruction is, once again, that most of the optimization pressure by strong intelligences was pointing in that directly, that it was coming from humans, and the tools weren’t applying intelligence or optimization pressure. That’s about to change.

https://thezvi.wordpress.com/2023/03/28/response-to-tyler-cowens-existential-risk-ai-and-the-inevitable-turn-in-human-history/

Intelligence has no upper limit, instead of diminishing sharply in relative utility

It seems to me that there is a large space of intermediate claims that I interpret the letter as falling into. Namely, that if there exists an upper limit to intelligence, or a point at which the utility diminishes enough to not be worth throwing more compute cycles at it, humans are not yet approaching that limit. Returns can diminish for a long time while still being worth pursuing.

you have NO EVIDENCE that AGI is hostile or is as capable as you claim or support for any of

It's always possible to fudge the numbers and decide that some values are unimportant and some are super important and lo and behold, the calculation turns in your favour! In the end it's no better than deontology or simply saying "I think this is good"; there is no point trying to vest it with a semblance of objectivity that just isn't there.

Is this not simply the fallacy of gray?

As saying goes, it's easy to lie with statistics, but even easier to lie without them. Certainly you can fudge the numbers to make the result say anything, but if you show your work then the fudging gets more obvious.

I think you missed a follow-on edit:

"Let’s unpack what that 0.36 bits means,"

Metrics are only useful for comparison if they're accepted by a sufficient broad cross section of society. Since nearly everyone engages in discourse.

I note that "sufficiently broad" might mean something like "most of LessWrong users" or "most people attending this [set of] meetups". Just as communication is targeted at a particular audience, discourse norms are (presumably) intended for a specific context. That context probably includes things like intended users, audience, goals, and so on. I doubt "rationalist discourse" norms will align well with "tele... (read more)

"Physicist motors" makes little sense because that position won out so completely that the alternative is not readily available when we think about "motor design". But this was not always so! For a long time, wind mills and water wheels were based on intuition.

But in fact one can apply math and physics and take a "physicist motors" approach to motor design, which we see appearing in the 18th and 19th centuries. We see huge improvements in the efficiency of things like water wheels, the invention of gas thermodynamics, steam engines, and so on, playing a ma... (read more)

Obvious kinds of humans include:

Dead humans. (Who didn't manage to leave the coins to their heirs.)

Cryonically preserved humans hoping to use them later. (Including an obvious specific candidate.)

Humans optimistic enough about Bitcoin to think current prices are too low. (We know Nakamoto had resources, so it seems a safe bet that they could keep living on ordinary means for now.)

And the obvious: you don't know that all of Nakamoto's coins fit the standard assumed profile. It's entirely possible they intentionally mined some with the regular setup and are spending a few from that pool.

The advanced answer to this is to create conditional prediction markets. For example: a market for whether or not the Bank of Japan implements a policy, a market for the future GDP or inflation rate of Japan (or whatever your preferred metric is), and a conditional market for (GDP given policy) and (GDP given no policy).

Then people can make conditional bets as desired, and you can report your track record, and so on. Without a prediction market you can't, in general, solve the problem of "how good is this prediction track record really" except by looking at it in detail and making judgment calls.

I hope you have renter's insurance, knowledge of a couple evacuation routes, and backups for any important data and papers and such.

I'm not aware of any legal implications in the US. US gambling laws basically only apply when there is a "house" taking a cut or betting to their own advantage or similar. Bets between friends where someone wins the whole stake are permitted.

As for the shady implications... spend more time hanging out with aspiring rationalists and their ilk?

The richer structure you seek for those two coins is your distribution over their probabilities. They're both 50% likely to come up heads, given the information you have. You should be willing to make exactly the same bets about them, assuming the person offering you the bet has no more information than you do. However, if you flip each coin once and observe the results, your new probability estimate for next flips are now different.

For example, for the second coin you might have a uniform distribution (ignorance prior) over the set of all possible probabilities. In that case, if you observe a single flip that comes up heads, your probability that the next flip will be heads is now 2/3.

Well, in general, I'd say achieving that reliability through redundant means is totally reasonable, whether in engineering or people-based systems.

At a component level? Lots of structural components, for example. Airplane wings stay attached at fairly high reliability, and my impression is that while there is plenty of margin in the strength of the attachment, it's not like the underlying bolts are being replaced because they failed with any regularity.

I remember an aerospace discussion about a component (a pressure switch, I think?). NASA wanted documenta... (read more)

You might also want a mechanism to handle "staples" that individuals want. I have a few foods / ingredients I like to keep on hand at all times, and be able to rely on having. I'd have no objections to other people eating them, but if they did I'd want them to take responsibility for never leaving the house in a state of "no X on hand".

Those numbers sound like reasonable estimates and goals. Having taught classes at TechShop, that first handful of hours is important. 20 hours of welding instruction ought to be enough that you know whether you like it and can build some useful things, but probably not enough to get even an intro-level job. It should give you a clue as to whether signing up for a community college class is a good idea or not.

Also I'm really confused by your inclusion of EE in that list; I'd have put it on the other one.

However, I'm skeptical of systems that require 99.99% reliability to work. Heuristically, I expect complex systems to be stable only if they are highly fault-tolerant and degrade gracefully.

On the other hand... look at what happens when you simply demand that level of reliability, put in the effort, and get it. From my engineering perspective, that difference looks huge. And it doesn't stop at 99.99%; the next couple nines are useful too! The level of complexity and usefulness you can build from those components is breathtaking. It's what makes the 21st... (read more)

What happens when the committed scorched-earth-defender meets the committed extortionist? Surely a strong precommitment to extortion by a powerful attacker can defeat a weak commitment to scorched earth by a defender?

It seems to me this bears a resemblence to Chicken or something, and that on a large scale we might reasonably expect to see both sets of outcomes.

What's that? If I don't give into your threat, you'll shoot me in the foot? Well, two can play at that game. If you shoot me in the foot, just watch, I'll shoot my other foot in revenge.

On the other hand... what level do you want to examine this at?

We actually have pretty good control of our web browsers. We load random untrusted programs, and they mostly behave ok.

It's far from perfect, but it's a lot better than the desktop OS case. Asking why one case seems to be so much farther along than the other might be instructive.

Again, I'm going to import the "normal computer control" problem assumptions by analogy:

• The normal control problem allows minor misbehaviour, but that it should not persist over time

Take a modern milling machine. Modern CNC mills can include a lot of QC. They can probe part locations, so that the setup can be imperfect. They can measure part features, in case a raw casting isn't perfectly consistent. They can measure the part after rough machining, so that the finish pass can account for imperfections from things like temperature variation.... (read more)

It bends the nails, leaves dents in the surface and given the slightest chance will even attack your fingers!

We've mostly solved that problem.

I'm not sure that being able to nearly perfectly replicate a fixed set of physical actions is the same thing as solving a control problem.

It's precisely what's required to solve the problem of a hammer that bends nails and leaves dents, isn't it?

Stuxnet-type attacks

I think that's outside the scope of the "hammer control problem" for the same reasons that "an unfriendly AI convinced my co-wor... (read more)

They usually don't have any way to leverage their models to increase the cost of not buying their product or service though; so such a situation is still missing at least one criterion.

Modern social networks and messaging networks would seem to be a strong counterexample. Any software with both network effects and intentional lock-in mechanisms, really.

And honestly, calling such products a blend of extortion and trade seems intuitively about right.

To try to get at the extortion / trade distinction a bit better:

Schelling gives us definitions of promises ... (read more)