All of Daphne_W's Comments + Replies

I asked you to try out avoiding quotation marks. You said I hate you.

I'm guessing that people who "made it" have a bunch of capital that they can use to purchase AI labor under the scenario you outline (i.e., someone gets superintelligence to do what they want). 

If the superintelligence is willing to deprive people of goods and services because they lack capital, then why would it be empathetic towards those that have capital? The superintelligence would be a monopsony and monopoly, and could charge any amount for someone existing for an arbitrarily short amount of time. Assuming it even respects property law when it i... (read more)

The Demon King does not solely attack the Frozen Fortress to profit on prediction markets. The story tells us that the demons engage in regular large-scale attacks, large enough to serve as demon population control. There is no indication that these attacks decreased in size when they were accompanied with market manipulation (and if they did, that would be a win in and of itself).

So the prediction market's counterfactual is not that the Demon King's forces don't attack, but that they attack at an indeterminate time with the same approximate frequency and ... (read more)

That's probably the only "military secret" that really matters.

The soldiers guarding the outer wall and the Citadel treasurer that pays their overtime wages would beg to differ. 

Yes, that is the price she got for giving the information.

I think that AI people that are very concerned about AI risk tend to view loss of control risk as very high, while eternal authoritarianism risks are much lower.

I'm not sure how many people see the risk of eternal authoritarianism as much lower and how many people see it as being suppressed by the higher probability of loss of control^[1]. Or in Bayesian terms:

P(eternal authoritarianism) = P(eternal authoritarianism | control is maintained) $\cdot$ P(control is maintained)

Both sides may agree that P(eternal authoritarianism | control is maintained) is h... (read more)

As far as I understand, "a photon in a medium" is a quasiparticle. Actual photons always travel at the speed of light, and the "photon" that travels through glass at a lower speed is the sum of an incredibly complicated process that cancels out perfectly into something that can be described as one or several particles if you squint a little because the energy of the electromagnetic field excitation can't be absorbed by the transparent material and because of preservation of momentum.

The model of the photon "passing by atoms and plucking them" is a lie to c... (read more)

Though compare and contrast Dune's test of the gom jabbar:

You've heard of animals chewing off a leg to escape a trap? There's an animal kind of trick. A human would remain in the trap, endure the pain, feigning death that he might kill the trapper and remove a threat to his kind.

Even if you are being eaten, it may be right to endure it so that you have an opportunity to do more damage later.

You're suggesting angry comments as an alternative for mass retributive downvoting. That easily implies mass retributive angry comments.

As for policing against systemic bias in policing, that's a difficult problem that society struggles with in many different areas because people can be good at excusing their biases. What if one of the generals genuinely makes a comment people disagree with? How can you determine to what extent people's choice to downvote was due to an unauthorized motivation?

It seems hard to police without acting draconically.

Just check their profile for posts that do deserve it that you were previously unaware of. You can even throw a few upvotes at their well-written comments. It's not brigading, it's just a little systemic bias in your duties as a user with upvote-downvote authority.

Are you trying to prime people to harass the generals?

Besides, it's not mass downvoting, it's just that the increased attention to their accounts revealed a bunch of poorly written comments that people genuinely disagree with and happen to independently decide are worthy of a downvote :)

"why not just" is a standard phrase for saying what you're proposing would be simple or come naturally if you try. Combined with the rest of the comment talking about straightforwardness and how little word count, and it does give off a somewhat combatitive vibe.

I agree with your suggestion and it is good to hear that you don't intend it imply that it is simple, so maybe it would be worth editing the original comment to prevent miscommunication for people who haven't read it yet. For the time being I've strong-agreed with your comment to save it from a negativity snowball effect.

No. I would estimate that there are fewer rich people willing to sacrifice their health for more income than there are poor people willing to do the same. Rich people typically take more holidays, report higher job satisfaction, suffer fewer stress-related ailments, and spend more time and money on luxuries rather than reinvesting into their careers (including paying basic cost of living to be employable).

And not for lack of options. CEOs can get involved with their companies and provide useful labor by putting their nose to the grindstone, or kowtow to in... (read more)

There are rich people pushing themselves work 60+ hour days struggling to keep a smile on their face while people insult and demean them. And there are poor people who live as happy ascetics, enjoying the company of their fellows and eating simple meals, choosing to work few hours even if it means forgoing many things the middle class would call necessities.

There are more rich people that choose to give up the grind than poor people. It's tougher to accept a specific form of suffering if you see that 90% of your peers are able to solve the suffering with w... (read more)

You seem to approach the possible existence of a copy like a premise, with as question whether that copy is you. However, what if we reverse that? Given we define 'a copy of you' as another one of you, how certain is it that a copy of you could be made given our physics? What feats of technology are necessary to make that copy?

Also, what would we need to do to verify that a claimed copy is an actual copy? If I run ChatGPT-8 and ask it to behave like you would behave based on your brain scan and it manages to get 100% fidelity in all tests you can think of,... (read more)

In the post you mention Epistemic Daddies, mostly describing them as sources that are deferred to for object-level information.

I'd say there is also a group of people who seek Epistemic Mommies. People looking for emotional assurance that they're on the right path and their contribution to the field is meaningful; for assurance that making mistakes in reasoning is okay; for someone to do the annoying chores of epistemic hygiene so they can make the big conclusions; for a push to celebrate their successes and show them off to others; etc.

Ultimately both are... (read more)

I kind of... hard disagree?

Effective Samaritans can't be a perfect utility inverse of Effective Altruists while keeping the labels of 'human', 'rational', or 'sane'. Socialism isn't the logical inverse of Libertarianism; both are different viewpoints on how to achieve the common goal of societal prosperity.

Effective Samaritans won't sabotage an EA social experiment any more than Effective Altruists will sabotage an Effective Samaritan social experiment. If I received a letter from Givewell thanking me for my donation that was spent on sabotaging a socialis... (read more)

On third order, people who openly worry about X-Risk may get influenced by their environment, becoming less worried as a result of staying with a company whose culture denies X-Risk, which could eventually even cause them to contribute negatively to AI Safety. Preventing them from getting hired prevents this.

That sounds like something a cross between learned helplessness and madman theory.

The madman theory angle is "If I don't respond well to threats of negative outcomes, people (including myself) have no reason to threaten me". The learned helplessness angle is "I've never been able to get good sets of tasks and threats, and trying to figure something out usually leads to more punishment, so why put in any effort?"

Combine the two and you get "Tasks with risks of negative outcomes? Ugh, no."

With learned helplessness, the standard mechanism for (re)learni... (read more)

I'd say "fuck all the people who are harming nature" is black-red/rakdos's view of white-green/selesnya. The "fuck X" attitude implies a certain passion that pure black would call wasted motion. Black is about power. It's not adversarial per se, just mercenary/agentic. Meanwhile the judginess towards others is an admixture of white. Green is about appreciating what is, not endorsing or holding on to it.

Black's view of green is "careless idiots, easy to take advantage of if you catch them by surprise". When black meets green, black notices how the commune's... (read more)

Fighting with tigers is red-green, or Gruul by MTG terminology. The passionate, anarchic struggle of nature red in tooth and claw. Using natural systems to stay alive even as it destroys is black-green, or Golgari. Rot, swarms, reckless consumption that overwhelms.

Pure green is a group of prehistoric humans sitting around a campfire sharing ghost stories and gazing at the stars. It's a cave filled with handprints of hundreds of generations that came before. It's cats louging in a sunbeam or birds preening their feathers. It's rabbits huddling up in their d... (read more)

That doesn't seem like a good idea. You're ignoring long-term harms and benefits of the activity - otherwise cycling would be net positive - and you're ignoring activity duration. People don't commute to work by climbing Mount Everest or going skydiving.

I don't think it's precisely true. The serene antagonism that comes from having examined something and recognizing that it is worth taking your effort to destroy is different from the hot rage of offense. But of the two, I expect antagonism to be more effective in the long term.

• Rage is accompanied with a surge of adrenalin, sympathetic nervous activation, and usually parasympathetic nervous suppression, that is not sustainable in the long term. Antagonism is compatible with physiological rest and changes in the environment.
• Consequently, antagonism has acce

As far as I can tell, the AI has no specialized architecture for deciding about its future strategies or giving semantic meaning to its words. It outputting the string "I will keep Gal a DMZ" does not have the semantic meaning of it committing to keep troops out of Gal. It's just the phrase players that are most likely to win use in that boardstate with its internal strategy.

Like chess grandmasters being outperformed by a simple search tree when it was supposed to be the peak of human intelligence, I think this will have the same effect of disenchanting th... (read more)

Dear M.Y. Zuo,

I hope you are well.

It is my experience that the conventions of e-mail are significantly more formal and precise in expectation when it comes to phrasing. Discord and Slack, on the other hand, have an air of informal chatting, which makes it feel more acceptable to use shortcuts and to phrase things less carefully. While feelings may differ between people and conventions between groups, I am quite confident that these conventions are common due to both media's origins, as a replacement for letters and memos and as a replacement for in-p... (read more)

That's a bit of a straw man, though to be fair it appears my question didn't fit into your world model as it does in mine.

For me, the insurrection was in the top 5 most informative/surprising US political events in 2017-2021. On account of its failure it didn't have as major consequences as others, but it caused me to update my world model more. For me, it was a sudden confrontation with the size and influence of anti-democratic movements within the Republican party, which I consider Trump to be sufficiently associated with to cringe from the notion of vot... (read more)

With Trump/Republicans I meant the full range of questions from from just Trump, through participants in the storming of congress, to all Republican voters.

It seems quite easy for a large fraction of a population to be a threat to the population's interests if they share a particular dangerous behavior. I'm confused why you would think that would be difficult. Threat isn't complete or total. If you don't get a vaccine or wear a mask, you're a threat to immune-compromissd people but you can still do good work professionally. If you vote for someone attempti... (read more)

Hey, I stumbled on this comment and I'm wondering if you've updated on whether you consider Trump/Republicans a threat to America's interests in light of the January 6th insurrection.

People currently give MIRI money in the hopes they will use it for alignment. Those people can't explain concretely what MIRI will do to help alignment. By your standard, should anyone give MIRI money?

When you're part of a cooperative effort, you're going to be handing off tools to people (either now or in the future) which they'll use in ways you don't understand and can't express. Making people feel foolish for being a long inferential distance away from the solution discourages them from laying groundwork that may well be necessary for progress, or even from exploring.

As a concrete example of rational one-hosing, here in the Netherlands it rarely gets hot enough that ACs are necessary, but when it does a bunch of elderly people die of heat stroke. Thus, ACs are expected to run only several days per year (so efficiency concerns are negligible), but having one can save your life.

I checked the biggest Dutch-only consumer-facing online retailer for various goods (bol.com). Unfortunately I looked before making a prediction for how many one-hose vs two-hose models they sell, but even conditional on me choosing to make a point... (read more)

It feels more to me like we're the quiet weird kid in high school that doesn't speak up or show emotion because we're afraid of getting judged or bullied. Which, fair enough, the school is sort of like - just look at poor cryonics, or even nuclear power - but the road to popularity (let along getting help with what's bugging us) isn't to try to minimize our expressions to 'proper' behavior while letting us be characterized by embarrassing past incidents (e.g. Roko's Basilisk) if we're noticed at all.

It isn't easy to build social status, but right now we're trying next to nothing and we've seen it doesn't seem to do enough.

Agree that it's too shallow to take seriously, but

If it answered "you would say during text input batch 10-203 in January 2022, but subjectively it was about three million human years ago" that would be something else.

only seems to capture AI that managed to gradient hack the training mechanism to pass along its training metadata and subjective experience/continuity. If a language model were sentient in each separate forward pass, I would imagine it would vaguely remember/recognize things from its training dataset without necessarily being able to place them, like a human when asked when they learned how to write the letter 'g'.

Interventions on the order of burning all GPUs in clusters larger than 4 and preventing any new clusters from being made, including the reaction of existing political entities to that event and the many interest groups who would try to shut you down and build new GPU factories or clusters hidden from the means you'd used to burn them, would in fact really actually save the world for an extended period of time and imply a drastically different gameboard offering new hopes and options.

I suppose 'on the order of' is the operative phrase here, but that specifi... (read more)

AI can run on CPUs (with a certain inefficiency factor), so only burning all GPUs doesn't seem like it would be sufficient. As for disruptive acts that are less deadly, it would be nice to have some examples but Eliezer says they're too far out of the Overton Window to mention.

If what you're saying about Eliezer's claim is accurate, it does seem disingenuous to frame "The only worlds where humanity survives are ones where people like me do something extreme and unethical" as "I won't do anything extreme and unethical [because humanity is doomed anyway]". I... (read more)

I'm confused about A6, from which I get "Yudkowsky is aiming for a pivotal act to prevent the formation of unaligned AGI that's outside the Overton Window and on the order of burning all GPUs". This seems counter to the notion in Q4 of Death with Dignity where Yudkowsky says

It's relatively safe to be around an Eliezer Yudkowsky while the world is ending, because he's not going to do anything extreme and unethical unless it would really actually save the world in real life, and there are no extreme unethical actions that would really actually save the world

Your method of trying to determine whether something is true or not relies overly much on feedback from strangers. Your comment demands large amounts of intellectual labor from others ('disprove why all easier modes are incorrect'), despite the preamble of the post, while seeming unwilling to put much work in yourself.

I think Yudkowsky would argue that on a scale from never learning anything to eliminating half your hypotheses per bit of novel sensory information, humans are pretty much at the bottom of the barrel.

When the AI needs to observe nature, it can rely on petabytes of publicly available datasets from particle physics to biochemistry to galactic surveys. It doesn't need any more experimental evidence to solve human physiology or build biological nanobots: we've already got quantum mechanics and human DNA sequences. The rest is just derivation of the consequence... (read more)

• Solve protein folding problem
• Acquire human DNA sample
• Use superintelligence to construct a functional model of human biochemistry
• Design a virus that exploits human biochemstry
• Use one of the currently available biochemistry-as-a-service providers to produce a sample that incubates the virus and then escapes their safety procedures (e.g. pay someone to mix two vials sent to them in the mail. The aerosols from the mixing infect them)

Hey, it's now officially no longer May 27th anywhere, and I can't find any announcements yet. How's it going?

Edit: Just got my acceptance letter! See you all this summer!

Sorry that automation is taking your craft. You're neither the first nor the last this will happen to. Orators, book illuminators, weavers, portrait artists, puppeteers, cartoon animators, etc. Even just in the artistic world, you're in fine company. Generally speaking, it's been good for society to free up labor for different pursuits while preserving production. The art can even be elevated as people incorporate the automata into their craft. It's a shame the original skill is lost, but if that kept us from innovating, there would be no way to get common... (read more)

Before learning about reversible computation only requiring work when bits are deleted I would have treated each of my points as roughly independent with about 10^1.5 , 10^4 , 10^4 , 10^2.5 odds against respectively. The last point is now down to 10^1.5 .

Dumping waste information in the baryonic world would be visible.

#1 - Caution doesn't solve problems, it finds solutions if they exist. You can't use caution to ignore air resistance when building a rocket. (Though collapse is not necessarily expected - there's plenty of interstellar dust).

#4 - I didn't know about Landauer's principle, though going by what I'm reading, you're mistaken on its interpretation - it takes 'next to nothing' times the part of the computation you throw out, not the part you read out, where the part you throw out increases proportional to the negentropy you're getting. No free lunch, still, but ... (read more)

1:10^12 odds against the notion, easily. About as likely as the earth being flat.

1. Dark matter does not interact locally with itself or visible matter. If it did, it would experience friction (like interstellar gas, dust and stars) and form into disk shapes when spiral galaxies form into disk shapes. A key observation of dark matter is that spiral galaxies' rotational velocity behaves as one would expect from an ellipsoid.
2. The fraction of matter that is dark does not change over time, nor does the total mass of objects in the universe. Sky surveys do not find

Unlike what you would expect with black holes, we can see that the Boötes void contains very little mass by looking for gravitational lensing and the movement of surrounding galaxies.

There definitely seem to be (relative) grunt work positions in AI safety, like this, this or this. Unless you think these are harmful, it seems like it would be better to direct the Alec-est Alecs of the world that way instead of risking them never contributing.

I understand not wanting to shoulder responsibility for their career personally, and I understand wanting an unbounded culture for those who thrive under those conditions, but I don't see the harm in having a parallel structure for those who do want/need guidance.

Well, it's better, but in I think you're still playing into [Alec taking things you say as orders], which I claim is a thing, so that in practice Alec will predictably systematically be less helpful and more harmful than if he weren't [taking things you say as orders].

There seems to be an assumption here that Alec would do something relatively helpful instead if he weren't taking the things you say as orders. I don't think this is always the case: for people who aren't used to thinking for themselves, the problem of directing your career to reduce AI risk ... (read more)

This is where things go wrong. The actual credence of seeing a hypercomputer is zero, because a computationally bounded observer can never observe such an object in such a way that differentiates it from a finite approximation. As such, you should indeed have a zero percent probability of ever moving into a state in which you have performed such a verification, it is a logical impossibility. Think about what it would mean for you, a computationally bounded approximate bayesian, to come into a state of belief that you are in possession of a hypercomputer (a

That's not a middle ground between a good world and a neutral world, though, that's just another way to get a good world. If we assume a good world is exponentially unlikely, a 10 year delay might mean the odds of a good world rise from 10^-10 to 10^-8 (as opposed to pursuing Clippy bringing the odds of a bad world down from 10^-4 to 10^-6 ).

If you disagree with Yudkowsky about his pessimism about the probability of good worlds, then my post doesn't really apply. My post is about how to handle him being correct about the odds.

That's a fair point - my model does assume AGI will come into existence in non-negative worlds. Though I struggle to actually imagine a non-negative world where humanity is alive a thousand years from now and AGI hasn't been developed. Even if all alignment researchers believed it was the right thing to pursue, which doesn't seem likely.