All of scarcegreengrass's Comments + Replies

It sounds like the core idea is a variant of the Intelligence Manhattan Project idea, but with a focus on long term international stability & a ban on competitors.

Perhaps the industry would be more likely to adopt this plan if GUARD could seek revenue the way corporations currently do: by selling stock & API subscriptions. This would also increase productivity for GUARD & shorten the dangerous arms race interval.

Answer by scarcegreengrass20

I think this sounds fun! The versions of this i'd be most likely to use would be:

  • Puzzling over scenarios of satisfying complexity. There could be numerical details, selection bias, unreliable narrator obstacles, cases where users with different values might disagree, etc. Even if the scenario-poster is arguably wrong about the right answer, that could still be interesting.
  • Scenarios that you puzzle over & then read a comment section about. Scenarios that you remember & talk about with friends later.
  • User-submitted anecdotes from their real lives. Th
... (read more)

Imperial Radch series by Ann Leckie

Very well-crafted world. Some might dislike the robotic narrator, some might enjoy it as a fun layer in a complex plot puzzle. High scifiosity.

Southern Reach trilogy by Jeff VanderMeer

Surreal & unusual novels. Good tone & imagery. Unlike Radch, i think this is more about style & perspective than a style layer over a intricate, hidden plot layer.

Too Like the Lightning by Ada Palmer

I read a lot of scifi, but i haven't gotten this obsessed with a book since Green Mars! Like Radch, a unreliable narrator presents a... (read more)

I quite like the Arguman format of flowcharts to depict topics. In a live performance, participants might sometimes add nodes to the flowchart, or sometimes ask for revision to another participant's existing node. For example, asking for rewording for clarity.

Perhaps the better term would be tree, not flowchart. Each node is a response to its parent. This could perhaps be implemented with bulleted lists in a Google Doc.

It's nice for the event to output a useful document.

I call all those examples opinions.

Sure, opinions come to people from a few different sources. I speculate that interpersonal transmission is the most common, but they can also originate in someone's head, either via careful thought or via a brief whim.

1CstineSublime
You're probably right that interpersonal transmission is the most common source. I guess now I have to ask what do you mean by an "opinion". Is a simple proposition like "That popstar is bad" or "that's too much (food/money/time)" enough to warrant an opinion? I ask because now thinking about it, most endogenous opinions (which presumedly most interpersonally transmitted opinions began as, with the exception of those that are the result of Chinese-whispers) are just post-rationalizations of emotive or felt experiences.  To pull an example out of thin air: "Looking at this art work doesn't make me feel 'good'... it must be because it is a non-figurative painting" - like is the simple emotive expression "this painting isn't good" as vague as it is an opinion, or is the attempt to explain it "it's a messy nonfigurative painting that doesn't depict anything" an opinion?

People don't have opinions - opinions have people.

Often, one hears someone express a strange, wrong-seeming opinion. The bad habit is to view this as that person's intentional bad action. The good habit is to remember that the person heard this opinion, accepted it as reasonable, & might have put no further thought into the matter. 

Opinions are self-replicating & rarely fact-checked. People often subscribe to 2 contradictory opinions.

Epistemic status: I'm trying this opinion on. It's appealing so far. 

1CstineSublime
  Are you specifically referring to opinions that are held by multiple people, albeit are unpopular (and, as you say, aren't the result of intentional bad action - they are not opinions held out of contrariety). Because isn't it possible for people to form their own endogenous strange opinions? 

I like it! In addition, I suppose you could use a topic-wide prior for those groups that you don't have much data on yet.

Personally I'd rather have the public be fascinated with how chatbots think than ignorant of the topic. Sure, non experts won't have a great understanding, but this sounds better than likely alternatives. And I'm sure people will spend a lot of time on either future chatbots, or future video games, or future television, or future Twitter, but I'm not convinced that's a bad thing.

The regulation you mention sounds very drastic & clumsy to my ears. I'd suggest starting by proposing something more widely acceptable, such as regulating highly effective self modifying software that lacks security safeguards.

1otto.barten
I've made an edit and removed the specific regulation proposal. I think it's more accurate to just state that it needs to be robust, do as little harm as possible, and that we don't know yet what it should look like precisely.
1otto.barten
I agree that it's drastic and clumsy. It's not an actual proposal, but a lower bound of what would likely work. More research into this is urgently needed. Aren't you afraid that people could easily circumvent the regulation you mention? This would require every researcher and hacker, everywhere, forever, to comply. Also, many researchers are probably unaware that their models could start self-improving. Also, I'd say the security safeguards that you mention amount to AI Safety, which is of course currently an unsolved problem. But honestly, I'm interested in regulation proposals that would be sufficiently robust while minimizing damage. If you have those, I'm all ears.

Basing ethical worth off of qualia is very close to dualism, to my ears. I think instead the question must rest on a detailed understanding of the components of the program in question, & the degree of similarity to the computational components of our brains.

1green_leaf
It's enough to talk to the system. Any system that can pass the Turing test implements the state-machine-which-is-the-person of which the Turing test has been passed. (This is necessarily true, because if no elements of the system implemented the answer-generating aspects of the state machine, the system couldn't generate the Turing-test-passing answer.)
1TAG
You think there is no possible non-dualistic account of qualia? You think qualia exist?
1Shiroe
Degree of computational similarity here is a heuristic to relate our experiences to the purported experiences of agents whose ethical concern is under our consideration.
3MSRayne
I'm not sure what you mean. Qualia clearly exist. You're having them right now. Whatever they are, they're also "components of programs". That's just what qualia look like from the outside - there is no separation. I am by no means a dualist - I think consciousness and information processing are the same exact thing.

Excellent point. We essentially have 4 quadrants of computational systems:

  • Looks nonhuman, internally nonhuman - All traditional software is in this category
  • Looks nonhuman, internally humanoid - Future minds that are at risk for abuse (IMO)
  • Looks humanoid, internally nonhuman - Not a ethical concern, but people are likely to make wrong judgments about such programs. 
  • Looks humanoid, internally humanoid - Humans. The blogger claims LaMDA also falls into this category.

Good point. In my understanding it could go either way, but I'm open to the idea that the worst disasters are less than 50% likely, given a nuclear war.

Good point. Unless of course one is more likely to be born into universes with high human populations than universes with low human populations, because there are more 'brains available to be born into'. Hard to say.

2[anonymous]
I don't think an all-out nuclear war would even substantially (i.e. by orders of magnitude) reduce world population, though this claim is a bit more controversial. That may sound morbid but it's what is relevant for anthropics.

In general, whenever Reason makes you feel paralyzed, remember that Reason has many things to say. Thousands of people in history have been convinced by trains of thought of the form 'X is unavoidable, everything is about X, you are screwed'. Many pairs of those trains of thought contradict each other. This pattern is all over the history of philosophy, religion, & politics. 

Future hazards deserve more research funding, yes, but remember that the future is not certain.

1superads91
"Thousands of people in history have been convinced by trains of thought of the form 'X is unavoidable, everything is about X, you are screwed'." Care to give a few examples? Because I'd venture saying that, except for religious and other superstitious beliefs, and except for crazy lunatics too like fascists and communists, they were mostly right. "the future is not certain" Depends on what you mean by that. If you mean that it's not extremely likely, like 90% plus, that we will develop some truly dangerous form of AI this century that will pose immense control challenges, then I'd say you're deeply misguided given the smoke signals that have been coming up since 2017. I mean, it's like worrying about nuclear war. Is it certain that we'll ever get a big nuclear war? No. Is it extremely likely if things stay the same and if enough time passes (10, 50, 100, 200, 300 years)? Hell yes. I mean, just look at the current situation... Though I don't care about nuclear war much because it is also extremely likely that it will come with a warning, so you can also run to the countryside, and even then if things go bad like you're starving to death or dying of radiation poisoning, you can always put an end to your own suffering. With AI you might not be so lucky. You might end in an unbreakable dictatorship a la With Folded Hands. How can you not feel paralyzed when you see chaos pointed at your head and at the heads of other humans, coming in as little as 5 or 10 years, and you see absolutely no solution, or much less anything you can do yourself? We can't even build a provably safe plane, how are we gonna build a provably safe TAI with the work of a few hundred people over 5-30 years, and with complete ignorance by most? The world would have to wake up, and I don't think it will. Really, the only ways we will not build dangerous and uncontrollable AI is if either we destroy ourselves by some other way first (or even just with narrow AI), or the miracle happens that s

What's the status of this meetip, CitizenTen? Did you hear back?

2CitizenTen
All set.  I got transferred to host.  Nothing should change.  Though tbh, it shouldn't matter much.  It's mainly just a shelling point.  I'm sure even if Jessie did nothing someone would show up.  But yeah.  See you there!
Answer by scarcegreengrass90

I have similar needs. I use a spreadsheet, populated via a Google Form accessible via a shortcut from my phone's main menu. I find it rewarding to make the spreadsheet display secondary metrics & graphs too.

Other popular alternatives include Habitica & habitdaily.app (iPhone only). I'm still looking for a perfect solution, but my current tools are pretty good for my needs.

I'm not sure either. Might only be needed for the operating fees.

Agreed. We might refer to them as 'leaderless orgs' or 'staffless networks'.

Does this reduction come from seniority? Is the idea that older organizations are generally more reliable?

1gareth
The reduction comes from the passage of time. Let’s say that a company predicts 10% growth over the year but 6 months into the year they have an equivalent annual growth rate of 2%. That doesn’t mean that they won’t make 10% at the end of the year, but it makes it less likely, so the value of the company changes to reflect that new reality. It’s important to define risk and uncertainty. Risk in this case means probability of winning or loosing something, and can be measured, whereas uncertainty is about the lack of information about a situation and can not be measured as it is unknown. Uncertainty is reduced as more knowledge is gained about reality and as long-term risks become short-term risks, due to the passage of time, only to be replaced with new longer term risks. There are some really interesting concepts around the future value of money, opportunity costs and how to value companies. I’d recommend coursera finance 101 courses.

Are you saying there would be a causal link from the poor person's vaccine:other ratio to the rich person's purchasing decision? How does that work?

Can you clarify why the volcano triggering scheme in 3 would not be effective? It's not obvious. The scheme sounds rather lethal.

3[anonymous]
Not even the right order of magnitude. Yellowstone magma chamber is 5km beneath the surface. If you had a nuke large enough to set off a supervolcano, you wouldn't need to set off a supervolcano. Not to mention Yellowstone isn't ready to blow anyway.

Welcome! Discovering the rationalsphere is very exciting, isn't it? I admire your passion for self improvement.

I don't know if I have advice that isn't obvious. Read whoever has unfamiliar ideas. I learned a lot from reading Robin Hanson and Paul Christiano.

As needed, journal or otherwise speak to yourself.

Be wary of the false impression that your efforts have become ruined. Sometimes i encounter a disrespectful person or a shocking philosophical argument that makes me feel like giving up on a wide swathe of my life. I doubt giving up is appropriate in the... (read more)

8WrongPlanet
Thank you very much for your motivation and advice! I will follow your suggestions and read about those two you mentioned.

Okay, deciding randomly to exploit one possible simulator makes sense.

As for choosing exactly what to see the output cells of the simulation to... I'm still wrapping my head around it. Is recursive simulation the only way to exploit these simulations from within?

Great post. I encountered many new ideas here.

One point confuses me. Maybe I'm missing something. Once the consequentialists in a simulation are contemplating the possibility of simulation, how would they arrive at any useful strategy? They can manipulate the locations that are likely to be the output/measurement of the simulation, but manipulate to what values? They know basically nothing about how the input will be interpreted, what question the simulator is asking, or what universe is doing the simulation. Since their universe is very simple, presumably... (read more)

3Mark Xu
Consequentialists can reason about situations in which other beings make important decisions using the Solomonoff prior. If the multiple beings are simulated them, they can decide randomly (because having e.g. 1/100 of the resources is better than none, which is the expectation of "blind mischievousness"). An example of this sort of reasoning is Newcomb's problem with the knowledge that Omega is simulating you. You get to "control" the result of your simulation by controlling how you act, so you can influence whether or not Omega expects you to one-box or two-box, controlling whether there is $1,000,000 in one of the boxes.

I indeed upvoted it for the update / generally valuable contribution to the discussion.

a) Agreed, although I don't find this inappropriate in context.

b) I do agree that the fact that many successful past civilizations are now in ruins with their books lost is a important sign of danger. But surely there is some onus of proof in the opposite direction from the near-monotonic increase in population over the last few millennia?

c) These are certainly extremely important problems going forwards. I would particularly emphasize the nukes.

d) Agreed. But on the centuries scale, there is extreme potential in orbital solar power and fusion.

e) Agre... (read more)

Epistemics: Yes, it is sound. Not because of claims (they seem more like opinions to me), but because it is appropriately charitable to those that disagree with Paul, and tries hard to open up avenues of mutual understanding.

Valuable: Yes. It provides new third paradigms that bring clarity to people with different views. Very creative, good suggestions.

Should it be in the Best list?: No. It is from the middle of a conversation, and would be difficult to understand if you haven't read a lot about the 'Foom debate'.

Improved: The same concepts... (read more)

This is a little nitpicky, but i feel compelled to point out that the brain in the 'human safety' example doesn't have to run for a billion years consecutively. If the goal is to provide consistent moral guidance, the brain can set things up so that it stores a canonical copy of itself in long-term storage, runs for 30 days, then hands off control to another version of itself, loaded from the canonical copy. Every 30 days control is handed to a instance of the canonical version of this person. The same scheme is possible for a group of peopl... (read more)

I appreciate this disentangling of perspectives. I had been conflating them before, but i like this paradigm.

I found this uncomfortable and unpleasant to read, but i'm nevertheless glad i read it. Thanks for posting.

I think the abridgement sounds nice but don't anticipate it affecting me much either way.

I think the ability to turn this on/off in user preferences is a particularly good idea (as mentioned in Raemon's comment).

I can follow most of this, but i'm confused about one part of the premise.

What if the agent created a low-resolution simulation of its behavior, called it Approximate Self, and used that in its predictions? Is the idea that this is doable, but represents a unacceptably large loss of accuracy? Are we in a 'no approximation' context where any loss of accuracy is to be avoided?

My perspective: It seems to me that humans also suffer from the problem of embedded self-reference. I suspect that humans deal with this by thinking about a highly approx... (read more)

1Liam Donovan
In order to do this, the agent needs to be able to reason approximately about the results of their own computations, which is where logical uncertainty comes in

It's relevant to some forms of utilitarian ethics.

3Said Achmiz
That is an answer to a different question—shminux did not ask why it’s relevant, but why it’s meaningful. In other words, the question is not about the desirability of having simple aggregate measures of human experience, but about the possibility of their existence.

I think this is a clever new way of phrasing the problem.

When you said 'friend that is more powerful than you', that also made me think of a parenting relationship. We can look at whether this well-intentioned personification of AGI would be a good parent to a human child. They might be able to give the child a lot of attention, a expensive education, and a lot of material resources, but they might take unorthodox actions in the course of pursuing human goals.

(I'm not zhukeepa; i'm just bringing up my own thoughts.)

This isn't quite the same as a improvement, but one thing that is more appealing about normal-world metaphilosophical progress than empowered-person metaphilosophical progress is that the former has a track record of working*, while the latter is untried and might not work.

*Slowly and not without reversals.

It implies that the Occamian prior should work well in any universe where the laws of probability hold. Is that really true?

Just to clarify, are you referring to the differences between classical probability and quantum amplitudes? Or do you mean something else?

3zulupineapple
Not at all. I'm repeating a truthism: to make a claim about the territory, you should look at the territory. "Occamian prior works well" is an empirical claim about the real world (though it's not easy to measure). "Probabilities need to be multiplied" is a lot less empirical (it's about as empirical as 2+2=4). Therefore the former shouldn't follow from the latter.

Why do you think so? It's a thought experiment about punitive acausal trade from before people realized that benevolent acausal trade was equally possible. I don't think it's the most interesting idea to come out of the Less Wrong community anymore.

Noted!

Sorry, i couldn't find the previous link here when i searched for it.

Just to be clear, i'm imagining counterfactual cooperation to mean the FAI building vaults full of paperclips in every region where there is a surplus of aluminium (or a similar metal). In the other possibility branch, the paperclip maximizer (which thinks identically) reciprocates by preserving semi-autonomous cities of humans among the mountains of paperclips.

If my understanding above is correct, then yes, i think these two would cooperate IF this type of software agent shares my perspective on acausal game theory and branching timelines.

In the last 48 hours i've felt the need for more than one of the abilities above. These would be very useful conversational tools.

I think some of these would be harder than others. This one sounds hard: 'Letting them now that what they said set off alarms bells somewhere in your head, but you aren’t sure why.' Maybe we could look for both scripts that work between two people who already trust each other, and scripts that work with semi-strangers. Or scripts that do and don't require both participants to have already read a specific blog post, etc.

1meedstrom
Why can't you say it word-for-word? "What you said set off alarm bells in my head and I'm not sure why".
1Hazard
I think the script for that one needs two parts for it to work. The first is this-problem-specific and is conveying the belief that "People don't automatically have access to their motives, and it's super easy for one to confabulate their motives." I've got a feeling that to really get someone to understand that point would require at least some reading on the topic. Actually, you might need to pair this one with a tangent explaining this idea. The second ingrediant seems to be a more generic one, and it's establishing the rule that "Us disagreeing with each other doesn't mean we have to be on opposite teams." That second one is probably the more important part when interacting with a semi-stranger.

Something like a death risk calibration agency? Could be very interesting. Do any orgs like this exist? I guess the CDC (in the US govt) probably quantitively compares risks within the context of disease.

One quote in your post seems more ambitious than the rest: 'helping retrain people if a thing that society was worried about seems to not be such a problem'. I think that tons of people evaluate risks based on how scary they seem, not based on numerical research.

Note on 3D printing: Yeah, that one might take a while. It's actually been around for decades, but still hasnt become cheap enough to make a big impact. I think it'll be one of those techs that takes 50+ years to go big.

Source: I used to work in the 3D printer industry.

2Elo
I actually see the quality being the bigger problem than the price. You just can't get some materials properties that you can get out of other manufacturing methods unless you can layer on atoms to get perfect microstructures. Which is exorbitantly difficult. Source: was a materials science student

I first see the stems, then i see the leaves.

I think humans spend a lot of time looking at our models of the world (maps) and not that much time looking at our actual sensory input.

A similar algorithm appears in Age of Em by Robin Hanson ('spur safes' in Chapter 14). Basically, a trusted third party allows copies of A and B to analyze each other's source code in a sealed environment, then deletes almost everything that is learned.

A and B both copy their source code into a trusted computing environment ('safe'), such as an isolated server or some variety of encrypted VM. The trusted environment instantiates a copy of A (A_fork) and gives it B_source to inspect. Similarly, B_fork is instantiated and allowed to

... (read more)
1bryjnar
I haven't read Age of Em, but something like "spur safes" was an inspiration (I'm sure I've come across the idea before). My version is similar except that 1. It's stripped down. 1. B only needs to make a Validator, which could be a copy of themself, but doesn't have to be. 2. It only validates A to B, rather than trying to do both simultaneously. You can of course just run it twice in both directions. 2. You don't need a trusted computing environment. I think that's a pretty big deal, because the trusted computing environment has to be trusted enough to run its end of A/B's secure channels. In order for A/B to trust the output, it would need to e.g. be signed by their private keys, but then the computing envionment has access to those keys and can do whatever it wants! The trick with FHE is to let B run a computation using their secret key "inside" the safe without letting anyone else see the key.

Favorite highlight:

'Likewise, great literature is typically an integrated, multi-dimensional depiction. While there is a great deal of compression, the author is still trying to report how things might really have happened, to satisfy their own sense of artistic taste for plausibility or verisimilitude. Thus, we should expect that great literature is often an honest, highly informative account of everything except what the author meant to put into it.'

The techniques you outline for incorporating narrow agents into more general systems have already been demoed, I'm pretty sure. A coordinator can apply multiple narrow algorithms to a task and select the most effective one, a la IBM Watson. And I've seen at least one paper that uses a RNN to cultivate a custom RNN with the appropriate parameters for a new situation.

I'm updating because I think you outline a very useful concept here. Narrow algorithms can be made much more general given a good 'algorithm switcher'. A canny switcher/coordinator program can be given a task and decide which of several narrow programs to apply to it. This is analogous to the IBM Watson system that competed in Jeopardy and to the human you describe using a PC to switch between applications. I often forget about this technique during discussions about narrow machine learning software.

Load More