All of [deactivated]'s Comments + Replies
It's also false; there were lots of replies.
There were comments on Facebook, to be sure, but I never saw anyone (except me) reply to my comment here on LessWrong, ever after (what felt like) several days.
For anyone curious, you can view the original comment here.
You might be surprised!
You might want to add (1.5) also evaluate whether what's going on is that some group of people wants to be referred to differently, and then (2') generally don't resist in that case even if no harm is apparent, because (a) maybe there's harm you haven't noticed and (b) giving people what they want is usually good. I'd certainly be on board with that. (I suspect Scott would too.)
I think this is pretty much my argument. I think Scott wouldn't agree because he wrote:
...On the other hand, the people who want to be the first person in a new cascade, like USC’s soc
The roots of "Black" go back further than 1966. For example, here are two excerpts from Martin Luther King's "I Have a Dream" speech in 1963 (emphasis mine):
When the architects of our republic wrote the magnificent words of the Constitution and the Declaration of Independence, they were signing a promissory note to which every American was to fall heir. This note was a promise that all men — yes, Black men as well as white men — would be guaranteed the unalienable rights of life, liberty and the pursuit of happiness.
...And when this happens, and when we
I think this is the crux of the matter:
...Was the process in this case a bad thing overall, as we should probably expect on Scott's model? (Bad: risk of mis-classifying people as racist whose only sin was not to adjust their language quickly enough; inconvenience during the transition; awkwardness after the transition of reading material written before it. Good: morale-boosting effects on black people of feeling that they were using a term of their own choosing and taking more control of their own destiny; if SC/KT was correct about "negro" bringing along unw
I commented on Scott’s blog post with a link to this post.
The post makes the claim hyperstitious cascades are bad, where previously innocent words that noone took offense to become taboo
A major claim I’m making is that this has never actually happened in history, and certainly not in any of the examples Scott uses. Words become taboo because they are used offensively.
Out of the two options, this is closer to my view:
"this is completely and horribly incorrect in approach and model"
I think Scott’s model of how changes in the words we use for minority groups happen is just factually inaccurate and unrealistic. Changes are generally slow, gradual, long-lasting, and are primarily advocated for in good faith by conscientious members of the minority group in question.
My contention is that this model of the process is basically just wrong for the examples of minority group labels that have actually caught on.
There's a difference between "labels should never change" (which you say Scott is saying, but he isn't) and "the thing where a previously harmless label becomes offensive, not because it was really offensive all along, but because someone has decided to try to make it offensive and then there's a positive feedback loop, generally does more harm than good" (which he is saying).
Per (4) in the OP, I think this process that Scott describes is simply an incorrect model of why some words for minority groups come to be seen as derogatory and why the acceptable...
Potential solutions to foreseeable problems with biological superintelligence include: a) only upgrading particularly moral and trustworthy humans or b) ensuring that upgrading is widely accessible, so that lots of people can do it.
I guess we could say governance remains a problem with biological superintelligence? As it does with normal humans, just more so.
Beautifully written! Great job! I really enjoyed reading this story.
in comparison to a morally purified version of SimplexAI,we might be the baddies."
Did you link to the wrong thing here or is there some reference to generative grammar I'm not getting?
Most of the writing on ethics in our society and its cultural forebears over the last few millennia has been written by moral absolutists: people who believed that there is one, true, and correct set of ethics, and were either trying to figure out what it is, or more often thought they already have, and are now trying to persuade others.
This is not my understanding of moral absolutism. One definition from a University of Texas blog:
...Moral absolutism asserts that there are certain universal moral principles by which all peoples’ actions may be judged.
It seems too long for a comment. Also, it uses markdown formatting.
Again that's not Scotts point. Scott is concerned about deliberate attempts to rapidly make a perfectly innocent word taboo, causing bother and potential ostracism to everyone for no reason, not natural long term evolution of words.
I don't think such an attempt has ever happened and succeeded. I'm open to counterexamples, though.
...The problem isn't that most of the black people in the USA got together and said they prefer to be called black. It's that due to a single bad actor making up a fake history for an innocent word, lots of old grandpas get ostra
If someone in the US uses the word Jew and they're not obviously Jewish, they sound antisemitic.
This seems not universal and highly context-dependent.
Honestly, saying his examples ("asian" and "field work") are worse than yours ("black" and "gay") is very close to strawman arguing.
Well, my examples are both real and non-fringe, whereas "Asian" and "field work" are fictional and fringe, respectively. So, I think "gay" and "Black" are more central examples.
Scott also seems annoyed by "Black", but doesn't explain why he's (seemingly) annoyed.
There's a bit more here than I can readily respond to right now, but let me know if you think I've avoided the crux of the matter and you'd like me to address it in a future comment.
This doesn't apply to more central cases like "gay" and "Black".
Fair! I should have said 1,000 years to make the point more clear-cut.
It would be much more helpful if Scott used a real example rather than a fictional one. I don't think his fictional example is very realistic.
Thanks!
Thanks for posting this. I am still a bit fuzzy on what exactly the Superalignment plan is, or if there even is a firm plan at this stage. Hope we can learn more soon.
I think that me not wearing shoes at university is evidence that I might also disdain sports, but not evidence that I might steal.
it is not actually the case that violating one specific social norm for specific reason is a substantial update that someone is a Breaking Social Boundaries Type Pokemon in general.
If I can attempt to synthesize these two points into a single point: don't assume weird people are evil.
If someone walks around barefoot in an urban environment, that's a good clue they might also be weird in other ways. But weird ≠ evil.&...
Longtermism question: has anyone ever proposed a discount rate on the moral value of future lives? By analogy to discount rates used in finance and investing.
This could account for the uncertainty in predicting the existence of future people. Or serve as a compromise between views like neartermism and longtermism, or pro-natalism and anti-natalism.
Now he’s free to run for governor of California in 2026:
...I was thinking about it because I think the state is in a very bad place, particularly when it comes to the cost of living and specifically the cost of housing. And if that doesn’t get fixed, I think the state is going to devolve into a very unpleasant place. Like one thing that I have really come to believe is that you cannot have social justice without economic justice, and economic justice in California feels unattainable. And I think it would take someone with no loyalties to sort of very powerful
William Nordhaus estimates that firms recover maybe 2% of the value they create by developing new technologies.
Isn’t this the wrong metric? 2% of the value of a new technology might be a lot of money, far in excess of the R&D cost required to create it.
I think you are way overestimating your ability to tell who is trans and way underestimating the ability of trans people to pass as cis. Sometimes, you just can’t tell.
What on Earth? Why does it require being “devious” to be in the closet? If you were given a choice between lifelong celibacy and loneliness, on the one hand, or, on the other hand, seriously endangering yourself, risking being imprisoned or institutionalized, and ruining your life (economically and socially) by having relationships and disclosing them, would it make you “devious” to choose a third option and keep your relationships secret?
Were Jews who hid from the Nazis “devious”? Were people who helped them hide “devious”? Only in a sense that drains the...
Most 'trans-women' I see online (in videos etc) who expliciltly identify themselves as such do not, to my eye, visually pass as women, even if I try and account for the fact that knowledge of their identity could skew my perception. It would be weird that every one of them I encounter IRL would fall into the minority of who clearly pass.
Bezzi also said "meet", not just "see", which implies talking to the person, and I find that the numebr of 'trans-women' I see online who have voices that pass is even smaller than those who visually pass. It therefore seems extremely unlikely that hundreds of the people I've met in my city who I thought were women were actually trans-women.
1
1He said:
At a high level you can think of Gemini as combining some of the strengths of AlphaGo-type systems with the amazing language capabilities of the large models
What do you think he meant by "AlphaGo-type systems"? I could be wrong, but I interpreted that as a reference to RL.
This seems super important to the argument! Do you know if it's been discussed in detail anywhere else?
We are on track to build many superhuman AI systems. Unless something unexpectedly good happens, eventually we will build one that has a failure of inner alignment. And then it will kill us all. Does the probability of any given system failing inner alignment really matter?
Yes, because if the first superhuman AGI is aligned, and if it performs a pivotal act to prevent misaligned AGI from being created, then we will avert existential catastrophe.
If there is a 99.99% chance of that happening, then we should be quite sanguine about AI x-risk. On the other hand, if there is only a 0.01% chance, then we should be very worried.
I have a question about "AGI Ruin: A List of Lethalities".
These two sentences from Section B.2 stuck out to me as the most important in the post:
...outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction.
...on the current optimization paradigm there is no general idea of how to get particular inner properties into a system, or verify that they're there, rather than just observable outer ones you can run a loss function over.
My question is: supposing this is all true, what is the probabil...
I don't know if anyone still reads comments on this post from over a year ago. Here goes nothing.
I am trying to understand the argument(s) as deeply and faithfully as I can. These two sentences from Section B.2 stuck out to me as the most important in the post (from the point of view of my understanding):
...outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction.
......on the current optimization paradigm there is no general idea of how to get particular inner properties into a system, or ve
Wolfram's article is very confusing indeed.
Most important sentence:
A reward function reshapes an agent's cognition to be more like the sort of cognition that got rewarded in the training process.
Wow. That is a tremendous insight. Thank you.
On another topic: you quote Yudkowsky in 2008 expressing skepticism of deep learning. I remember him in 2016 or 2017 still expressing skepticism, though much more mildly. Does anyone else recall this? Better yet, can you link to an example? [Edit: it might have been more like 2014 or 2015. Don’t remember exactly.]
I guess sort of the point of this post is that, in the broadest sense, the political critique of so-called “TESCREAL” lacks imagination — about the possible connections between these -isms and social justice.
I don’t think there’s anything inherently disparaging about the acronym.
Thanks for your comment.
...note that indefinite life extension, reversing the aging process, etc, have never become a public priority in any polity.
Is this really strong evidence for anything? For example, the Methuselah Foundation was founded in 2001 and the SENS Research Foundation was founded in 2009. Calico was founded in 2013. Altos Labs was founded in 2021. All this to say, the science of radical life extension is extremely new. There hasn't been much time for life extension to become a political cause.
...One motivation of the left is to lift up o
"Someone gives a lot of compliments to me but I don't think they're being genuine"
Au contraire. This is a common tactic of manipulation and abuse.
"I feel 'low-value'"
I think the point is that they were treated as low-value by their bosses.
Wish I knew why this post is getting downvoted to karma hell! :(
I meant "extract" more figuratively than literally. For example, GPT-4 seems to have acquired some ability to do moral reasoning in accordance with human values. This is one way to (very indirectly) "extract" information from the human brain.
Extract from the brain into, say, weights in an artificial neural network, lines of code, a natural language "constitution", or something of that nature.
...I think the human brain’s intrinsic-cost-like-thing is probably hundreds of lines of pseudocode, or maybe low thousands, certainly not millions. (And the part that’s relevant for AGI is just a fraction of that.) Unfortunately, I also think nobody knows what those lines are. I would feel better if they did.
So, the human brain's pseudo-intrinsic cost is not intractably complex, on your view, but difficult to extract.
I think this functionality already exists.
Next to “Latest Posts” on the front page there is a “Customize Feed” button. You can set the topics “AI” and “AI risk” to “Hidden” and set the topic “Rationality” to “Promoted”.
Hope that helps.
There are so, so many examples of things like this that happen all the time in the world. I used two hypothetical examples in the post. I thought that would suffice.