I don’t find it that mysterious ? That "Stop !" is obviously, nakedly a call to the superego. The authority you’re seeking is the same authority that shapes the superego and is social in nature. It’s an appeal to the shared values that the society is expecting its members to have.
Pretty sure people that are using the "demanding" objection are not articulated analytical philosophers. They’re just trying to communicate their intuitive objection, not making a theoretical claim. Or : demandingness is not a definitive proof that it’s false, demandingness is intuitive evidence false.
And I think I can point to a less instrumental reason and a more theoretical one ?
It is well accepted that from the Universe (or God, or Objective, or whatever) point of view, everyone has the same moral weight (1). Let’s take that for granted.
Then utilitarians go around and say "therefore, using morality to guide you action, you should weight every human being equally" (2).
But (2) does absolutely not follow from (1), and I think the "demanding" objection is actually a objection to (2). It’s the good old question of "is distance relevant for moral considerations ?" (physical and/or in the social graph), and those saying "too demanding" are those answering "yes" against your "no" to that old question.
I don't understand how it is an answer to gradual disempowerement.
In the GD scenario, both individual humans and AI understand that :
The problem is not in the understanding. It is in the incentives. The threat model was never "oops, we did not understood that dynamics, and now we are disempowered… if only we knew…". We already know before it happens — that’s the point of the original GD paper. Having an LLM understand and explain that dynamics does not help avert that scenario ?
You’re missing the point that many, many humans will reply like the LLMs. "It’s complicated". Given that, I fail to see it as a "clear misalignment with respect to human values" problem rather than "what are those humans values in the first place ?" problem.
I think it was correct to ask the question, because, no, sorry, I don’t think you can equate that specific understanding of "property is downstream of power" with the same level of consensus/reasonableness as "prices are downstream of supply and demand", whereas the understanding given by Viliam is pretty much inside of what I would expect to be as consensual as "prices are downstream to supply and demand".
For example, regarding your second point : I think most economists would reply that that distinction is fallacious when considering creative destruction, both mistake (having a more correct view of the market and business organization) and conflict (and thereby driving your competitors bankrupt, forcing the... (read more)
I wish that both the "tip top" and the "medium top" of educated people had consensus that both "property is downstream of power" and "prices are downstream of supply and demand".
I really don’t like that quote. I can see multiple interpretations for "property is downstream of power", many of them subject to debate, and none of the firm grounding (and non-ambiguity) of Econ 101 enjoyed by "price are downstream of supply and demand".
It looks like to me a false equivalence, trying to say "see ? both sides clearly have a point", where one point is actually clear, and the other isn’t so clear.
Would you mind clarifying what the consensus should be exactly on this point ? Maybe I’m just not interpreting "property is downstream of power" correctly.
very bad fates for all humans
I believe there’s also a disagreement here where the same scenario will be considered fine by some and very bad by others (humans as happy pets comes to mind).
How to generalize to multiple humans is... not an unimportant question, but a question whose salience is far, far out of proportion to its relative importance
I expect it to be the hardest problem, not from a technical point of view, but from a lack of ground truth.
The question "how do I model the values of a human" has a simple ground truth : the human in question.
I doubt there’s such a ground truth with "how do I compress the values of all humans in one utility function ?". "All models are wrong, some are useful", and all that, except all the different humans have a different opinion on "useful", ie their own... (read more)
Failure in itself is valuable to you?
What I sense from this is that what you’re not getting is that my value system is made of tradeoff of let’s call it "Primitive Values" (ie one that are at least sufficiently universal in human psychology that you kind of can describe them with compact words).
I obviously don’t value failure. If I did I would plan for failure. I don’t. I value/plan for success.
But if all plans ultimately lead to success, what of use/fun/value is planning ?
So failure has to be part of the territory, if I want my map-making skills to… matter ? make sense ? make a difference ?
... (read 1492 more words →)It feels to me like
[Epistemic Status: internally strongly convinced that it is centrally correct, but only from armchair reasoning, with only weak links to actual going-out-in-the-territory, so beware: outside view tells it is mostly wrong]
I have been binge-watching the excellent Dwarkesh Patel during my last vacations. There is, however, one big problem in his AI-related podcasts, a consistent missing mood in each of his interviewees (excepting Paul Christiano) and probably in himself.
"Yeah, AI is coming, exciting times ahead", say every single one, with a bright smile on their face.
The central message of this post is: the times ahead are as exciting as the perspective of jumping out of a plane without a parachute. Or how "exciting... (read 2603 more words →)
That bar for "good enough" may also be above "unacceptable", requiring eusocial levels of coordination where individuals are essentially drones.