All of Viliam's Comments + Replies

Viliam148

Yeah, there are two different cultures, and it is important to know which one is at your job. For some people, following this advice could cost them their jobs.

Viliam23

Yeah, same here. This feels like a crossover between the standard Buddhist woo and LLM slop, sprinkled with "quantum" and "Gödel". The fact that it has positive karma makes me feel sad about LW.

Since it was written using LLM, I think it is only fair to ask LLM to summarize it:

Summary of "Eliminative Nominalism"

This philosophical essay introduces Eliminative Nominalism (EN), a theoretical framework that extends Eliminative Materialism (EM) to address what the author considers its foundational oversight: the conflation of first-order physical description wit

... (read more)
0milanrosko
"Since it was written using LLM" "LLM slop." Some of you soooo toxic. First of all, try debating an LLM about illusory qualia—you'll likely find it—attributing the phenomenon to self-supervised learning—It has a strong bias toward Emergentism, likely stemming from... I don't know, humanities slight bias towards it's own experience. But yes, I used LLM for proofreading. I disclosed that, and I am not ashamed of it.
0milanrosko
That concern is understandable — and in fact, it’s addressed directly and repeatedly in the text. The argument doesn't claim that humans are formal proof systems in a literal or ontological sense. Rather, it explores how any system capable of symbolic self-modeling (like the brain) inherits formal constraints analogous to those found in expressive logical systems — particularly regarding incompleteness, self-reference, and verification limits. It's less about reducing humans to Turing machines and more about using the logic of formal systems to expose the structural boundaries of introspective cognition. You’re also right to be skeptical — extraordinary claims deserve extraordinary scrutiny. But the essay doesn’t dodge that. It explicitly offers a falsifiable framework, makes empirical predictions, and draws from well-established formal results (e.g. Gödel, Conant & Ashby) to support its claims. It’s not hiding behind abstraction — it’s leaning into it, and then asking to be tested. And sure, the whole thing could still be wrong. That’s fair. But dismissing it as “crackpottery” without engaging the argument — especially on a forum named LessWrong — seems to bypass the very norms of rational inquiry we try to uphold here. If the argument fails, let’s show how — not just that. That would be far more interesting, and far more useful.
2Rafael Harth
I agree that this sounds not very valuable; sounds like a repackaging of illusionism without adding anything. I'm surprised about the votes (didn't vote myself).
Viliam20

Is it possible that the AI was actually told in the prompt to generate those specific answers?

(People on internet do various things just to get other people's attention.)

2Daniel Tan
Definitely possible, I’m trying to replicate these myself. Current vibe is that AI mostly gives aligned / boring answers
Viliam93

A few months ago I complained that automatic translation sucks when you translate between two languages which are not English, and that the result is the same as if you translated through English. When translating between two Slavic languages, even sentences where you practically just had to transcribe Cyrillic to Latin and change a few vowels, both Google Translate and DeepL succeeded to randomize the word order, misgender every noun, and mistranslate concepts that happen to be translated to English as the same word.

I tried some translation today, and fro... (read more)

Viliam21

What would be your proposed alternative to Harry Potter fanfiction? Something else fan fiction? Original fiction? Not a fiction?

As I see it, choosing the Harry Potter universe that many readers already know allows it to use the contrast between how the different versions of Harry Potter behave.

Harry Potter in the books is frankly an idiot. Every year the most powerful dark wizard is trying to murder him, and he always only survives because he gets lucky... and yet the only thing he worries about is quidditch. Voldemort doesn't seem very smart either; his o... (read more)

1Mohit Gore
I don't dislike the decision to make it Harry Potter fanfiction; I agree with everything you're saying about the original series and I think it fits with Yudkowsky's goal. My gripe is with how diluted the fanfiction element is in HPMOR. So far, the dialogue is stiff and uninteresting, the pacing is unbelievably dull, the story moves in sudden stops and starts, and our protagonist is insufferable. It seems as though the fiction part of the project was a hand-waved afterthought, which is strange because it's the thing that houses the thought the series is meant to inspire in the first place. Looking online, it seems like most members of the rationalist community adore HPMOR and most members of the fanfiction community despise it. That makes sense; Yudkowsky's goal does not seem to be to write an effective piece of fiction, but rather to communicate personal philosophy... but if he was going to use the medium of fanfiction at all, I don't see the point in forgetting about its standards entirely. If he wasn't going to adapt to the medium he chose, why use it at all?
Viliam20

Are there many people who pay for 3 Saturdays and then skip one? I would be surprised.

What age is the target group? An adult person can probably easily find 3 free Saturdays in a row. For a student living with parents it will probably be more difficult, because it means 3 weekends when the parents cannot organize any whole-weekend activity.

Viliam100

How about "make computers stupid again"?

Viliam20

The actions and statements of this current Trump administration show more support than ever for bold governance such as revitalizing letters of marque.

Against Russia? (As far as I know, most cyber attacks come from there.) In my opinion, unlikely.

Viliam5-1

I suspect that many people in the rationalist community have a blind spot about prediction markets, and see them as some kind of cooperative effort to make true predictions.

Instead, from the financial perspective, they are zero-sum games, and the best players play accordingly. If making the correct prediction is the winning move, so be it. If it is something else, that works, too.

I suspect that anonymous prediction markets (where people cannot get famous as superforecasters, only either gain or lose money) would make even better predictions that the current ones where many people have a conflict of interest.

3Petropolitan
Isn't Polymarket already anonymous?
Viliam20

I don't think I'm interested. You didn't update at all based on our previous bet.

That should make you more interested (financially) in betting against the person.

2Ape in the coat
You may assume that it's the way how Albert managed to persuade Barry to continue)
Viliam20

So, as I see it, the best case is when the skill degrades gracefully (provides the benefits even if other people are unaware of it or doing it wrong); and the second best case is if it has tests, so you know when you can safely use it, and when you need to switch to some plan B.

In case of rationality, I think there is "individual rationality" and "group rationality". Some things you can do alone, for example keep a diary of your predictions. You can get more benefit from talking to other rational people, but there is also the risk that they turn out to be ... (read more)

4Screwtape
Even if a skill isn't as useful if you're the only one to know it, if the skill is still somewhat useful that can work. I like literacy as an example; crazy good if most people have it, still useful if only you have it, usually obvious pretty quickly if other people don't have it. Individual and group rationality are pretty relevant here. In a sense, one thing I'm pointing at is a way to bootstrap (some) rationality skills from the easier individual domain in to the harder group domain; focus on places where the same skill is relevant in both arenas. It's also a small argument in favour of following a textbook; mandatory education is one of society's big shots on putting skills in everyone's heads and it might not be worth (making numbers up) a 50% boost to this one classroom overall if it means they happen to miss a particular skill that the rest of society is going to expect everyone to have. Still, that side of things is more tentative.  (One of my favourite questions to ask rationalists is "if you could pick one rationalist skill and make it as common as literacy, what do you pick?")
Viliam40

I don't have a coherent theory of motivation, because when I look at myself, it seems that different parts of my life work quite differently.

For example, when I used to write SF, the social motivation was very important. I had friends who liked SF books and movies, we talked about that a lot, many of us hoped to write something good one day. So I had a community, and an audience. My first attempts had some small success, which inspired me to work harder and achieve more. And then... at some moment this all went down in flames... skipping the unimportant de... (read more)

6cousin_it
Your examples sound familiar to me too, but after rereading your comment and mine, maybe it all can be generalized in a different way. Namely, that internal motivation leads to a low level of effort: reading some textbooks now and then, solving some exercises, producing some small things. It still feels a bit like staying in place. Whereas it takes external motivation to actually move forward with math, or art, or whatever - to spend lots of effort and try to raise my level every day. That's how it feels for me. Maybe some people can do it without external motivation, or maybe they lucked into getting external motivation in the right way, I don't know.
Viliam20

Is it possible that the relation between GLP-1 and willpower is basically about willpower depletion? The more mental energy you spend fighting your urge to eat, the less is left for everything else. GLP-1 reduces the hunger, suddenly you have more willpower for everything else.

2Mateusz Bagiński
Then the effect would be restricted to people who are trying to control their eating which we would probably have heard of by now.
Viliam20

I suspect this may be related to the feedback one gets. Importantly, not just feedback on having accomplished something, but also on working towards something even if you are not there yet -- because this is where you will realistically spend most of your time when working on nontrivial projects.

Writing is probably easy (for an intelligent person) if you have a friendly audience. The question is how to get it before you learn how to write well. Sometimes, the parents provide the service.

4Mateusz Bagiński
Relevant previous discussion: https://www.lesswrong.com/posts/XYYyzgyuRH5rFN64K/what-makes-people-intellectually-active 
5cousin_it
I agree feedback is a big part of it. For example, the times in my life when I've been most motivated to play musical instruments were when I had regular opportunities to play in front of people. Whenever that disappeared, the interest went away too. But also I think some of it is sticky, or due to personality factors. We could even say it's not about willpower at all, but about value differences. Some people are just more okay with homeostasis, staying at a certain level (which can be lower or higher for different people) and using only as much effort as needed for that. While others keep climbing and applying effort without ever reaching a level that lets them relax. Many billionaires seem to be of that second type. I'm more of the first type, with many of my active periods being prompted by external changes, threats to homeostasis. It's clear that type 2 achieves more than type 1, but it's not clear which type is happier and whether one should want to switch types.
Viliam40

I am not entirely sure what specific thing were the rationalists wrong about (the quotes are about various things) and what specifically is the correct version we should update to.

For example, Eliezer's quote seems to be about how China would prefer a narrow AI (that can be controlled by the Chinese Communist Party) over a general AI, for completely selfish reasons. Do you believe that this is wrong?

3thedudeabides
"Various things" - ugh ok how about this ONE. "No...There is no appreciable risk from non-Western countries whatsover" - @Connor Leahy 
Viliam40

A general source of problems is that when people try to get a new partner, they try to be... more appealing than usual, in various ways. Which means that after the partner is secured, the behavior reverts to the norm, which is often a disappointment.

One way how people try to impress their partners is that the one with lower sexual drive pretends to be more enthusiastic about sex than they actually are in long term. So the moment one partner goes "amazing, now I finally have someone who is happy to do X every day or week", the other partner goes "okay, now ... (read more)

Viliam1012

Put more simply, could we posit that an omnibenevolent God might care more about what kind of person we become than what circumstances we endure?

If you redefine "benevolent" to mean someone who doesn't care about suffering, we are no longer speaking the same language.

Why is so much suffering needed to figure out "what kind of person we become"? Couldn't less sadistic circumstances answer this question just as well?

Also, many people die as little kids, so they apparently don't get a chance to become any kind of person.

"Doesn't exist, or doesn't give a fuck about suffering" is the answer that matches the data, sorry.

1Eggs
I agree with you. (Though I might rephrase the second as 'doesn't care about suffering the way we do'. Either way, your point is valid.)  My point wasn't to say 'doesn't exist' is wrong, but that there is more than one possibility. If you or anyone has taken the time to evaluate the possibilities and come to the conclusion that 'doesn't exist' is the more likely / simple / predictive model, then I commend you. That is what rationality is about.  All I ask is the same courtesy as I might be exploring a different set of models than you are. 
Viliam40

Occam's razor says that Trump makes populist statements that appeal to the kind of person who votes for Trump. Whether those are actually good ideas is irrelevant. Mercantilism sounds good to a person who doesn't know much about economy. Territorial expansion rhymes with "make great".

21st-century economics is just as irrelevant as e.g. 21-st century medicine. Scientists are not an important group of voters.

Viliam218

This can be quite frustrating if you want to figure out "what happens if I do X", and all the answers provided by science turn out to be about "what happens if people kinda want to do X, but then most of them don't".

I mean, it is good and potentially important to know that most people who kinda want to do X will fail to actually do it... but it doesn't answer the original question.

Viliam40

a majority of long-term monogamous, hetero relationships are sexually unsatisfying for the man after a decade or so.

This seems supported by the popular wisdom. Question is, how much this is about relationships and sex specifically, and how much it is just another instance of a more general "life is full of various frustrations" or "when people reach their goals, after some time they became unsatisfied again" i.e. hedonistic treadmill.

sexual satisfaction is basically binary

Is it?

most women eventually settle on a guy they don't find all that sexually attract

... (read more)
7VivaLaPanda
Worth noting that this pattern occurs among gay couples as well! (i.e. sexless long-term-relationship, where one party is unhappy about this).  I think that conflict in desires/values is inherent in all relationship, and long-term-relationships have more room for conflict because they involve a closer/longer relationship. Sex drive is a major area where partners tend to diverge especially frequently (probably just for biological reasons in het couples).  It's not obvious to me that sex in marriages needs much special explanation beyond the above. Unless of course the confusion is just "why don't people immediately end all relationships whenever their desires conflict with those of their counterparty".
Viliam20

This could be addressed by making a user interface which not only gives the user's prompt to the LLM, but also provides additional instructions and automatically asks additional questions. The answers to those additional questions could be displayed in smaller font as a side note, or maybe as graphical icons. One such question would be "in this answer, did you simplify things? if yes, tell me a few extra things I could pay attention to in order to get a better understanding of the topic" or something like that.

2Gunnar_Zarncke
This is an interesting UI proposal and, if done right, might provide the needed transparency. Most people wouldn't read it, but some would, esp. for critical answers.
Viliam42

If you know a fact about humans, then mammals are not important. Humans like stories. Doesn't matter if mammals in general don't.

If you don't know a fact about humans, but somehow you know the fact about mammals, you can use it as evidence (although not as a proof). For example, in a culture with a strong religious taboo against human autopsy they could dissect various mammals, and make probabilistic statements about human anatomy.

Today, a more typical situation is two groups of people, each declaring that they know for a fact that humans are / are not X. ... (read more)

Viliam20

Good point.

I think there are still two different topics, although with some overlap: how to budget and how to get rich. Good budgeting is good, whether you are poor or rich. If you are poor, it can help avoid losing everything. If you are rich, it can help avoid wasting all your money and becoming non-rich.

But the (implied) idea that budgeting can make poor people rich, or that it is the main force that keeps poor and rich apart... that does not automatically follow, and actually many people doubt it. Hypothetically they may be wrong, but this needs to be ... (read more)

2Davidmanheim
I mostly agree with you... but: 1. You created a false dichotomy where budgeting excludes investing, then said you can't ever make money with budgeting, because by definition anything that is in your budget cannot make money - but spending on a house instead of rent, for example, clearly violates that assumption. 2. Budgeting can include time, in addition to money, and among other things, that matters because income isn't a fixed quantity over time. Things that people can do to use time to drastically change income include starting businesses, or taking night classes to get a more lucrative job. 3. The last point is conflating questions, because yes, inheriting or a trust fund is already being rich (but inheritances and trust funds are not the same thing!) However, most second and third generation nouveau riche folks do, in fact, spend down and waste the inherited fortune, once they are old enough to actually inherit instead of living on a managed trust fund.
Viliam42

Then maybe I can link to those posts in a larger post

Yes, this seems to me like a good strategy for posting on LW. Start with smaller, then generalize (and link to previous posts when needed).

One advantage is that when things go wrong -- if one of the smaller articles is strongly rejected -- it gives you an opportunity to stop and reflect. Maybe you were wrong, in which case it is good that you didn't write the more general article (because it would be downvoted). Maybe the LW readers were wrong, but that still means that you should communicate your (small... (read more)

Viliam40

I would suggest choosing a less grandiose topic. Something more specific; perhaps something that you know well. ("What is true of one apple may not be true of another apple; thus more can be said about a single apple than about all the apples in the world." -- source)

As a reader I prefer it when the posts are self-contained; when I get a value from the post even without clicking any of the links. The information linked should be optional to the experience.

Looking at the topics of my posts... books I have read (1, 2, 3), things happening in the rationality ... (read more)

3Oxidize
Thanks for the advise. I see how the linked posts are a lot more specific than the one I made. I'll try making some posts confined to specific domains of psychology, maybe in a very detailed & rational structure. Then maybe I can link to those posts in a larger post where I use those understandings/pieces of information to make a claim about a vehicle for using the information for practical change in the real world. I'm not sure I'm capable of giving up on macro-directional efforts like attempts to improve humanity as a whole, but I'll try and change the way I structure writings to be self-contained and linked externally for supplemental information as opposed to the entire post being dependent on a linked doc.
Viliam31

feedback expressing painpoints/emotion is valuable, whereas feedback expressing implementation/solutions is not.

Yep. Or, let's say that the kind of feedback that provides solutions is worthless 99% of time. Because it is possible in principle to provide a good advice, it's just that most people do not have the necessary qualification and experience but may be overconfident about their qualification.

I find it ironical that popular wisdom seems to go the other way round, and "constructive criticism" is praised as the right thing to do. Which just doesn't mak... (read more)

Viliam199

I agree that something being "natural" doesn't make it "right". But "natural" is still a force of nature that you have to include in your predictions... unless you are okay with getting your predictions systematically wrong.

Applying this to the examples you mentioned:

Males are more aggressive in most animals; therefore, male humans should be expected to be more aggressive.

Yes, 100% agree. Notice that this doesn't say anything about aggression being good or otherwise desirable; the statement is positive, not normative.

Do you have an alternative explanation ... (read more)

1CrimsonChin
Thanks you prodded at a real issue I failed to notice in my post. I claim the article is about: "natural does not mean good" Then I go off and seem to try to make a sneaky second claim: "Natural processes can't be proven/verified easily and should hold no weight" With the second claim being a lot weaker. But I still stand by it. If I could try to succinctly rephrase it in the context of your response here: "Patterns are statistically significant and improbable without some outside force. So if we recognize a pattern, it's LIKELY that there is some attractor or gravity creating this pattern. BUT, the fact that something is naturally occuring or biological in nature should add absolutely no credence or change the nature of how we normally pattern match." So I agree "male humans are on average more aggressive than females." That is great pattern that needs to be in any social model. But if then someone says "this is because all mammals work that way", that second statement is usually unverifiable and should add no credence to the pattern or model. Same with "assertive men are often leaders". This is a great pattern that should be considered. To then say "this is because pack animals designate an alpha male" is not verifiable and should not add weight to the claim. Now to me it feels like the biology bit is added to a LOT of arguments. This could be because any good logical thinker want to also pin down causation and upstream effects. I am concerned"it's biological/natural" is tacked onto arguments in an attempt to artificially strengthen arguments, and this post was my attempt at a response to that.
1AaronF
Well-said. To add to comment:  OP: "Some examples to illustrate the absurdity of this logic: Mammals live outdoors; therefore, homelessness is good." A positive statement would be that, it may be an evolutionary 'good' even if distasteful. An example might be that homelessness people may have more partners than a high IQ autist that has a mansion. Or we can say, all else being equal, it is healthier for humans to be outside more, much more than in the modern world. Designed housing and modern urban systems need to take this into account. OP: "Animals are illiterate; therefore, illiteracy is good." It may be distasteful, but many studies show that years of education means lower fertility; and that sexual selection TODAY actually does select for genes that are less-intelligent* (ADHD or even bad habits like alcohol and smoking.) Nature works in mysterious ways. A good way is that we need to think quite hard about underlying behaviors.  Why are literacy rates so low? And persistently so?  *See: Life without sex: Large-scale study links sexlessness to physical, cognitive, and personality traits, socioecological factors, and DNA: https://www.medrxiv.org/content/10.1101/2024.07.24.24310943v1.full
Viliam20

The people outside the town who buy houses here either expect to rent them expensively, or to use them as an investment because they expect the costs of housing to grow. (Or a combination of both.)

Refusing to build more houses means doing exactly the thing they want -- it keeps the rents high, and it keeps the costs growing.

If you have 500 000 people in the town, and 100 000 houses are owned by people outside the town, you should build more houses until there are 600 000 of them (i.e. not only 500 000). Then the people outside the town will find it difficult to rent their houses expensively, and may start worrying that the costs will not grow sufficiently to justify their investments.

2Ape in the coat
Thankfully rising land prices due to agglomeration effect is not a thing and the number of people in town is constant... Don't get me wrong, building more housing is good, actually. But it's going to be only marginal improvement, without addressing the systemic issues with land capturing a huge share of economic gains, renting economy and real-estate speculators. These issues are not solvable without a substantial Land Value Tax.
Viliam20

My problem with democracy is that most people are stupid. More precisely, they are relatively good at saying whether they are happy or unhappy, starving or fed, etc. They can give a relatively reliable "thumb up" or "thumb down" feedback to the government. But if you ask them about the specific steps that should be taken to make them better fed, etc., many of the popular suggestions will be quite insane.

For example, people can have a legitimate complaint about healthcare being inaccessible for them, and yet the suggestion many would propose will be somethi... (read more)

3Davey Morse
Yes. This brings to mind a general piece of wisdom for startups collecting product feedback: that feedback expressing painpoints/emotion is valuable, whereas feedback expressing implementation/solutions is not. The ideal direct-democratic system, I think, would do this: dividing comments like "My cost of living is too high" (valuable) from "Taxes need to go down because my cost of living is too high" (possibly, but an incomplete extrapolation).  This parsing seems possible in principle. I could imagine a system where feedback per person is capped, which would incentivize people to express the core of their issues rather than extraneous solution details (unless they happen to be solution-level experts).
Viliam20

Ah, then I believe the answer is "no".

On the time scale of current human lifespan, I guess I could point out that some old people are unkind, or that some criminals keep re-offending a lot, so it doesn't seem like time automatically translates to more kindness.

But an obvious objection is "well, maybe they need 200 years of time, or 1000", and I can't provide empirical evidence against that. So I am not sure how to settle this question.

On average, people get less criminal as they get older, so that would point towards human kindness increasing in time. On t... (read more)

2Mateusz Bagiński
When I read Tsvi's OP, I was imagining something like a (trans-/post- but not too post-)human civilization where everybody by default has an unbounded lifespan and healthspan, possibly somewhat boosted intelligence and need for cognition / open intellectual curiosity. (In which case, "people tend to X as they get older", where X is something mostly due to things related to default human aging, doesn't apply.) Now start it as a modern-ish democracy or a cluster of (mostly) democracies, run for 1e4 to 1e6 years, and see what happens.
5TsviBT
Ok, now we have a reasonable question. I don't know, but I provided two argument-sketches that I think are of a potentially relevant type. At an abstract level, the answer would be "mathematico-conceptual reasoning", just like in all previous instances where there's a thing that has never happened before, and yet we reason somewhat successfully about it--of which there are plenty examples, if you think about it for a minute.
Viliam20

Well, if we assume that humans are fundamentally good / inevitably converging to kindness if given enough time... then, yeah, giving someone God-emperor powers is probably going to be good in long term. (If they don't accidentally make an irreparable mistake.)

I just strongly disagree with this assumption.

3TsviBT
It's not an assumption, it's the question I'm asking and discussing.
Viliam812

Sounds to me like wishful thinking. You basically assume that in 1 000 000 years people will get bored of doing the wrong thing, and start doing the right thing. My perspective is that "good" is a narrow target in the possibility space, and if someone already keeps missing it now, if we expand their possibility space by making them a God-emperor, the chance of converging to that narrow target only decreases.

Basically, for your model to work, kindness would need to be the only attractor in the space of human (actually, post-human) psychology.

A simple exampl... (read more)

3TsviBT
No, I ask the question, and then I present a couple hypothesis-pieces. (Your stance here seems fairly though not terribly anti-thought AFAICT, so FYI I may stop engaging without further warning.) I'm seriously questioning whether it's a narrow target for humans. Curious to hear other attractors, but your proposals aren't really attractors. See my response here: https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=jfAoxAaFxWoDy3yso Ah I see you saw Ruthenis's comment and edited your comment to say so, so I edited my response to your comment to say that I saw that you saw.
Viliam20

Yes, if you have 500 000 people in town, you need to produce food for 500 000 people all the time. While if you have 500 000 people in town, you only need to build houses for 500 000 once.

But the logic of "there is a shortage of X, therefore the proper solution is to ban the production of X and hope that the problem will magically go away" is insane either way.

2Ape in the coat
Unless, of course, some people who already have a house or people who do not even live in your town can buy houses in your town in order to use them as investment and/or to rent them to people living in your town. Thankfully, this is a completely ridiculous counterfactual and noone ever does that... For some things this logic is valid. See induced demand and attempts to solve traffic jams by building even larger roads. Now housing isn't exactly like that, but neither its like food. This kind of appeals to simplified heuristics and resulting referent class tennis is non-productive.
Viliam20

"I'm tired of this simplified idea that building more buildings here will solve the affordable housing problem"

I am tired of this simplified idea that giving people more food will solve the hunger problem. (Insert sophisticated arguments about how supermarkets throw away lots of perfectly okay food, etc.) Therefore, making more food should remain banned.

2Jiro
Food gets used up quickly, but it takes a long while to use up housing, so banning new housing really isn't comparable to banning making food.
Viliam50

Are people fundamentally good?

Maybe some people are, and some people are not?

Are they practically good?

Not sure if we are talking about the same thing, but I think that there are many people who just "play it safe", and in a civilized society that generally means following the rules and avoiding unnecessary conflicts. The same people can behave differently if you give them power (even on a small scale, e.g. when they have children).

But I think there are also people who try to do good even when the incentives point the other way round. And also people who c... (read more)

2TsviBT
Right. Presumably, maybe. But I am interested in considering quite extreme versions of the claim. Maybe there's only 10,000 people who would, as emperor, make a world that is, after 1,000,000 years, net negative according to us. Maybe there's literally 0? I'm not even sure that there aren't literally 0, though quite plausibly someone else could know this confidently. (For example, someone could hypothetically have solid information suggesting that someone could remain truly delusionally and disorganizedly psychotic and violent to such an extent that they never get bored and never grow, while still being functional enough to give directions to an AI that specify world domination for 1,000,000 years.)
Viliam102

I'd like to know how you handle arguing for something like "usability" in the face of a morally urgent argument like "don't be eurocentric."

I would probably start with rejecting the premise that I have to listen to other people's arguments.

(This makes even more sense when we know that the people who loudly express their opinions are often just a tiny minority of users. However, it is perfectly possible to ignore the majority, too.)

I think this is a mistake that many intelligent people make, to believe that you need to win verbal fights. Perhaps identifying... (read more)

Viliam50

Similar here. I wouldn't want to constrain my 100 years older self too much, but that doesn't mean that I identify with something very vague like "existence itself". There is a difference between "I am not sure about the details" and "anything goes".

Just like my current self is not the same as my 20 years old self, but that doesn't mean that you could choose any 50 years old guy and say that all of them have the same right to call themselves a future version of my 20 years old self. I extrapolate the same to the future: there are some hypothetical 1000 yea... (read more)

Viliam60

Notice how Elon Musk has positioned himself so that he has a chance to win both if the governments will control the AIs and if they won't.

2Seth Herd
Interesting. I hadn't thought about Musk's influence and how he is certainly AGI-pilled.
Viliam40

I like the concept of Community Notes on Xitter, as a pushback against spreading misinformation. But now it seems that Musk will "fix" it, because his own tweets get often contradicted. Are there similar features on other social networks? How do they compare?

3Sam G
It was announced just yesterday that meta is planning to incorporate community notes as well using open source X algorithms. Ostensibly it could become the state-of-the-art bandaid solution. In my opinion, community notes systems are a good step, but they don't give much space for real deliberation around news. 
Viliam198

And this part is what Robin Hanson predicted about a decade ago. If I remember it correctly, he wrote that AI Safety was a low-status thing, therefore everyone associated with it was low-status. And if AI Safety ever becomes a high-status thing, then the people in the field will not want to be associated with their low-status predecessors. So instead of referencing them, an alternative history will be established, where someone high-status will be credited for creating the field from scratch (maybe using some inspiration from high-status people in adjacent fields).

8David Duvenaud
As someone who writes these kinds of papers, I try to make an effort to cite the original inspirations when possible.  And although I agree with Robin's theory broadly, there are also some mechanical reasons why Yudkowsky in particular is hard to cite. The most valuable things about the academic paper style as a reader are: 1) Having a clear, short summary (the abstract) 2) Stating the claimed contributions explicitly 3) Using standard jargon, or if not, noting so explicitly 4) A related work section that contrasts one's own position against others' 5) Being explicit about what evidence you're marshalling and where it comes from. 6) Listing main claims explicitly. 7) The best papers include a "limitations" or "why I might be wrong" section. Yudkowsky mostly doesn't do these things.  That doesn't mean he doesn't deserve credit for making a clear and accessible case for many foundational aspects of AI safety.  It's just that in any particular context, it's hard to say what, exactly, his claims or contributions were. In this setting, maybe the most appropriate citation would be something like "as illustrated in many thought experiments by yudkowsky [cite particular sections of the sequences and hpmor], it's dangerous to rely on any protocol for detecting scheming by agents more intelligent than oneself".  But that's a pretty broad claim.  Maybe I'm being unfair - but it's not clear to me what exactly yudkowsky's work says about the workability of these schemes other than "there be dragons here".
Viliam184

I believe it is a clear demonstration that misalignment likely does not stem from the model being “evil.” It simply found a better way to achieve its goal using unintended means.

It is fascinating to see that the official science has finally discovered what Yudkowsky wrote about a decade ago. Better late than never, I guess.

mishka100

They should actually reference Yudkowsky.

I don't see them referencing Yudkowsky, even though their paper https://cdn.openai.com/pdf/34f2ada6-870f-4c26-9790-fd8def56387f/CoT_Monitoring.pdf lists over 70 references, but I don't see them mentioning Yudkowsky (someone should tell Schmidhuber ;-)).

This branch of the official science is younger than 10 years (and started as a fairly non-orthodox one, it's only recently that this has started to feel like the official one; certainly no earlier than formation of Anthropic, and probably quite a bit later than that).

Viliam20

As I see it, there are two possible explanations for a fine-tuned universe:

  • an insane amount of luck (or something other than luck, such as a supernatural force)
  • multiverse

The argument for multiverse is precisely that it does not require the insane amount of luck.

Your argument is that with the insane amount of luck, the multiverse is not necessary.

That is correct, but then we are back to the original question: why did we get so extremely lucky?

1Fraser
If the only observable outcome is the one in which we got that extremely lucky, then it doesn't need explaining. You only can observe outcomes compatible with the fact of you making an observation.
Viliam5-1

I agree with the spirit of your suggestion -- often "it is known" that something couldn't possibly work, based only on armchair reasoning, or one half-assed attempt made a few decades ago, in a different country, with N=20.

That said, literally cash for babies feels somewhat dysgenic (though, maybe if we actually did the experiment, the results might surprise us). It seems like it would appeal most to people with short-term thinking, the most poor people (which is probably correlated with various dysfunctions), and psychopaths who only want cash and don't c... (read more)

2Yair Halberstadt
I agree there's better ways to do this, but: a) the point is even the brute force stupid ways are doable and would likely work. Obviously try to the cleverer ways first. b) the drop in fertility rate is so bad and so destructive that if we can't get this done the good way, even the dysgenic way is very much worth it.
Viliam20

Okay, that pretty much ruins the idea.

Makes me think, what about humans who would do the same thing? But probably the difference is that humans can build their credibility over time, and if someone new posted an unlikely comment, they would be called out on that.

1xpym
Presumably, this happens: https://slatestarcodex.com/2016/12/12/might-people-on-the-internet-sometimes-lie/ I do often notice how the top upvoted reddit comment in big subs is confidently wrong, with a more correct/nuanced take sitting much lower.
3Canaletto
It's really hard for humans to match the style / presentation / language without putting a lot of work into understanding the target of the comment. LLMs are inherently worse (right now) at doing the understanding, coming up with things worth saying, being calibrated about being critical AND they are a lot better at just imitating the style.  This just invalidates some side signals humans habitually use on one another.
Viliam60

build a social circle which can maintain its own attention, as a group, without just reflecting the memetic currents of the world around it.

Note that it is not necessary for the social circle to share your beliefs, only to have a social norm that people express interest in each other's work. Could be something like: once or twice in a week the people will come to a room and everyone will give a presentation about what they have achieved recently, and maybe the other people will provide some feedback (not in the form of "why don't you do Y instead", but with the assumption that X is a thing worth doing).

Viliam53

I think you don't need a lot of agency to write comments on Less Wrong. I imagine an algorithm like this:

  • After an article is written, wait a random time interval between 15 minutes and 24 hours.
  • Read the article and the existing comments.
  • Prepare a comment that you believe would get most karma (you can learn from existing LW comments and their karma).
  • Think again how much karma would your prepared comment probably get. In your estimation, include the fact that LW readers do not like comments that seem like written by an LLM.
  • If you conclude that the expected k
... (read more)
4Canaletto
This should be probably only attempted with clear and huge warning that it's a LLM authored comment. Because LLMs are good at matching style without matching the content, it could go with exploiting heuristics of the users calibrated only for human level of honesty / reliability / non-bulshitting.  Also check this comment about how conditioning on the karma score can give you hallucinated strong evidence: https://www.lesswrong.com/posts/PQaZiATafCh7n5Luf/gwern-s-shortform?commentId=smBq9zcrWaAavL9G7 
2Nathan Helm-Burger
This seems quite technologically feasible now, and I expect the outcome would mostly depend on the quality and care that went into the specific implementation. I am even more confident that if the detail of 'the comments of the bot get further tuning via feedback, so that initial flaws get corrected', then the bot would quickly (after a few hundred such feedbacks) get 'good enough' to pass most people's bars for inclusion.
Viliam*-21

Q: How many Zizians does it take to change a light bulb?

A: At least two (one left hemisphere, and one right hemisphere), and the light bulb will end up broken... but that's okay, because for complicated game-theoretic reasons there is now more light in a parallel universe, and the electron-eating light bulb deserved it anyway.

Viliam20

The best way to be seen as trustworthy is to be trustworthy.

Depends on the environment. Among relatively smart people who know each other, trust their previous experience, and communicate their previous experience with each other -- yes. But this strategy breaks down if you keep meeting strangers, or if people around you believe the rumors (so it is easy to character-assassinate a honest person).

Viliam20

Sometimes being known as smart is already a disadvantage, because some people assume (probably correctly) that it would be easier for a smarter person to deceive them.

I wonder how many smart people are out there who have concluded that a good strategy is to hide their intelligence, and instead pretend to be merely good at some specific X (needed for their job). I suspect that many of them actually believe that (it is easier to consistently say something if you genuinely believe that), and that women are over-represented in this group.

Viliam30

I agree. One problem of grade inflation is that we lose the ability to measure excellence.

If you have a scale e.g. from 1 to 5, where the average grade is 3, if you somehow hire a magical teacher, you could see an improvement of the mean student, let's say from 3 to 2. Then you might conclude that the teacher does something right, and maybe try to replicate that.

But if instead the average grade is 1, hiring the same magical teacher would... leave the average grade at 1. Are all students "okay" in the subject? That's 1. Are they excellent? Also 1. Do half o... (read more)

1James Camacho
Grade inflation originally began in the United States due to the Vietnam War draft. University students where exempt from the draft as long as they maintained high enough grades, so students became less willing to stretch their abilities and professors less willing to accurately report their abilities. The issue is that grades are trying to serve three separate purposes: 1. Regular feedback to students on how well they understand the material. 2. Personal recommendations from teachers to prospective employers/universities. 3. Global comparisons between students. The administration mostly believe grades serve the third purpose, so they advocate for fudging the numbers. "Last year, our new policies implemented at Goodhart School of Excellence improved the GPA by 0.5 points! Look at how successful our students are compared to others." Teachers, on the other hand, usually want grades to serve the first two purposes. If we want to prevent Goodharting, we can either give teachers back their power, or use other comparison systems. This is already kind-of a thing. Top universities no longer use GPA as a metric, except as a demerit for imperfect grades, relying more on standardized test scores. There was a brief period where they tried going test-optional, but MIT quickly reversed that trend. I don't think a standardized exam is a perfect solution—how do you compare project- or lab-based classes, like computer science and chemistry? I think in these scenarios we could have students submit their work to third parties, much like the capstone project in AP Seminar & Research. If we can get administrators to use a better (unfudgible) comparator, I'm not actually terribly worried whether teachers use grades to give regular feedback or recommend their students. It's just important to make sure the comparator is hard enough to actually see a spread, even at the very top. The number of "perfect" ACT scores has increased by 25x in the past 25 years, and I understand why fro
Load More