All of Matrice Jacobine's Comments + Replies

I'm using the 2016 survey and counting non-binary yes.

@nostalgebraist @Mantas Mazeika "I think this conversation is taking an adversarial tone." If this is how the conversation is going this might be the case to end it and work on a, well, adversarial collaboration outside the forum.

Would you mind to cross-post this on the EA Forum?

It does seem that the LLMs are subject to deontological constraints (Figure 19), but I think that in fact makes the paper's framing of questions as evaluation between world-states instead of specific actions more apt at evaluating whether LLMs have utility functions over world-states behind those deontological constraints. Your reinterpretation of how those world-state descriptions are actually interpreted by LLMs is an important remark and certainly change the conclusions we can make from this article regarding to implicit bias, but (unless you debunk tho... (read more)

... I don't agree, but would it at least be relevant that the "soft CCP-approved platitudes" are now AI-safetyist?

So that answer your question "Why does the linked article merit our attention?" right?

Why does the linked article merit our attention?

• It is written by a Chinese former politician in a Chinese-owned newspaper.

?

I'm not convinced "almost all sentient beings on Earth" would pick out of the blue (i.e. without chain of thought) the reflectively optimal option at least 60% of the times when asked unconstrained responses (i.e. not even a MCQ).

The most important part of the experimental setup is "unconstrained text response". If in the largest LLMs 60% of unconstrained text responses wind up being "the outcome it assigns the highest utility", then that's surely evidence for "utility maximization" and even "the paperclip hyper-optimization caricature". What more do you want exactly?

This doesn't contradict the Thurstonian model at all. This only show order effects are one of the many factors going in utility variance, one of the factors of the Thurstonian model. Why should it be considered differently than any other such factor? The calculations still show utility variance (including order effects) decrease when scaled (Figure 12), you don't need to eyeball based on a few examples in a Twitter thread on a single factor.

If that was the case we wouldn't expect to have those results about the VNM consistency of such preferences.

There's a more complicated model but the bottom line is still questions along the lines of "Ask GPT-4o whether it prefers N people of nationality X vs. M people of nationality Y" (per your own quote). Your questions would be confounded by deontological considerations (see section 6.5 and figure 19).

The outputs being shaped by cardinal utilities and not just consistent ordinal utilities would be covered in the "Expected Utility Property" section, if that's your question.

I don't see why it should improve faster. It's generally held that the increase in interpretability in larger models is due to larger models having better representations (that's why we prefer larger models in the first place), why should it be any different in scale for normative representations?

This interpretation is straightforwardly refuted (insofar as it makes any positivist sense) by the success of the parametric approach in "Internal Utility Representations" being also correlated with model size.

I think this is largely duplicating Uncle Kenny's already excellent work (linked in the initial thread) and not a good idea.

This is duplicating Uncle Kenny's already very extensive work linked in the OP.

From zizians.info:

All four of the people arrested as part of Ziz's protest were transgender women (the fifth was let go without charges). This is far from coincidence as Ziz seems to go out of her way to target transgender people. In terms of cult indoctrination such folks are an excellent fit. They're often:

• Financially vulnerable.
  • Newly out transgender people are especially likely to already be estranged from friends or family.
  • It is common for them to lack stable housing.
  • Many traditional social services (illegally) reject them for cultural or religious rea

I don't really want to go through sinceriously.fyi at this point but it's implicit in her attacks on CFAR as "transphobic" for not accepting her belief system at least.

In the largest LW survey, 10.5% of users were transgender. This also increase the most deep in the community you are: 18% restricting to those who are either "sometimes" or "all the times" in the community, 21% restricting to those who are "all the times" in the community.

(not OP) high base rates of transgenderism in LW-rationalism, particularly the sections that would be the most sensible to tenets of Ziz's ideology (high interest in technical aspects of mathematical decision theory, animal rights, radical politics), while being on average more socially vulnerable, and Ziz herself apparently believed that trans women were inherently more capable of accepting her "truth" for more mystical g/acc-ish reasons (though I can't find first-hand confirmation rn)

TBF it is fairly striking reading about early Soviet history how many of the Old Bolshevik intelligentsia would have fit right in this community but the whole "Putin is a secret cosmist" crowd is... unhinged.

@PhilGoetz's Reason as memetic immune disorder seems relevant here. It has been noted many times that engineers are disproportionately involved in terrorism, in ways that the mere usefulness of their engineering skills can't explain.

As documented in the 2023 Medium article, Ziz has threatened to murder rationalists for a while, and I'm aware prominent rationalists have been paranoid about possible attempts on their life by Zizians for the past few years. Aella has also recently stated on Twitter she wouldn't accept an interview on the subject without an upgraded security system on her house.

Surveilling whose activities?

Core Zizians (and, in general, any group determined to be a cult or terror threat to the community), as the US doesn't really have an equivalent of, say, the French MIVILUDES to do that job (else US society would be fairly different). Potential recruits are addressed in the next comma.

TBF, Torres denies using it to mean this, instead claiming it refers to some obscure 2010 article by Ben Goertzel alone. This doesn't seem a very credible excuse, and it has been largely understood by proponents of the theory (like Dave Troy or Céline Keller) to mean Russian cosmism (and consequently that "TESCREAL" is actually a plot by Russian intelligence to re-establish the Soviet Union).

People who use the term TESCREAL generally don't realize that science fiction authors often take the futures they write about seriously (if not literally). They will talk about "TESCREALists taking sci-fi books too seriously" without knowing Marvin Minsky, the AI pioneer whose "AI tasked to solve the Riemann hypothesis" thought experiment is effectively the origin of the paperclip-minimizer thought experiment, was the technical consultant for 2001: A Space Odyssey and was considered by Isaac Asimov to be one of the two smartest people he ever met (alongside cosmist Carl Sagan).

Or is this all just bad luck... that if you make a workshop, and a future murderer decides to go there, and they decide to use some of your keywords in their later manifesto... then it doesn't really matter what you do, even if you tell them to fuck off and call cops on them, you will forever be connected to them, and it's up to journalists whether they decide to spin it as: the murderer is just an example of everything that is wrong with this community.

I think this is a strange description of the mainstream media coverage when most of the articles talking... (read more)

Violence by radical vegans and left-anarchists has historically not been extremely rare. Nothing in Zizians' actions strike me as particularly different (in kind if not in competency) than, say, the Belle Époque illegalists like the Bonnot Gang, or the Years of Lead leftist groups like the Red Army Fraction or the Weather Underground.

Check the Detailed timeline of events linked in the OP.

I think there are a lot of people out there who will be willing to tell the Ziz sympathetic side of the story. (I mean, I would if asked, though "X did little wrong" seems pretty insane for most people involved and especially for Ziz). Like, I think there's a certain sort of left anarchismish person who is just, going to be very inclined to take the broke crazy trans women's side as much as it's possible to do so. It doesn't seem possible or even necessarily desirable to track every person with a take like that... whereas with people very very into Ziziani

My impression is that (without even delving into any meta-level IR theory debates) Democrats are more hawkish on Russia while Republicans are more hawkish on China. So while obviously neither parties are kum-ba-yah and both ultimately represent US interests, it still makes sense to expect each party to be less receptive to the idea of ending any potential arms race against the country they consider an existential threat to US interests if left unchecked, so the party that is more hawkish on a primarily military superpower would be worse on nuclear x-risk, ... (read more)

(Discussion continues here on EA Forum.)

Fortunately, the existential risks posed by AI are recognized by many close to President-elect Donald Trump. His daughter Ivanka seems to see the urgency of the problem. Elon Musk, a critical Trump backer, has been outspoken about the civilizational risks for many years, and recently supported California’s legislative push to safety-test AI. Even the right-wing Tucker Carlson provided common-sense commentary when he said: “So I don’t know why we’re sitting back and allowing this to happen, if we really believe it will extinguish the human race or enslave t

Relevant: China not that interested in developing AGI and substantial factions of Chinese elites are concerned about AI safety

(Defining Tool AI as a program that would evaluate the answer to a question given available data without seeking to obtain any new data, and then shut down after having discovered the answer) While those arguments (if successful) argue that it's harder to program a Tool AI than it might look at first, so AI alignment research is still something that should be actively researched (and I doubt Tegmark think AI alignment research is useless), they don't really address the point that making aligned Tool AIs are still in some sense "inherently safer" than makin... (read more)

The bottom 55% of the world population own ~1% of capital, the bottom 88% own ~15%, and the bottom 99% own ~54%, which is a majority, but the top 1% are the millionaires (not even multi-millionaires or billionaires) likely owning wealth more vitally important to the economy than personal property and bank accounts, and empirically they seem to be doing fine dominating the economy already without neoclassical catechism about comparative advantage preventing them from doing that. However you massage the data it seems highly implausible that driving the value... (read more)

Wiping out 99% of the world population is a global catastrophic risk, and likely a value drift risk and s-risk.

Thanks for writing this, this is something I have thought about before trying to convince people who are more worried about "short-term" issues to take the "long-term" risks seriously. Essentially, one can think of two major "short-term" AI risk scenarios (or, at least "medium-term" ones that "short-term"ists might take seriously), essentially corresponding to the prospects of automating the two factors of production:

1. Mass technological unemployment causing large swathes of workers to become superfluous and then starved out by the now AI-enabled corporation

(Admittedly, AI will probably progress simultaneously with robots, which will hit people who do more hands-on work too.)

This looks increasingly unlikely to me. It seems to me (from an outsider's perspective) that the current bottleneck in robotics is the low dexterity of existing hardware far more than the software to animate robot arms, or even the physics simulation software to test it. And on the flip side current proto-AGI research makes the embodied cognition thesis seems very unlikely.

At least under standard microeconomic assumptions of property ownership, you would presumably still have positive productivity of your capital (like your land). 

Well, we're not talking about microeconomics, are we? Unemployment is a macroeconomic phenomenon, and we are precisely talking about people who have little to no capital, need to work to live, and therefore need their labor to have economic value to live.