Systemic opacity, state-driven censorship, and state control of the media means AGI development under direct or indirect CCP control would probably be less transparent than in the US, and the world may be less likely to learn about warning shots, wrongheaded decisions, reckless behaviour, etc.

That's screened off by actual evidence, which is, top labs don't publish much no matter where they are, so I'd only agree with "equally opaque".

Aidan McLaughlin (OpenAI): ignore literally all the benchmarks the biggest o3 feature is tool use. Ofc it’s smart, but it’s also just way more useful. >deep research quality in 30 seconds >debugs by googling docs and checking stackoverflow >writes whole python scripts in its CoT for fermi estimates McKay Wrigley: 11/10

Newline formatting is off (and also for many previous posts).

I'm not sure if this will be of any use, since your social skills will surely be warped when you expect iterating on them (in a manner like radical transparency reduces awareness of feelings ).

Most cases, you could build a mini-quadcopter, "teach" it some tricks and try showcasing it, having video as a side effect!

Browsing through recent rejects, I found an interesting comment that suggests an automatic system "build a prior on whether something is true by observing social interactions around it vs competing ideas".

@timur-sadekov While it will fail most certainly, I do remember that the ultimate arbiter is experiment, so I shall reserve the judgement. Instead, I'm calling for you to test the idea over prediction markets (Manifold, for instance) and publish the results.

The AI industry is different—more like biology labs testing new pathogens: The biolab must carefully monitor and control conditions like air pressure, or the pathogen might leak into the external world.

This looks like a cached analogy, because given previous paragraph this would fit more: "the AI industry is different, more like testing planes; their components can be internally tested, but eventually the full plane must be assembled and fly, and then it can damage the surroundings".

Same goes for AI: If we don’t keep a watchful eye, the AI might be able to

I suggest additional explanation.

The bigger the audience is, the more people there are who won't know a specific idea/concept/word (xkcd's comic #1053 "Ten Thousand" captures this quite succinctly), so you'll simply have to shorten.

I took logarithm of sentence length and linearly fitted it against logarithm of world population (that shouldn't really be precise since authors presumably mostly cared about their society, but that would be more time-expensive to check).

Given the gravity of Sam Altman's position at the helm of the company leading the development of an artificial superintelligence which it does not yet know how to align -- to imbue with morality and ethics -- I feel Annie's claims warrant a far greater level of investigation than they've received thus far. 

Then there's a bit of shortage of something public... and ratsphere-adjacent... maybe prediction markets?

I can create (subsidize) a few, given resolution criteria.

Alphabet of LW Rationality, special excerpts

• Cooperation: you need to cooperate with yourself to do anything. Please do so iff you are building ASI;
• Defection: it ain't no defection if you're benefitting yourself and advancing your goals;
• Inquiry: a nice way of making others work for your amusement or knowledge expansion;
• Koan: if someone is not willing to do research for you, you can waste their time on pondering a compressed idea.

I claim that I exist, and that I am now going to type the next words of my response. Both of those certainly look true. As for whether these beliefs are provable, I do not particularly care; instead, I invoke the nameless:

Every step of your reasoning must cut through to the correct answer in the same movement. More than anything, you must think of carrying your map through to reflecting the territory.

My black-box functions yield a statement "I exist" as true or very probable, and they are also correct in that.

After all, If I exist, I do not want to deny my... (read more)

What's useful about them? If you are going to predict (the belief in) qualia, on the basis of usefulness , you need to state the usefulness.

There might be some usefulness!

The statement I'd consider is "I am now going to type the next characters of my comment". This belief turns out to be true by direct demonstration, it is not provable because I could as well leave the commenting to tomorrow and be thinking "I am now going to sleep", not particularly justifiable in advance, and it is useful for making specific plans that branch less on my own actions.

I object to the original post because of probabilistic beliefs, though.

I don’t think there is a universal set of emojis that would work on every human, but I totally think that there is a set of such emojis (or something similar) that would work on any given human at any given time, at least a large percentage of the time, if you somehow were able to iterate enough times to figure out what it is.

Probably something with more informational content than emojis, like images (perhaps slightly animated). Trojan Sky, essentially.

The core idea behind confidentiality is to stop social pressure (whether on the children or the parents).

Parents have a strong right to not use genomic engineering technology, and if they do use it then they have a strong right to not alter any given trait.

How to prevent all the judgements better than to make information sharing completely voluntary?

Important, and would be nice even if passed as-is! Admittedly, there's some space to have even stronger ideas, like...

"Confidentiality of genomic interventions. Human has natural right for details of which aspects of their genome were custom-chosen, if any, to be kept confidential" (probably also prohibit parents/guardians from disclosing that, since knowledge cannot be sealed back into the box).

A nice scary story! How fortunate that it is fiction...

... or is it? If we get mind uploads, someone will certainly try to gradient-ascent various stimulus (due to simple hostility or Sixth Law of Human Stupidity), and I do believe the underlying fact that a carefully crafted image could hijack mental processes to some point.

It's 16:9 (modulo possible changes in the venue).

I have seen your banner and it is indeed one of the best choices out there! For announcing the event I preferred another one.

Hi! By any chance, do you have HPMOR banners (to display on screen, for instance)?

I get the impression that you're conflating two meanings of «personal» - «private» and «individual». The fact that I might feel uncomfortable discussing this in a public forum doesn’t mean it «only works for me» or that it «doesn’t work, but I’m shielded from testing my beliefs due to privacy». There are always anonymous surveys, for example. Perhaps you meant something else?

I meant to say that private values/things are unlikely to coincide between different people, though now I'm a bit less sure.

Moreover, even if I were to provide yet another table of my

Seeing you know the exact numbers, I wonder if you could connect with those other families? It's harder to get the best outcome if players do not cooperate and do not even know the wishes of others. Adding that this would be a valuable socializing opportunity would be somewhat hypocritical from me but it's still so.

Upvoted as a good re-explanation of CEV complexity in simpler terms! (I believe LW will benefit from recalling the long understood things so that it has a chance on predicting future in greater detail.)

In essence, you prove the claim "Coherent Extrapolated Volition would not literally include everything desirable happening effortlessly and everything undesirable going away". Would I be wrong to guess it argues against position in https://www.lesswrong.com/posts/AfAp8mEAbuavuHZMc/for-the-sake-of-pleasure-alone?

That said, current wishes of many people includ... (read more)

And introspectively, I don’t see any barriers to comparing love with orgasm, with good food, with religious ecstasy, all within the same metric, even though I can’t give you numbers for it.

Why not? It'd be interesting to hear valuations from your experience and experiments, if that wasn't very personal.

(On the other hand, if it IS too personal, then who would choose to write the metric down for an automatic system optimizing it by their whims?)

The follow-up post has a very relevant comment:

Can you just give every thief a body camera?

Well of course this is illegal under current US laws, however this would help against being unjustly accused as in your example of secondary crime. It would also be helpful against repeat offences for a whole range of other crimes.

I'd maintain that those problems already exist in 20M-people cities and will not necessarily become much worse. However, by increasing city population you bring in more people into the problems, which doesn't seem good.

Is there any engineering challenge such as water supply that prevents this from happening? Or is it just lack of any political elites with willingness + engg knowledge + governing sufficient funds?

That dichotomy is not exhaustive, and I believe going through with the proposal will necesarily make the city inhabitants worse off.

1. Humans' social machinery is not suited to live in such large cities, as of the current generations. Who to get acquainted with, in the first place? Isn't there lots of opportunity cost to any event?
2. Humans' biomachinery is not suited

[Un]surprisingly, there's already a Sequences article on this, namely Is That Your True Rejection?.

(I thought this comment would be more useful with call-for-action "so how should we rewrite that article and make it common knowledge for everyone who joined LW recently?" but was too lazy to write it.)

In general if you "defect" because you thought the other party would that is quite sketchy. But what if proof comes out they really were about to defect on you?

By the way, if we consider game theory and logic to be any relevant, then there's a corollary of Löb's Theorem: if you defect given proof that counterparty will defect, and another party will defect given proof that you will, then you both will, logically, defect against each other, with no choice in the matter. (And if you additionally declare that you cooperate given proof that partner will cooper... (read more)

Weak-upvoted because I believe this topic merits some discussion, but the discourse level should be higher since setting NSFW boundaries for user relates to many other topics:

1. Estimating social effect of imposing a certain boundary.
   Will stopping rough roleplaying scenarios lead to less people being psychopaths? That seems to be an empirical question, since intuitively effect might go either way - doing the same in real world instead OR internalizing rough and inconsiderate actions as not normal.
2. Simulated people's opinion on being placed in the user-requeste

That's all conditional on P = NP, isn't it? Also, which part do you consider weaker: digital signatures or hash functions?

Line breaks seem to be broken (unless it was your intention to list all the Offers-Mundane-Utility-s and so on in a single paragraph).

Acknowledge it is not visible anymore!

Hi! I believe this post, not one for the 2021 review, is meant to be pinned at the front page?

I'd like there to be a reaction of "Not Exhaustive", meant for a step where comment (or top-level post, for that matter) missed an important case - how a particular situation could play out, perhaps, or an essential system's component is not listed. An example use: on statement "to prevent any data leaks, one must protect how their systems transfer data and how they process it" with the missed component being protection of storage as well.

I recall wishing for it like three times since the New Year, with the current trigger being this comment:

Elon alr

Now I feel like rationality itself is an infohazard. I mean, rationality itself won't hurt you if you are sufficiently sane, but if you start talking about it, insufficiently sane people will listen, too. And that will have horrible consequences. (And when I try to find a way to navigate around this, such as talking openly only to certifiably sane people, that seems like the totally cultish thing to do.)

There is an alternative way, the other extreme: get more and more rationalists.
If the formed communities do not share the moral inclinations of LW communit... (read more)

Actually, AIs can use other kinds of land (to suggest from the top of the head, sky islands over oceans, or hot air balloons for a more compact option) to be run, which are not usable by humans. There have to be a whole lot of datacenters to make people short on land - unless there are new large factories built.

It seems that an alternative to AI unlearning is often overlooked: just remove dataset parts which contain sensitive (or, to that matter, false) information or move training on it towards beginning to aid with language syntax only. I don't think a bit of inference throughout the dataset is any more expensive than training on it.

In practice I do not think this matters, but it does indicate that we’re sleeping on the job – all the sources you need for this are public, why are we not including them.

I'd reserve judgement whether that matters, but I can attest a large part of content is indeed skipped... probably for those same reasons why market didn't react to DeepSeek in advance: people are just not good at knowing distant information which might still be helpful to them.

If you want to figure out how to achieve good results when making the AI handle various human conflicts, you can't really know how to adapt and improve it without actually involving it in those conflicts.

I disagree. There is such a lot of conflicts (some kinds make it into writing, some just happen) of different scales, both in history and now; I believe they span human conflict space almost fully. Just aggregating this information could lead to very good advice on handling everything, which AI could act upon if it so needed.

But if I know that there are external factors, I know the bullet will deviate for sure. I don't know where but I know it will.

You assume that blur kernel is non-monotonic, and this is our entire disagreement. I guess that different tasks have different noise structure (for instance, if somehow noise geometrically increased - $\pm 1,\pm 2,\dots ,\pm 2^i$ - we wouldn't ever return to an exact point we had left).

However, if noise is composed from many i.i.d. small parts, then it has normal distribution which is monotonic in the relevant sense.

Can you think of any good reason why I should think that?

Intuition. Imagine a picture with bright spot in the center, and blur it. The brightest point will still be in center (before rounding pixel values off to the nearest integer, that is; only then may a disk of exactly equiprobable points form).

My answer: because strictly monotonic^[1] probability distribution prior to accounting for external factors (either "there might be negligible aiming errors" or "the bullet will fly exactly where needed" are suitable) will remain strictly monotonic when blur... (read more)

Like why are time translations so much more important for our general work than space translations?

I'd imagine that happens because we are able to coordinate our work across time (essentially, execute some actions), while work coordination across space-separated instances is much harder (now, it is part of IT's domain under name of "scalability").

An interesting framing! I agree with it.

As another example: in principle, one could make a web server use an LLM connected to database to serve any requests, not coding anything. It would even work... till the point someone would convince the model to rewrite the database to their whims! (A second problem is that normal site should be focused on something, in line with famous "if you can explain anything, your knowledge is zero".)

That article is suspiciously scarce on what microcontrols units... well, glory to LLMs for decent macro management then! (Though I believe that capability is still easier to get without text neural networks.)

I don't think so as I had success explaining away the paradox with concept of "different levels of detail" - saying that free will is a very high-level concept and further observations reveal a lower-level view, calling upon analogy with algorithmic programming's segment tree.

(Segment tree is a data structure that replaces an array, allowing to modify its values and compute a given function over all array elements efficiently. It is based on tree of nodes, each of those representing a certain subarray; each position is therefore handled by several - specif... (read more)

Doesn't the "threat" to delete the model have to be DT-credible instead of "credible conditioned on being human-made", given that LW with all its discussion about threat resistance and ignoring is in training sets?

(If I remember correctly, a decision theory must ignore "you're threatened to not do X, and the other agent is claiming to respond in such a way that even they lose in expectation" and "another agent [self-]modifies/instantiates an agent making them prefer that you don't do X".)

The surreal version of the VNM representation theorem in "Surreal Decisions" (https://arxiv.org/abs/2111.00862) seems to still have a surreal version of the Archimedean axiom.

That's right! However it is not really a problem unless we can obtain surreal probabilities from the real world; and if all our priors and evidence are just real numbers, updates won't lead us into the surreal area. (And it seems non-real-valued probabilities don't help us in infinite domains, as I've written in https://www.lesswrong.com/posts/sZneDLRBaDndHJxa7/open-thread-fall-2024?c... (read more)

Yes, many people will have problems with the Archimedes' axiom because it implies that everything has a price (that any good option can be probability-diluted enough that a mediocre is chosen instead), and people don't take it kindly when you tell "you absolutely must have a trade-off between value A and value B"  - especially if they really don't have a trade-off, but also if they don't want to admit or consciously estimate it.

Thankfully, that VNM property is not that critical for rational decision-making because we can simply use surreal numbers ins... (read more)

But technology is not a good alternative to good decision making and informed values.

After thinking on this a bit, I've somewhat changed my mind.

(Epistemic status: filtered evidence.)

Technology and other progress has two general directions: a) more power for those who are able to wield it; b) increasing forgiveness, distance to failure. For some reason, I thought that b) was a given at least on average. However, now it came to mind that it's possible for someone to
1) get two dates to accidentally overlap (or before confirming with partners-to-be that poly ... (read more)

I object to the framing of society being all-wise, and instead believe that for most issues it's possible to get the benefits of both ways given some innovators on that issue. For example, visual communication was either face-to-face or heavily resource-bounded till the computer era - then there were problems of quality and price, but those have been almost fully solved in our days.
Consequently, I'd prefer "bunch of candy and no diabetes still" outcome, and there are some lines of research/ideas into how this can be done.
As for "nonmarital sex <...> ... (read more)

Nicely written!

A nitpick: I believe "Voluntary cooperation" shouldn't always be equal to "Pareto preferred". Consider an Ultimatum game, where two people have 10 goodness tokens to split; the first person suggests a split (just once), then the second may accept or reject (when rejecting, all tokens are discarded). 9+1 is Pareto superior to 0+0 but one shouldn't [100%] accept 9+1 lest that becomes anything they are ever suggested. Summarizable with "Don't do unto yourself what you wouldn't want to be done unto you", or something like that.