The whole AI safety concern implies that the intersection of "thinkers" and "disasters" is non-null, yes?

I had a white-blue upbringing (military family) and a blue-green career (see below); my hobbies are black-green-white (compost makes my garden grow to feed our community); my vices are green-red; and my politics are five-color (at least to me).

Almost all of my professional career has been in sysadmin and SRE roles: which is tech (blue) but cares about keeping things reliable and sustainable (green) rather than pursuing novelty (red). Within tech's blue, it seems to me that developer roles run blue-red (build the exciting new feature!); management roles run blue-white (what orderly social rules will enable intelligence?); and venture capital runs blue-black (how do we get cleverness to make money for us?); while SRE and similar roles run blue-green.

My garden runs on blood, bones, rot, and worm poop: green-black Golgari territory, digesting the unwanted to feed the wanted. I stick my hands in the dirt to feel the mycorrhizae. But the point of the garden is to share food with others (green-white) because it makes me feel good (black-red). I'm actually kinda terrible at applying blue to my hobbies, and should really collect some soil samples for lab testing one of these days.

Here are some propositions I think I believe about consciousness:

1. Consciousness in humans is an evolved feature; that is, it supports survival and reproduction; at some point in our evolutionary history, animals with more of it out-competed animals with less.
2. Some conscious entities sometimes talk truthfully about their consciousness. It is often possible for humans to report true facts about their own objects of consciousness (e.g. self-awareness, qualia, emotions, thoughts, wants, etc.; "OC" for short).
3. Consciousness is causally upstream of humans emitting truthful sentences about OC. (When I truthfully report on my OC, there is nothing especially Gettier going on.)
4. If a zombie could exist, and were to emit sentences that purport to be "about" its OC, those sentences would all be false; in the same sense that the sentences "I am able to play grandmaster-level chess", "I find tarantulas erotically appealing", "I intend to bike naked across the Bay Bridge today", or "I see an ultraviolet-colored flower" would be false if I were to say them.
5. The ability to notice and monitor one's own OC is practically useful for humans. It is a prerequisite for certain kinds of planning our future actions that we do.
6. The ability to truthfully talk about one's OC is practically useful for humans. It is a prerequisite for certain kinds of cooperation with one another that we do. (For instance, we can make honest promises about what we intend to do; we can truthfully report if something scares us or pleases us; etc.)
7. Proposition #6 is true even when it is possible to undetectably lie about one's OC. (Promises are still useful even though some people do sometimes make promises with deceptive intent.)
8. If zombies could exist, they couldn't honestly promise one another anything, because they can't make true statements about their intentions: intentions are OC, and all statements a zombie makes about OC are false.
9. Consciousness in humans has the curiously strong character that it does because it is particularly useful for us to be able to cooperate with other humans by communicating about our OC; due to the sorts of complex behavior that groups of humans can exhibit when we work together.
10. Consciousness is not a requirement for generating human-like language (including sentences that purport to be about consciousness); just as it is not a requirement for playing grandmaster-level chess or discovering new mathematical proofs.
11. Consciousness in humans is suspended during deep sleep, general anesthesia, and other episodes of un-consciousness.
12. Consciousness is also interrupted by visual saccades, attentional shifts, and other sub-conscious processes that affect OC. (People can learn to notice many of these, but we don't do so automatically; mindfulness meditation is a learnable skill, not a default behavior.)
13. Consciousness nonetheless typically presents the impression of a continuous self. (Most humans do not go around all day in a state of ego-death or PNSE; such states are unusual and remarkable.)
14. The environment in which a human conscious mind develops is a human body; this affects the kinds of OC we can have. (For instance: We have visual qualia of redness and not of ultravioletness because our eyes don't register ultraviolet. There is nothing that it's like to see ultraviolet with human eyes. We have emotions for fight or flight, and for cuddle and care, but not for turn into a swarm of spiders — because our bodies can't do that!)
15. A design reason that consciousness (falsely) presents itself as a continuous mental self, is that there really is a continuous body that supports it. The conscious mind lacks continuity, but must generate actions as if it has continuity, because the body that it's piloting does.

If the text says that it is not holy, then who are we to disagree?

I think I agree: perceptions are fallible representations of reality, but infallible representations of themselves. If I think I see a cat, I may be wrong about reality (it's actually a raccoon) but I'm not wrong about having had the perception of a cat.

A standardized test measures some combination of ① the intended subject-matter skills and knowledge, and ② skills and knowledge that are specific to manipulating the test, also known as "test-taking skills". We know this because we can teach test-taking skills; which is to say, we can improve a student's performance on a standardized subject test by teaching them something that is not that subject.

A perfect standardized test would measure only ① with no influence from ②. If the subject is biology, then a perfect test would be one where the only way to get a better score would be to actually get better at biology, not at test-taking.

Some elements of moral thought seem to reflect an underlying reality akin to mathematical truth. "Fairness" for instance naturally relates to equal divisions, reciprocity, and symmetries akin to Rawls' veil of ignorance. Eliezer discusses this here: if Y thinks "fair" is splitting the pie evenly and Z thinks "fair" is "Z gets the whole pie", Y is just right and Z is just wrong.

Even though human moral judgment is evolved, what it is evolved to do includes mapping out symmetries like "it's wrong for A to murder B, or for B to murder A" → "it's wrong for anyone to murder anyone" → "murder is generically wrong".

Perceptions are infallible

Can you expand on this? Optical and auditory illusions exist, which seem to me to be repeatably demonstrable fallible perceptions: people reliably say that line A looks longer than line B in the Müller-Lyer illusion (the one with the arrowheads), even after measuring.

I'll just also say that commitment to a belief being the whole point is a very Abrahamic view and less common in other religions.

It seems to me that Anglo-American atheists often have a Protestant (or even specifically Lutheran) ontology of religion; they implicitly expect that "religion" must mean something creedal, evangelical, often sola scriptura, and various other things that aren't even universal among denominations of Christianity.

There are ways to address the problem of cheating in school without nuking the datacenters.

For instance: If a student gets an LLM to write their homework essay or term paper, that's not really very different from having another student or an essay-writing service write it — and those are problems that schools and colleges faced before LLMs came along. In these cases, the student will not be able to discuss their work very effectively in class. So, structure the class as a seminar or workshop, in which students are expected to discuss their work. In math classes, have students discuss proofs and constructions, do work on whiteboards, etc.

If the class is structured as a homework-based password-guessing exercise, then the LLM cheaters win. But if the class is structured as an in-person discussion, the LLM cheaters lose.