In a world where AI progress has wildly accelerated chip manufacture

This world?

What distinction are you making between "visualising" and "seeing"?

Good question!  By "seeing" I meant having qualia, an apparent subjective experience.  By "visualizing" I meant...something like using the geometric intuitions you get by looking at stuff, but perhaps in a philosophical zombie sort of way?  You could use non-visual intuitions to count the vertices on a polyhedron, like algebraic intuitions or 3D tactile intuitions (and I bet blind mathematicians do).  I'm not using those.  I'm thinking about a wireframe image, drawn flat.

I'm visualizing a rhombicosidodecahedron right now.  If I ask myself "The pentagon on the right and the one hiding from view on the left -- are they the same orientation?", I'll think "ahh, let's see...  The pentagon on the right connects through the squares to those three pentagons there, which interlock with those 2/4 pentagons there, which connect through squares to the one on the left, which, no, that left one is upside-down compared to the one on the right -- the middle interlocking pentagons rotated the left assembly 36° compared to the right".  Or ask "that square between the right pentagon and the pentagon at 10:20 above it <mental point>.  Does perspective mean the square's drawn as a diamond, or a skewed rectangle, weird quadrilateral?" and I think "Nah, not diamond shaped -- it's a pretty rectangular trapezoid.  The base is maybe 1.8x height?  Though I'm not too good at guessing aspect ratios?  Seems like I if I rotate the trapezoid I can fit 2 into the base but go over by a bit?"

I'm putting into words a thought process which is very visual, BUT there is almost no inner cinema going along with those visualizations.  At most ghostly, wispy images, if that.  A bit like the fleeting oscillating visual feeling you get when your left and right eyes are shown different colors?

...I do not believe this test.  I'd be very good at counting vertices on a polyhedron through visualization and very bad at experiencing the sensation of seeing it.  I do "visualize" the polyhedra, but I don't "see" them.  (Frankly I suspect people who say they experience "seeing" images are just fooling themselves based on e.g. asking them to visualize a bicycle and having them draw it)

Thanks for crossposting!  I've highly appreciated your contributions and am glad I'll continue to be able to see them.

Quick summary of a reason why constituent parts like of super-organisms, like the ant of ant colonies, the cells of multicellular organisms, and endosymbiotic organelles within cells^[1] are evolutionarily incentivized to work together as a unit:

Question: why do ants seem to care more about the colony than themselves?  Answer: reproduction in an ant colony is funneled through the queen.  If the worker ant wants to reproduce its genes, it can't do that by being selfish.  It has to help the queen reproduce.  Genes in ant workers have nothing to gain by making their ant more selfish and have much to gain by making their worker protect the queen.

This is similar to why cells in your pancreas cooperate with cells in your ear.  Reproduction of genes in the body is funned through gametes.  Somatic evolution does pressure the cells in your pancreas to reproduce selfishly at the expense of cells in your ear (this is pancreatic cancer).  But that doesn't help the pancreas genes long term.  Pancreas-genes and the ear-genes are forced to cooperate with each other because they can only reproduce when bound together in a gamete.

This sort of bounding together of genes making disperate things cooperate and act like a "super organism" is absent in members of a species.  My genes do not reproduce in concert with your genes.  If my genes figure out a way to reproduce at your expense, so much the better for them.

1. ^{^}

   Like mitochondria and chloroplasts, which were separate organisms but evolved to work so close with their hosts that they are now considered part of the same organism.

EDIT Completely rewritten to be hopefully less condescending.

There are lessons from group selection and the extended phenotype which vaguely reduce to "beware thinking about species as organisms".  It is not clear from this essay whether you've encountered those ideas.  It would be helpful for me reading this essay to know if you have.

Hijacking this thread, has anybody worked through Ape in the coat's anthropic posts and understood / gotten stuff out of them?  It's something I might want to do sometime in my copious free time but haven't worked up to it yet.

Sorry, that was an off-the-cuff example I meant to help gesture towards the main idea.  I didn't mean to imply it's a working instance (it's not).  The idea I'm going for is:

• I'm expecting future AIs to be less single LLMs (like Llama) and more loops and search and scaffolding (like o1)
• Those AIs will be composed of individual pieces
• Maybe we can try making the AI pieces mutually dependent in such a way that it's a pain to get the AI working at peak performance unless you include the safety pieces

This might be a reason to try to design AI's to fail-safe and break without controlling units.  E.g. before fine-tuning language models to be useful, fine-tune them to not generate useful content without approval tokens generated by a supervisory model.