If you do read up on the history of homology, I'd be grateful if you'd tell me whether it helps. I'm curious if my advice was actually a good idea, or if it just sounds good. 

Have you looked at the historical development of homology? IME that's generally a good way to build intuitions on a topic, as it tends to make concepts appear natural.

If you were to transport me and Algon into the wonderful universe of Mother of Learning [...] I'd absolutely mog him. I haven't even read the book, but now I know for sure I'd mog him.

As I told Croissanthology privately after he wrote this reply, I never said I wouldn't co-operate with my clone/original. 

Fairly active Less wrong users? Probably not <10%, though I wouldn't be shocked, just surprised.

consequent shame afterwards

Speak for yourself.

The upshot is that, it seems to me, oxytocin is pretty antithetic to ambition. And not just ambition “at the grand scale”; also smaller-scale ambitions, relevant to the whole range of non-oxytocin-driven values. 

Or in other words, "Family, religion, friendship. These are the three demons you must slay if you wish to succeed in business." 

What was the novel?

No, and that is a good idea. 

Tangentially relevant: Armstrong has a post on some general features that make it likely that Goodharting will be a problem. Conversely, he gives some features that make it less likely you'll wind up Goodharting in your search. 

So, as long as:

1. We use a Bayesian mix of reward functions rather than a maximum likelihood reward function.
2. An ideal reward function is present in the space of possible reward functions, and is not penalised in probability.
3. The different reward functions are normalised.
4. If our ideal reward functions have diminishing returns, this fact is explicitly included in the learning process.

Then, we shouldn't unduly fear Goodhart effects (of course, we still need to incorporate as much as possible about our preferences into the AI's learning process). The second problem, making sure that there are no unusual penalties for ideal reward functions, seems the hardest to ensure.

In the roguelike Angband, the player has to descend 100 floors of the titular dungeon to kill Morgoth and beat the game. Novices hang about the shallower floors, slaying mini-bosses, gaining levels, collecting loot and not descending to further levels. Inevitably, they die well before reaching floor 100. Old hands advise against this and say to beat the game you need to focus on going down. Inevitably, novices do not listen until they've died countless times and beaten the game. At which point, the cycle repeats. 

All of that is to say: it's true that you need to play to win. It's also true that people don't get this. One exception is people who've played to win in the past, usually after many failures as they stubbornly ignore advice on how to win.

I should know, as I was that novice in Angband at one point. Eventually, I gave in and gave in to victory, descending as fast as I could handle. For quite some time, this was difficult. Even though I explicitly aimed to win, for some reason I would eventually just lose the ability to focus on descending and start puttering about collecting loot like a magpie. Rest was required before I could get my head back in the game. If I didn't, the result was death. 

Why exactly my ability to play to win could be sapped so easily confused me. I wasn't sure if it was because of bad habits, or because of some generalized failure to follow an explicit goal or what. So once I beat Angband, I decided to determine why I struggled to win by playing another roguelike where you win by descending through a dungeon: Noita. 

And what do you know, I still struggled to play to win! I still struggled to descend or execute on any strategy like "acquire as much health as possible", "beat this dude to unlock this spell for future runs" etc. So it wasn't bad habits that killed me in Angband, but a generalized difficulty in optimizing for a goal for an extended period of time. 

Perhaps then playing to win is like a resource you can spend, and most people just don't care to spend that resource on board games. More optimistically, playing to win is like a muscle you can train. In which case, you want to play to win at board games. 

I'm not sure which is true. But either way, everyone should experience of playing to win at least a few times in their lives, till they learn what it feels like and can tell when they're playing to win, and when they're not.