When I was a teenager, I imagined that if you had just a tiny infinitesimally small piece of a curve - there would only be one moral way to extend it. Obviously, an extension would have to be connected to it, but also, you would want it to connect without any kinks. And just having straight-lines connected to it wouldn't be right, it would have to be curved in the same sort of way - and so on, to higher-and-higher orders. Later I realized that this is essentially what a Taylor series is.

I also had this idea when I was learning category theory that objects were points, morphisms were lines, composition was a triangle, and associativity was a tetrahedron. It's not especially sophisticated, but it turns out this idea is useful for n-categories.

Recently, I have been learning about neural networks. I was working on implementing a fairly basic one, and I had a few ideas for improving neural networks: making them more modular - so neurons in the next layer are only connected to a certain subset of neurons in the previous layer. I read about V1, and together, these led to the idea that you arrange things so they take into account the topology of the inputs - so for image processing, having neurons connected to small, overlapping, circles of inputs. Then I realized you would want multiple neurons with the same inputs that were detecting different features, and that you could reuse training data for neurons with different inputs detecting the same feature - saving computation cycles. So for the whole network, you would build up from local to global features as you applied more layers - which suggested that sheaf theory may be useful for studying these. I was planning to work out details, and try implementing as much of this as I could (and still intend to as an exercise), but the next day I found that this was essentially the idea behind convolutional neural networks. I'm rather pleased with myself since CNNs are apparently state-of-the-art for many image recognition tasks (some fun examples). The sheaf theory stuff seems to be original to me though, and I hope to see if applying Gougen's sheaf semantics would be useful/interesting.

I really wish I was better at actually implementing/working out the details of my ideas. That part is really hard.

What sophisticated ideas did you come up with independently before encountering them in a more formal context?

I'm pretty sure that in my youth I independently came up with rudimentary versions of the anthropic principle and the Problem of Evil. Looking over my Livejournal archive, I was clearly not a fearsome philosophical mind in my late teens, (or now, frankly), so it seems safe to say that these ideas aren't difficult to stumble across.

While discussing this at the most recent London Less Wrong meetup, another attendee claimed to have independently arrived at Pascal's Wager. I've seen a couple of different people speculate that cultural and ideological artefacts are subject to selection and evolutionary pressures without ever themselves having come across memetics as a concept.

I'm still thinking about ideas we come up with that stand to reason. Rather than prime you all with the hazy ideas I have about the sorts of ideas people converge on while armchair-theorising, I'd like to solicit some more examples. What ideas of this sort did you come up with independently, only to discover they were already "a thing"?

OK, so the second half of your post discussing the pros and cons of mechanical and biological modification assumes a world without AGI? Otherwise, the endeavor is useless because the AGI could simply figure it out for us.

My primary purpose of AGI: to create a perpetual state of bliss for all current humans (maybe future generations and other sentient beings as well, but that's a longer discussion). I'll trade "creative glorious work" for Heaven any day of the week. Even if you require the satisfaction from working to achieve bliss, the AGI can oblige you.

It actually poses the question, are the current limits on human intelligence due to the human being's genetic design or is it due to poor education?

Education can't make dumb people smart, but it can work wonders for naturally smart people. The other day I suggested that if we had to put a badly run country into receivership and straighten it out, I would pick North Korea because of the natural experiment that has happened on that peninsula. Their cousins to the south show that the extended Korean tribe has the genetic goods to benefit from the investment.

"Instead our army of AGI has robbed us of that."

If by the human valuation system this would be a loss compared to the alternative, and if the AGI accurately promoted human values, doesn't it follow that it would not choose to so "rob" us?

Given that a friendly AGI’s intellect dwarfs our own, why not ask it how to improve ourselves? It will consider all your concerns and more. Once you get close to the AGI’s intellect, then you will encounter and appreciate unsolved problems. Personally, I’d prefer to be a wirehead.

You missed the most straightforward way of enhancing humans: Education.

I long pondered on the concepts above. I had come up with the conclusion "Every movement needs a poet." In your discussion Jesus was one such poet. Its one thing to issue a command to a man's mind, it is quite another altogether to issue a command to a man's soul.

You used examples of revolutionary America, lets look at the details of that a bit more. We had a combination of excellent leaders leading up to that war all of them experts in the field of politics, including George Washington (whom claimed he didn't want the post of commander of the continental army but showed up to the meetings to pick one in a military uniform). As a violent civil disturbance turned successful the continental congress decided that succession was the best plan and commissioned the writing of the declaration of independence. Instead of a full committee writing the document the preamble in all of its poetic glory was written by Thomas Jefferson. Jefferson had filled the role of the fantastic poet and had collected all of the confused feelings and ideas of the founders into one easy to understand and powerful document. He had addressed those who opposed Britain over the issue of slavery as well as those who opposed Britain's increasing autocratic nature. Most importantly his document created a beacon to command that generation and later generations to live up to a set of ideals as the ultimate goal for our nation. The poet commanded the people to be and now as you argued they needed someone with practical skills to make it happen. Its important to note that with such a beacon men are willing to endure no end of hardship to see it happen.

So I wouldn't disagree that "Every Jesus needs a Paul" but I DO argue that if a great poet rises then so too will the fantastic men needed to make the poet's vision happen. Men grow, create and learn the most when necessity demands that they do so. If there is a poet commanding men to greatness then great men is what we will have. Its important to note that for poet to be successful in this manner they must profess ideals that are "virtuous", the more honorable and virtuous they appear the easier it is to find and create great men as well as avoid resistance.

Lets look at your example of Washington and the continental army: Washington was a gallant figure and dressed as such on purpose to command people emotionally. One of his core strategies was to create an image that his men could look up to. As such he was dutiful and never sought leave of the army so he was always there working to further the cause. He was courageous and brave, showing no personal concern at sitting on a horse in a hail of gunfire as he screamed at his men to stand strong. He was even merciful doing his utmost to see to the needs of the enemy wounded and captured. The combination of virtues and the vast quality of his moral superiority meant that great men flocked to him. The first were Nathaniel Green and Henry Knox. Yet he would soon have much needed support form men like Marquis de Lafayette, a man who served with such distinction that there are towns and cities across the United States that are named after him.

So yes, every Jesus needs a Paul. Yet interestingly possibly because every Jesus needs a Paul, for every great Jesus out there a Paul will surely be "born"

How DOES one change a group of people's moral compass?

Look at Germany in the 1930s.