Where do I start reading about this AI superintelligence stuff from the very basics on? I would especially be interesed in this: why do we consider our current paradigms of software and hardware 1) close enough to human intelligence in order to base a superintelligence on 2) why don't we think by the time we get there the paradigms will be different? I.e. AI rewriting its own source code? Why do we think AI is a software? Why do we think a software-hardware separation will make sense? Why do we think software will have a source code as we know it? Why would alphabetical letters invented thousands of years ago to express human sounds are ideal tools to define an intelligence? Even if yes, why would it be as today organized into words, functions and suchlike?
One obvious source if you haven’t already read it is Nick Bostrom’s Superintelligence. Bostrom addresses many of the issues that you list, e.g. an AI rewriting its own software, why an AI is likely to be software (and Bostrom discusses one or two non-software scenarios as well), etc. This book is quite informative and well worth reading, IMO.
Some of your questions are more fundamental than what is covered in Superintelligence. Specifically, to understand why “alphabetical letters invented thousands of years ago to express human sounds” are adequate for any...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.