Wanted to add the insights of Neil Gershenfeld which I think is how we should frame these problems:
We've already had a digital revolution; we don't need to keep having it. The next big thing in computers will be literally outside the box, as we bring the programmability of the digital world to the rest of the world.
He was talking about personal fabrication in this context, but the 'digitization' of the physical world is applicable to the sustainability goals I mentioned. Using operations research, loosely-coupled distributed architectures, nature-inspired routing algorithms, and other tricks of the IT trade applied to natural resources, we can finally transition to a sustainable world.
Surprised no one has mentioned anything involving sustainable/clean tech (energy, food, water, materials). This site does stress existential threats, and I'd think that, given many (most?) societal collapses in the past were precipitated, at least partly, by resource collapse, I'd want to concentrate much of the startup activity around trying to disrupt our short-term wasteful systems. Large pushes to innovate and disrupt the big four (energy, food, water, materials) would do more than anything I can think of to improve the condition of our world and minimize the major risks confronting us within the next 100 years (or sooner).
It's not as hopeless as it appears on first glance.... (read more)
Ok, so we can with confidence say that humans and other organisms with developed neural systems experience the world subjectively, maybe not exactly in similar ways, but conscious experience seems likely for these systems unless you are a radical skeptic or solipsist. Based on our current physical and mathematical laws, we can reductively analyse these systems and see how each subsystem functions, and, eventually, with sufficient technology we'll be able to have a map of the neural correlates that are active in certain environments and which produce certain qualia. Neuroscientists are on that path already. But, are only physical nervous systems capable of producing a subjective experience? If... (read 419 more words →)
That is the $64,000 question.
To summarize (mostly for my sake so I know I haven't misunderstood the OP):
We have evolved moral intuitions such as empathy and compassion that underly what we consider to be right or wrong. These intuitions only work because we consciously internalize another agent's subjective experience and identify with it. In other words, without the various quales that we experience we would have no foundation to act ethically. An unconscious AI that does not experience these quales could, in theory, act the way we think it should act by mimicking behaviors from a repertoire of rules (and ways to create further rules) that we give it, but that is a very brittle and complicated route, and is the route the SIAI has been taking because they have discounted qualia, which is what this post is really all about.
"How an algorithm feels from inside" discusses a particular quale, that of the intuitive feeling of holding a correct answer from inside the cognizing agent. It does not touch upon what types of physically realizable systems can have qualia.
"If everything real is made of physics, you still must either explain how certain patterns of neuronal excitations are actually green, or you must assert that nothing is actually green at any level of reality."
This is a 'why' question, not a 'how' question, and though some 'why' questions may not be amenable to deeper explanations, 'how' questions are always solvable by science. Explaining how neuronal patterns generate systems with subjective experiences of green is a straightforward, though complex, scientific problem. One day we may understand this so well that we could engineer quales on demand, or create new types of never before seen quales according to some transformation rules. However, explaining 'why' such arrangements of matter should possess such interiority or subjectivity is, I think at least based on everything we currently know, unanswerable.
In my opinion, the most relevant article was from Drew McDermott, and I'm surprised that such an emphasis on analyzing the computational complexity of approaches to 'friendliness' and self-improving AI has not been more common. For that matter, I think computational complexity has more to tell us about cognition, intelligence, and friendliness in general, not just in the special case of a self-improving optimization/learning algorithms, and could completely modify the foundational assumptions underlying ideas about intelligence/cognition and the singularity.
I'm thinking of specific concepts by Yudkowsky and others in the singularity/FAI crowd that seem uncontroversial at first glance, but upon further investigation, when analyzed in the light of computational complexity, become unconvincing.... (read more)
I am, and have been for most of my life, an information glutton. The internet has made my affliction worse by providing me with the equivalent of an unlimited buffet of both nutritious as well as junk food for my brain which never leaves my side. A fire hose of data focused straight into my mind's mouth. If the brain food is mostly high quality, and I'm exercising my grey matter vigorously enough to warrant such high volumes of knowledge, then it's not that much of a problem. However, I've recently crossed a threshold where I seem to be spending more time navigating this buffet rather than consuming the food.
Ok, dropping... (read more)
From what I've read of the source writings within the contemplative traditions, modern neuroscience studies and theories on meditation, and my own experiences and thoughts on the subject as well, I've come to view the practice of meditation as serving 3 different but interconnected purposes: 1.) ego loss, 2.) cultivation of compassion, and 3.) experience of non-dual reality.
Ego loss means inhibiting or eliminating the internal self-critic by changing the way you perceive the target of that critic, namely the concept of a stable 'self' that you identify with. Suffering is caused by the struggle of attachment, which is a manifestation of your internal self-critic admonishing your identified 'self' over... (read more)