Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

In response to Gatekeeper variation
Comment author: kilobug 10 August 2015 01:41:27PM 1 point [-]

This wont work, like with all other similar schemes, because you can't "prove" the gatekeeper down to the quark level of what makes its hardware (so you're vulnerable to some kind of side-attack, like the memory bit flipping attack that was spoken about recently), nor shield the AI from being able to communicate through side channels (like, varying the temperature of its internal processing unit which it turns will influence the air conditioning system, ...).

And that's not even considering that the AI could actually discover new physics (new particles, ...) and have some ability to manipulate them with its own hardware.

This whole class of approach can't work, because there are just too many ways for side-attacks and side-channels of communication, and you can't formally prove none of them are available, without going down to making proof over the whole (AI + gatekeeper + power generator + air conditioner + ...) down at Schrödinger equation level.

Comment author: kilobug 02 August 2015 07:22:58AM 0 points [-]

To be fair, the DRAM bit flipping thing doesn't work on ECC RAM, and any half-decent server (especially if you run an AI on it) should have ECC RAM.

But the main idea remains yes : even a program proven to be secure can be defeated by attacking one of the assumptions made (such as the hardware being 100% reliable, which it rarely is) in the proof. Proving a program to be secure down from applying Schrödinger's equation on the quarks and electrons the computer is made of is way beyond our current abilities, and will remain so for a very long time.

Comment author: kilobug 08 June 2015 02:31:05PM 2 points [-]

I see your point, but I think you're confusing a partial overlapping with an identity.

There are many bugs/uncertainty that appear as agency, but there are also many bugs/uncertainty which doesn't appear as agency (as you said about true randomness), and there are also behavior that are actually smart and that appear as agency because of smartness (like the way I was delighted with Emacs the first time I realized that if I asked it to replace "blue" with "red", it would replace "Blue" with "Red" and "BLUE" with "RED"), I got the same "feeling of agency" there that I could have on bugs.

So I wouldn't say that agency is bugs, but that we have evolved to mis-attribute attribute agency to things that are dangerous/unpleasant (because it's safest to mis-attribute agency to nothing that doesn't have it, than to not attribute it to something that does have it), the same way our ancestors used to see the sun, storms, volcanoes, ... as having agency.

Agency is something different, hard to exactly pinpoint (philosophers have been going at it for centuries), but that involves ability to have a representation of reality, to plan ahead for a goal, a complexity of representation and ability to explore solution-space in a way that will end up surprising us, not because of bugs, but because of its inherent complexity. And we have been evolved to mis-attribute agency to things which behave in unexpected ways. But that's a bug of our own ability to detect agency, not a feature of agency itself.

Comment author: gurugeorge 07 June 2015 12:09:56AM 1 point [-]

I remember reading a book many years ago which talked about the "hormonal bath" in the body being actually part of cognition, such that thinking of the brain/CNS as the functional unit is wrong (it's necessary but not sufficient).

This ties in with the philosophical position of Externalism (I'm very much into the Process Externalism of Riccardo Manzotti). The "thinking unit" is really the whole body - and actually finally the whole world (not in the Panpsychist sense, quite, but rather in the sense of any individual instance of cognition being the peak of a pyramid that has roots that go all the way through the whole).

I'm as intrigued and hopeful about the possibility of uploading, etc., as the next nerd, but this sort of stuff has always led me to be cautious about the prospects of it.

There may also a lot more to be discovered about the brain and body too, in the area of some connection between the fascia and the immune system (cf. the anecdotal connection between things like yoga and "internal" martial arts and health).

Comment author: kilobug 08 June 2015 01:22:52PM 1 point [-]

I'm really skeptical of claims like « the "thinking unit" is really the whole body », they tend to discard quantitative considerations for purely qualitative ones.

Yes, the brain is influenced, and influences, the whole body. But that doesn't mean the whole body has the same importance in the thinking. The brain is also influenced by lots of external factors (such as ambient light or sounds, ...) if as soon as there is a "connection" between two parts you say "it's the whole system that does the processing", you'll just end up considering the solar system as a whole, or even the entire event horizon sphere.

There is countless evidence that, while your body and your environment have significant influence on your thinking, it's just influence, not fundamentally being part of the cognition. For example, people who have graft or amputations rarely change personality, memory or cognitive abilities in any way comparable to what brain damage can do.

Comment author: kilobug 04 June 2015 08:43:28AM 1 point [-]

A little nitpicking about the "2 dice" thing : usually when you throw you two dices, it doesn't matter which dice gives which result. Sure you could use colored dices and have the "blue 2, red 3" be different than "blue 3, red 2", but that's very rarely the case. Usually you do the sum (or look for patterns like doubles) but "2, 3" and "3, 2" are equivalent, and in that case the entropy isn't the double, but lower.

What you wrote is technically right - but goes against the common usage of dices, so it would be worth adding a footnote or precision about that, IMHO.

Comment author: kilobug 26 May 2015 08:18:39AM 5 points [-]

I'm not really sure the issue is about "direction", but more about people who have enough time and ideas to write awesome (or at least, interesting) posts like the Sequences (the initial ones by Eliezer or the additional ones by various contributors).

What I would like to see are sequences of posts that build on each other, starting from the basics and going to deep things (a bit like Sequences). It could be collective work (and then need a "direction"), but it could also be the work of a single person.

As for myself, I did write a few posts (a few in main and a few in discussions) but if I didn't write recently is mostly because of three issues :

  1. Lack of time, like I guess many of us.

  2. The feeling of not being "good enough", that's the problem with a community of "smart" people like LW, with high quality base content (the Sequences), it's a bit intimidating.

  3. The "taboo" subjects (like politics) which I do understand and respect, but they limit what I could write about.

There are a few things I would like to write about, but either I feel I lack the skill/knowledge to do it at LW level (point 2) or they border too much the "taboo" subjects (point 3).

Comment author: Vaniver 20 April 2015 01:33:11PM *  2 points [-]

My view on the "Fermi paradox" is not that there is a single filter, cutting ~10 orders of magnitude (ie, from 10 billions of planets in our galaxy with could have a life to just one), but more a combination of many small filters, each taking their cuts.

I don't think that the Great Filter implies only one filter, but I think that if you're multiplying several numbers together and they come out to at least 10^10, it's likely that at least one of the numbers is big. (And if one of the numbers is big, that makes it less necessary for the other numbers to be big.)

Put another way, it seems more likely to me that there is one component filter of size 10^6 than two component filters each of size 10^3, both of which seem much more likely than that there are 20 component filters of size 2.

Comment author: kilobug 22 April 2015 09:07:38AM 1 point [-]

I don't see why it's likely one of the numbers has to be big. There are really are lots of complicated steps you need to cross to go from inert matter to space-faring civilizations, it's very easy to point a dozen of such steps that could fail in various ways or just take too long, and there many disasters that can happen to blow everything down.

If you've a long ridge to climb in a limited time and most people fail to do it, it's not very likely there is a very specific part of it which is very hard, but (unless you've actual data that most people fail at the same place) more likely that are lots of moderately difficult parts and few people succeed in all of them in time.

Or if you've a complicated project that takes 4x longer than expected to be done, it's much less likely that there was a single big difficulty you didn't foresee than many small-to-moderate difficulties you didn't foresee stacking on top of each other. The planning fallacy isn't usually due to black swans, but to accumulating smaller factors. It's the same here.

Comment author: kilobug 20 April 2015 11:35:17AM 4 points [-]

There is a thing which really upsets me with the "Great Filter" idea/terminology, is that implies that it's a single event (which is either in the past or the future).

My view on the "Fermi paradox" is not that there is a single filter, cutting ~10 orders of magnitude (ie, from 10 billions of planets in our galaxy with could have a life to just one), but more a combination of many small filters, each taking their cuts.

To have intelligent space-faring life, we need a lot of things to happen without any disaster (nearby supernova, too big asteroid, ...) disrupting it too much. It's more a like a "game of the goose", where the optimization process steadily advance but events will either accelerate it or make it go backwards, and you need to reach the "win" cell before time runs out (ie, your star becoming too hot as the Sun will be in less than a billion of years) or reaching a '"you lose" cell (a nearby supernova blasting away your atmosphere, or a thermonuclear warfare).

I don't see any reason to believe there is a single "Great Filter", instead of a much more continuous process with many intermediate filters you have to pass through.

Comment author: kilobug 07 April 2015 08:46:14AM 1 point [-]

Nicely put for an introduction, but of course things are in reality not as clear-cut, "rationality" changing the direction and "desire" the magnitude.

  1. Rationality can make you realize some contradictions between your desires, and force you to change them. It can also make you realize that what you truly desire isn't what you thought you desired. Or it can make you desire whole new things, that you didn't believe to be possible initially.

  2. Desire will affect the magnitude because it'll affect how much effort you put in your endeavor. With things like akrasia and procastination around, if you don't have strong desire to do something, you are much likely to do it, especially if there is an initial cost. That's what Eliezer calls "something to protect".

Of course those two are mostly positive feedback between rationality and desire, but there also be can negative feedbacks between the two, usually due to human imperfections.

Comment author: kilobug 20 March 2015 02:33:00PM 1 point [-]

Interesting idea, thanks for doing it, but saddly many questions are very US-centric. It would be nice to have some "tags" on the questions, and let the users select which kind of questions he wants (for example the non-US people could remove the US-specific ones).

View more: Next