Comment author: Eugine_Nier 14 August 2012 12:03:52AM *  2 points [-]

There must be some fundamental difference between how one draws inferences from mental states versus everything else.

Talking about "drawing inferences from mental states" strikes me as a case of the homunculus fallacy, i.e., thinking that there's some kind of homunculus sitting inside our brains looking at the mental states and drawing inferences. Whereas in reality mental states are inferences.

Comment author: TruePath 15 August 2012 09:57:07AM 3 points [-]

This objection points largely in the right direction but I don't think it's fair to accuse the view of adopting the homunculus fallacy. After all, the very suggestion is that our brains have circuitry that (in effect) performs Bayesian updating and that neurological damage and psychiatric conditions can cause this circuitry to misbehave. This is a way the brain could have worked. If the view adopted the homunculus fallacy then the Bayesian updating machinery couldn't, itself, be broken. It could only recieve bad input.

However, as I delineate in my comment, we have every reason to believe the brain doesn't have anything like a Bayesian updating module exercising control over all the other brain modules. Instead, the empirical evidence suggests a much simpler structure in which different brain regions vie to control our actions without any arbitration by some master Bayesian updating module. Otherwise, one couldn't explain our inclination to answer wrongly on tests that pit one part of the brain against another, e.g., our mistakes in identifying the color of text spelling the name of another color.

Also, to be pedantic the mental states aren't inferences. .The mental states merely determine behavior patterns that we can (sometimes) usefully describe as making certain inferences.

Comment author: TruePath 14 August 2012 09:55:21PM 13 points [-]

All of the theories presented in this post seem to make the implausible assumption that somehow the brain acts like a hypothetical ideally rational individual and that impairment somehow breaks some aspect of this rationality.

However, there is a great deal of evidence the brain works nothing like this. In contrast, it has many specific modules that are responsible for certain kinds of thought or behavior. These modules are not weighed by some rational actor that sifts through them, they are the brain. When these modules come into conflict, e.g., in the standard word/color test where yellow is spelled in red, fairly simply conflict resolution methods are brought into play. When things go wrong in the brain, either an impairment in conflict resolution mechanisms or in the underlying modules themselves, things will go wonky in specific (not general) ways.

Speaking from personal experience, being in a psychotic/paranoid state simply makes certain things seem super salient to you. You can be quite well aware of the rational arguments against the conclusion you are worrying about but it's just so salient that it 'wins.' In other words it also feels like there is just a failure in your ability to override certain misbehaving brain processes rather than some general inability to update appropriately. This is further supported by the fact that skizophrenics and others with delusions seem to be able to update largely appropriately in certain aspects, e.g., what answer is expected on a test, while still maintaining their delusional state.

Comment author: pnrjulius 19 June 2012 03:55:03AM -2 points [-]

No, it's still a tool, because Google Maps doesn't force you to go where it tells you, it only offers suggestions.

Comment author: TruePath 03 August 2012 04:22:58PM 0 points [-]

So does the evil manipulative psychologist or the manipulative lover who convinces you to commit crimes to prove you really love them.

And it's simply astounding some of the things unscrupulous psychologists and doctors have convinced people to do via mere suggestion. Psychologists have convinced people to sleep with their own fathers to 'resolve' their issues. Convincing people to do something that turns the AI into a direct (rather than indirect) agent seems fairly minor compared to what people convince each other to do all the time.

Hell, US presidents have prosecuted every major war we've been involved in, dropped the A-bomb, developed the H-bomb, etc... all merely by making suggestions to people. I doubt any president since Jackson has actually picked up a pistol or physically forced anyone to do anything. People are merely accustomed to doing as they suggest and that is the entirety of their power. Do you not believe people would become accustomed to just driving (or going, or doing) whatever the google recommend bot recommended?

Comment author: TruePath 03 August 2012 03:40:34PM 2 points [-]

I'm deeply confused. How can you even define the difference between tool AI and FAI?

I assume that even tool AI is supposed to be able to opine on relatively long sequences of input. In particular, to be useful it must be able to accumulate information over essentially unbounded time periods. Say if you want advise about where to position your air defenses you must be able to go back to the AI system each day hand it updates on enemy activity and expect it to integrate that information with information it received during previous sessions. Whether or not you upload this info each time you ask a quesiton or not in effect the AI has (periods) in which it is loaded with a significant amount of information about past events.

But now you face the problem that self-modification is indistinguishable from simple storing of data. The existence of universal Turing machines demonstrate that much. Simply by loading up information in memory one can generate behavior corresponding to any kind of (software) self-modification.

So perhaps the supposed difference is that this AI won't actually take direct actions, merely make verbal suggestions. Well it's awful optimistic to suppose no one will get lazy or exigencies won't drive them to connect a simple script up to the machine which takes say sentences of the form "I recommend you deploy your troops in this manner." and directly sends the orders. Even if so the machine still takes direct action in the form of making statements that influence human behavior.

You might argue that a tool AI is one in which the advice it generates doesn't require self-reference or consideration of it's future actions so it is somehow different in kind. However, again simple analysis reveals this can't be so. Imagine again the basic question of "How should I position my forces to defend against the enemy attack." Now, given that the enemy is likely to react in certain ways correct advice requires the tool AI to consider whether future responses will be orchestrated by itself or a human who will be unable to handle certain kinds of complexity or be inclined to different sorts of responses. Those even a purely advisory AI needs the ability to project likely outcomes based on it's on likely future behaviors.

Now it seems we are again in the realm of 'FAI' since one has to ensure that the advice given by the machine when presented with indefinitely long, complex historical records won't end up encouraging the outcome where someone ends up connecting permanent memory and wiring on the ability to take direct action. After all, if the advise is designed to be of maximum usefulness to the people asking the tool AI must be programmed to give advice that causes them to best achieve the goals they ask for advice in achieving. Since such goals could quite reasonably be advanced by the ability of the AI to take direct action and the reasons for the advice can't ever be entirely explained to humans (even deep blue goes beyond being able to do that to humans now) I don't see how the problem isn't just as complicated as 'FAI'.

I guess it comes down to my belief that if you can't formulate the notion precisely I'm skeptical it's coherent.

Comment author: TruePath 03 August 2012 12:16:45PM 1 point [-]

<blockquote> Another concern is that the true hypothesis may be incomputable. There are known definitions of binary sequences which make sense, but which no Turing machine can output. Solomonoff induction would converge to this sequence, but would never predict it exactly. </blockquote>

This statement is simply false. There are a bunch of theorems in machine learning about the class of sequences that an algorithm would converge to predicting (depending on your definition of convergence...is it an algorithm that is eventually right...an algorithm for guessing algorithms that eventually always give the correct answer etc..) but every single one of them is contained in the tiny class of sequences computable in 0'.

To put it another way given any algorithm that guesses at future outputs there is some sequence that simply disagrees with that algorithm at each and every value. Thus any computable process is maximally incorrect on some sequence (indeed is infinitely often wrong on the next n predictions for every n on a dense uncountable set of sequences...say the 1-generics).

However, this is not the end of the story. If you allow your hypothesis to be probabilistic things get much more interesting. However, exactly what it means for induction to 'work' (or converge) in such a case is unclear (you can't get it to settle on a single theory and never revise even if the theory is probabilistic).

Comment author: [deleted] 29 January 2012 06:13:21AM 0 points [-]

As with 3 we would expect a simulation to bottom out and not provide arbitrarily fine grained structure but in simulations precision issues also bring with them questions of stability. If the law's of physics turn out to be relatively unaffected by tiny computational errors that would push in the direction of simulation but if they are chaotic and quickly spiral out of control in response to these errors it would push against simulation.

We can expect the laws of physics to be relatively stable, simulation or no, due to anthropic reasoning. If we lived in a universe where the laws of physics were not stable (on a timescale short enough for us to notice), it would be very difficult for intelligent life to form.

In response to comment by [deleted] on Evidence For Simulation
Comment author: TruePath 29 January 2012 11:12:12PM 0 points [-]

Here stability refers to numerical stability, i.e., whether or not minor errors in computation accumulate over time and cause the results to go wildly astray or do small random errors cancel out or at least not blow up.

Evidence For Simulation

14 TruePath 27 January 2012 11:07PM

The recent article on overcomingbias suggesting the Fermi paradox might be evidence our universe is indeed a simulation prompted me to wonder how one would go about gathering evidence for or against the hypothesis that we are living in a simulation.  The Fermi paradox isn't very good evidence but there are much more promising places to look for this kind of evidence.  Of course there is no sure fire way to learn that one isn't in a simulation, nothing prevents a simulation from being able to perfectly simulate a non-simulation universe, but there are certainly features of the universe that seem more likely if the universe was simulated and their presence or absence thus gives us evidence about whether we are in a simulation.

 

In particular, the strategy suggested here is to consider the kind of fingerprints we might leave if we were writing a massive simulation.  Of course the simulating creatures/processes may not labor under the same kind of restrictions we do in writing simulations (their laws of physics might support fundamentally different computational devices and any intelligence behind such a simulation might be totally alien).  However, it's certainly reasonable to think we might be simulated by creatures like us so it's worth checking for the kinds of fingerprints we might leave in a simulation.

 

Computational Fingerprints

Simulations we write face several limitations on the computational power they can bring to bear on the problem and these limitations give rise to mitigation strategies we might observe in our own universe.  These limitations include the following:

  1. Lack of access to non-computable oracles (except perhaps physical randomness).

    While theoretically nothing prevents the laws of physics from providing non-computable oracles, e.g., some experiment one could perform that discerns whether a given Turing machine halts (halting problem = 0') all indications suggest our universe does not provide such oracles.  Thus our simulations are limited to modeling computable behavior.  We would have no way to simulate a universe that had non-computable fundamental laws of physics (except perhaps randomness).

    It's tempting to conclude that the fact that our universe apparently follows computable laws of physics modulo randomness provides evidence for us being a simulation but this isn't entirely clear.  After all had our laws of physics provided access to non-computable oracles we would presumably not expect simulations to be so limited either.  Still, this is probably weak evidence for simulation as such non-computable behavior might well exist in the simulating universe but be practically infeasable to consult in computer hardware.  Thus our probability for seeing non-computable behavior should be higher conditional on not being a simulation than conditional on being a simulation.
  2. Limited ability to access true random sources.

    The most compelling evidence we could discover of simulation would be the signature of a psuedo-random number generator in the outcomes of `random' QM events.  Of course, as above, the simulating computers might have easy access to truly random number generators but it's also reasonable they lack practical access to true random numbers at a sufficient rate.
  3. Limited computational resources. 

    We always want our simulations to run faster and require less resources but we are limited by the power of our hardware.  In response we often resort to less accurate approximations when possible or otherwise engineer our simulation to require less computational resources.  This might appear in a simulated universe in several ways.
    • Computationally easy basic laws of physics. For instance the underlying linearity of QM (absent collapse) is evidence we are living in a simulation as such computations have a low computational complexity.  Another interesting piece of evidence would be discovering that an efficient global algorithm could be used that generates/uses collapse to speed computation.
    • Limited detail/minimal feature size.  An efficient simulation would be as course grained as possible while still yielding the desired behavior.  Since we don't know what the desired behavior might be for a universe simulation it's hard to evaluate this criteria but the indications that space is fundamentally quantized (rather than allowing structure at arbitrarily small scales) seems to be evidence for simulation.
    • Substitution of approximate calculations for expensive calculations in certain circumstances.  Weak evidence could be gained here by merely observing that the large scale behavior of the universe admits efficient accurate approximations but the key piece of data to support a simulated universe would be observations revealing that sometimes the universe behaved as if it was following a less accurate approximation rather than behaving as fundamental physics prescribed.  For instance discovering that distant galaxies behave as if they are a classical approximation rather than a quantum system would be extremely strong evidence. 
    • Ability to screen off or delay calculations in regions that aren't of interest.  A simulation would be more efficient if it allowed regions of less interest to go unsimilated or at least to delay that simulation without impacting the regions of greater interest.  While the finite speed of light arguably provides a way to delay simulation of regions of lesser interest QM's preservation of information and space-like quantum correlations may outweigh the finite speed of light on this point tipping it towards non-simulation.
  4. Limitations on precision.

    Arguably this is just a variant of 3 but it has some different considerations.  As with 3 we would expect a simulation to bottom out and not provide arbitrarily fine grained structure but in simulations precision issues also bring with them questions of stability.  If the law's of physics turn out to be relatively unaffected by tiny computational errors that would push in the direction of simulation but if they are chaotic and quickly spiral out of control in response to these errors it would push against simulation.  Since linear systems are virtually always stable te linearity of QM is yet again evidence for simulation.
  5. Limitations on sequential processing power.

    We find that finite speed limits on communication and other barriers prevent building arbitrarily fast single core processors.  Thus we would expect a simulation to be more likely to admit highly parallel algorithms.  While the finite speed of light provides some level of parallelizability (don't need to share all info with all processing units immediately) space-like QM correlations push against parallelizability.  However, given the linearity of QM the most efficient parallel algorithms might well be semi-global algorithms like those used for various kinds of matrix manipulation.  It would be most interesting if collapse could be shown to be a requirement/byproduct of such efficient algorithms.
  6. Imperfect hardware

    Finally there is the hope one might discover something like the Pentium division bug in the behavior of the universe.  Similarly one might hope to discover unexplained correlations in deviations from normal behavior, e.g., correlations that occur at evenly spaced locations relative to some frame of reference, arising from transient errors in certain pieces of hardware.

Software Fingerprints

Another type of fingerprint that might be left are those resulting from the conceptual/organizational difficulties occuring in the software design process.  For instance we might find fingerprints by looking for:

  1. Outright errors, particularly hard to spot/identify errors like race conditions or the like.  Such errors might allow spillover information about other parts of the software design that would let us distinguish them from non-simulation physical effects.  For instance, if the error occurs in a pattern that is reminiscent of a loop a simulation might execute but doesn't correspond to any plausible physical law it would be good evidence that it was truly an error.
  2. Conceptual simplicity in design.  We might expect (as we apparently see) an easily drawn line between initial conditions and the rules of the simulation rather than physical laws which can't be so easily divided up, e.g., laws that take the form of global constraint satisfaction.  Also relatively short laws rather than a long regress into greater and greater complexity at higher and higher energies would be expected in a simulation (but would be very very weak evidence).
  3. Evidence of concrete representations.  Even though mathematically relativity favors no reference frame over another often conceptually and computationally it is desierable to compute in a particular reference frame (just as it's often best to do linear algebra on a computer relative to an explicit basis).  One might see evidence for such an effect in differences in the precision of results or rounding artifacts (like those seen in re-sized images).

Design Fingerprints

This category is so difficult I'm not really going to say much about it but I'm including it for completeness.  If our universe is a simulation created by some intentional creature we might expect to see certain features receive more attention than others.  Maybe we would see some really odd jiggering of initial conditions just to make sure some events of interest occurred but without a good idea what is of interest it is hard to see how this could be done.  Another potential way for design fingerprints to show up is in the ease of data collection from the simulation.  One might expect a simulation to make it particularly easy to sift out the interesting information from the rest of the data but again we don't have any idea what interesting might be.

 

Other Fingerprints

I'm hoping the readers will suggest some interesting new ideas as to what one might look for if one was serious about gathering evidence about whether we are in a simulation or not.

Comment author: TruePath 27 January 2012 08:51:49PM 0 points [-]

I don't think this kind of reaction is troubling. If you start to feel the same way about professional colleagues with supernatural beliefs then it's an issue but this seems to be just the normal human anger.dissapointment at not being able to find an appropriate social community.

Atheists, and rationalists more generally, find it very hard to feel at home in most churchs and even things like book clubs. This isn't prejudice, it's just the fact that psychologically you won't feel comfortable if you have to always hide your true feelings about the main focus of the event. Watch the way people behave at a party. <B>No one is comfortable talking about subjects they feel they need to hide their opinions to avoid giving offense</B>. The natural (and appropriate) response is to change the subject and talk about something else. But it just isn't possible to change the topic of church away from the supernatural or change the topic of book clubs away from giving arguments (often rationalists give arguments or voice standards that others don't relate to).

So yah, this guy's supernatural inclinations took over the meeting and effectively denied you the chance to feel comfortable and a member of the community. It would be surprising if this didn't annoy you.

If you start feeling the same way in situations where there is no real community (many workplaces) or where the topic can be easily avoided then it's an issue.

Comment author: Vaniver 22 November 2011 07:18:09PM 2 points [-]

However, I don't see where "value of information" shows up in this framework anywhere. Do I need to distinguish some choice nodes as "information gathering", and apply a default strategy to them as "don't gather any information", and then compute value of information as the difference between best I can do with my eyes closed and the best I can do flat out?

Think of VoI as going in the reverse direction. That is, beforehand you would have modeled your test outcome as a nature node because you didn't consider the option of not running the test. Now you stick in a choice node of "run the test" that leads to the nature node of the test output on the one branch, and the tree where you don't know the test output on the other branch. Like you suggest, you then use the work-backwards algorithm to figure out the optimal decision at the "run the test" node, and the difference between the branch node values is the absolute value of the VoI minus the test cost.

What if there is no natural partition of some actions as information gathering and not-information-gathering?

Then VoI won't help you very much. VoI is a concept that helps in specifying decision problems- building the tree- not computing a tree to find an optimal policy. It suggests modeling information-gathering activities as choice nodes leading to nature nodes, rather than just nature nodes. If you've got a complete decision problem already, then you don't need VoI.

I should point out that most tests aren't modeled as just information-gathering. If a test is costless, then why not run it, even if you throw the results away? Typically the test will have some cost, in either utility or prospects, and so in some sense there's rarely actions that are purely information gathering.

Comment author: TruePath 20 December 2011 08:53:51AM 1 point [-]

Think of VoI as going in the reverse direction. That is, beforehand you would have modeled your test outcome as a nature node because you didn't consider the option of not running the test. Now you stick in a choice node of "run the test" that leads to the nature node of the test output on the one branch, and the tree where you don't know the test output on the other branch. Like you suggest, you then use the work-backwards algorithm to figure out the optimal decision at the "run the test" node, and the difference between the branch node values is the absolute value of the VoI minus the test cost.

The problem with this model is that it doesn't necessarily give you the value of INFORMATION. Making the 'get info' node a choice point on the tree essentially allows arbitrary changes between the with info and without info branches of the tree. In other words it's not clear we are finding the value of information and not some other result of this choice.

That is why I choose to phrase my model in terms of getting to look at otherwise hidden results of nature nodes.

Comment author: Johnicholas 22 November 2011 04:48:53PM -1 points [-]

I'm a little confused about "value of information" as a precise concept.

Suppose that you have a tree with two kinds of interior nodes in it, and leaves labeled with utilities. One interior node is a choice node, and the other is a nature node, with an associated distribution over its subtrees. It's fairly obvious that you can work backwards up the tree, and find both an optimum strategy and an expected value of the entire tree.

However, I don't see where "value of information" shows up in this framework anywhere. Do I need to distinguish some choice nodes as "information gathering", and apply a default strategy to them as "don't gather any information", and then compute value of information as the difference between best I can do with my eyes closed and the best I can do flat out?

What if there is no natural partition of some actions as information gathering and not-information-gathering?

Is there some homomorphism from the first tree (which is normally a tree of evidence or belief states) to an "externally visible" tree, where two nodes are identified if the only difference is inside my head?

Comment author: TruePath 20 December 2011 08:48:18AM 1 point [-]

Here is a nice formal model.

Let T be a tree of height w and T' a subset of T regarded as the set of choice nodes in T. Now given a function f on T' assume there is a function P giving a probability measure Pf on the set of paths through T' coherent with f for each f. Further assume there is a function u taking paths through T to utilities and define UP(f) to be the integral of u with respect to Pf over all paths coherent with f and U(f|\sigma) to be the integral of u with respect to Pf over all paths extending \sigma coherent with f. Now either assume that each choice node has finitely many successors and U(f|\sigma) is always finite. By compactness we can always achieve a upper bound. We now define the value of information of some set S of non-choice nodes in T (i.e. the benefit of getting to look at those nodes in making future decisions).

Let E(T) (the informed maximum) be the max of U(f) for f a function on T' and let E(T/S) be the max of U(f) over those functions f satisfying f(\sigma)=f(\tau) if {x|\sigma(x) \neq \tau(x) } is a subset of S, i.e. those functions unaware of the result of nodes in S.

The value of getting to look at the results of nodes in S is thus E(T) - E(T/S). Note that to match the concept in the article S should be an initial segment of T (i.e. you get all that info immediately while our model includes the value of buying info that will only be revealed later).

View more: Prev | Next