All of Ilio's Comments + Replies

almost any improvement will indirectly help at designing AI

That may be too strong of a statement. Say some new tool helps improve AI legislation more than AI design, this might turn slowing down the wheel.

Commit to only use superhuman persuasion when arguing towards a valid conclusion via valid arguments, in a manner that doesn't go against the interests of the person being persuaded.

In this plan, how should the AI define what’s in the interest of the person being persuaded? For example, say you have a North Korean soldier who can be persuaded to quite for the west (at the risk of getting the shitty jobs most migrants have) or who can be persuaded to remain loyal to his bosses (at the risk of raising his children in the shitty country most north korean have), what set of rules would you suggest?

the asteroid would likely burn up, but perhaps you have a solution for that

Yes, there’s a well known solution: just make the asteroid fast enough, and it will burn less in the atmosphere.

My understanding of this framework is probably too raw to go sane (A natural latent is a convolution basis useful for analyzing natural inputs, and it’s powerful because function composition is powerful) but it could fit nicely with Agency is what neurons in the biological movement area detect.

That’s great analogy. To me the strength of the OP is to pinpoint that LLMs already exhibit the kind of general ability we would expect from AGI, and the weakness is to forget that LLMs do not exhibit some specific ability most thought easy, such as the agency that even clownfishes exhibit.

In a way this sounds like again the universe is telling us we should rethink what intelligence is. Chess is hard and doing the dishes is easy? Nope. Language is hard and agency is central? Nope.

4jmh
I'm not even sure where I would try to start but do wonder if John Wemtworth's concept of Natural Latents might not offer a useful framework for better grounding the subject for this type of discussion.

Thanks for the prompt! If we ask Claude 3 to be happy about x, don’t you think that counts as nudging it toward implementing a conscious being?

Perhaps you could identify your important beliefs

That part made me think. If I see bright minds falling in this trap, does blindness goes with importance of the belief for that person? I would say yes I think. As if that’s where we tend to make more « mistakes. that can behave as ratchets of the mind ». Thanks for the insight!

that also perhaps are controversial

Same exercise: if I see bright minds falling in this trap, does blindness goes with controversial beliefs? Definitely! Almost by definition actually.

each year write down the

... (read more)
2RussellThor
"most likely story you can think of that would make it be wrong" - that can be the hard part. For investments its sometimes easy - just they fail to execute, their competitors get better, or their disruption is itself disrupted. Before the debate I put Lab leak at say 65-80%, now more like <10%. The most likely story/reason I had for natural origin being correct (before I saw the debate) was that the host was found, and the suspicious circumstances where a result of an incompetent coverup and general noise/official lies  mostly by the CCP around this. Well I can't say for sure that LL was wrong of course, but I changed my mind for a reason I didn't anticipate - i.e. a high quality debate that was sufficiently to my understanding. For some other things its hard to come up with a credible story at all, i.e. AGW being wrong I would really struggle to do.

Yup. Thanks for trying, but these beliefs seem to form a local minima, like a trap for the rational minds -even very bright ones. Do you think you understand how an aspiring rationalist could 1) recover and get out of this trap 2) don’t fall for it in the first place?

To be clear, my problem is not with the possibility of a lab leak itself, it’s with the evaluation that present evidences are anything but posthoc rationalizations fueled by unhealthy levels of tunnel vision. If bright minds can fall for that on this topic specifically, how do I know I’m not making the same mistake on something else?

2RussellThor
Some advice I heard that was for investing was when committing to a purchase, write a story of what you think is most likely to make you lose your money. Perhaps you could identify your important beliefs that also perhaps are controversial and each year write down the most likely story you can think of that would make it be wrong? I also believe that you can only full learn from you  own experience so building up a track record is necessary.

(Spoiler warning)

(Also I didn’t check the previous survey nor the comments there, so expect some level of redondance)

The score itself (8/18) is not that informative, but checking the « accepted » answers is quite interesting. Here’s my « errors » and how much I’m happy making them:

You should be on the outlook for people who are getting bullied, and help defend them against the bullies.

I agree some rationalist leaders are toxic characters who will almost inevitably bully their students and collaborators, and I’m happy to keep strongl... (read more)

4tailcalled
Please remember: The test is a wild extrapolation based on a little bit of data about rationalists and a tons of data about what random nonrationalists believe. If you want to see what rationalists actually believe, you should view the analytics: https://docs.google.com/forms/d/e/1FAIpQLSclPFQb2xUddV0opa8eY7Z1SA-yRCP0jVesUhr-TXt5_c8ehw/viewanalytics

I am an old person. They may not let you do that in chemistry any more.

Absolutely! In my first chemistry lab, a long time ago, our teacher warned us that she had just lost a colleague to cancer at the age of forty, and she swore that if we didn't follow the security protocols very seriously, she would be our fucking nightmare.

I never heard her swear after that.

Not bad! But I stand by « random before (..) » as a better picture in the following sense: neurons don’t connect once to an address ending in 3. It connects several thousands of times to an address ending in 3. Some connexion are on the door, some on windows, some on the roof, one has been seen trying to connect to the dog, etc. Then it’s pruned, and the result looks not that far from a crystal. Or a convnet.

(there’s also long lasting silent synapses and a bit of neurogenesis, but that’s details for another time)

6Nathan Helm-Burger
For those interested in more details, I recommend this video: 
2Nathan Helm-Burger
Yes, that's fair. I think we've now described the situation well enough that I don't think future readers of this thread will end up with a wrong impression. To expand on Ilio's point: the connection point on the "building" (recipient neuron's dendrites) matters a lot because the location of of the synapse on the dendrites sets a floor and ceiling on the strength of the connection which cannot be exceeded by weight modifications due to temporal synchronicity. Also, yes, there is neurogenesis ongoing throughout the lifespan. Never long range (e.g. cross country in our metaphor), only short range (within same metropolitan area). The long range connections are thus special in that they are irreplaceable.

Hmmm, I disagree with the randomness.

I don’t think you do. Let me rephrase: the weights are picked at random, under a distribution biased by molecular cues, then pruned through activity dependent mechanisms.

In other words, our disagreement seems to count as an instance of Bertrand’s paradox.

2Nathan Helm-Burger
Yes, I think I technically agree with you. I just think that describing the cortex's connections as "largely random" gives a misleading sense of the pattern. Randomness actually plays a relatively small and constrained role in brain wiring even before learning occurs. The analogy I've come up with is: Original growth and connection For an analogy of the scale involved, I describe this as a neuron being like a house in America. That house grows a neuron guided by chemotaxis to a particular block of buildings on the other coast. Once there, the neuron forms a synapse according to a chemical compatibility rule. In this analogy, let's say that the neuron in our example must connect to an address ending in 3. Refinement Picking a different member of the closest ten allowed options (respecting the rule of ending in 3 and respecting the neighborhood boundary) according to the Hebbian rule. The Hebbian rule is "Neurons that fire together, wire together." The highest temporally synchronous candidate from among the set will be chosen for a new connection. Existing connections with poor temporal synchronicity will get gradually weaker. Synchronicity changes over time, and thus the set of connections fluctuates. Pruning Of the candidates matched with following refinement, those which are consistently poorly synchronized will be considered 'bad'. The poor quality connections will weaken until below threshold, then be pruned (removed). A neuron with no connections above threshold for a lengthy period of time will be killed (also called pruning). Connections can break and later be reformed, but neurons which are killed are not replaced.

The story went that “Perceptrons proved that the XOR problem is unsolvable by a single perceptron, a result that caused researchers to abandon neural networks”. (…) When I first heard the story, I immediately saw why XOR was unsolvable by one perceptron, then took a few minutes to design a two-layered perceptron network that solved the XOR problem. I then noted that the NAND problem is solvable by a single perceptron, after which I immediately knew that perceptron networks are universal since the NAND gate is.

Exactly the same experience and thoughts in ... (read more)

4Nathan Helm-Burger
[edit: I am referring to the brain here, not Artificial Neural Nets.] Hmmm, I disagree with the randomness. A lot of projections in the brain are ordered before pruning. Indeed, given the degree of order, and the percentage of neurons pruned, it would be impossible to establish that much order with pruning alone. https://www.cell.com/fulltext/S0092-8674(00)80565-6

He can be rough and on rare occasion has said things that could be considered personally disrespectful, but I didn't think that people were that delicate.

You may wish to update on this. I’ve only exchange a few words with one of the name, but that was enough to make clear he doesn’t bother being respectful. That may work in some non delicate research environment I don’t want to know about, but most bright academic I know like to have fun at work, and would leave any non delicate work environment (unless they make their personal duty to clean the place).

What do you think orthogonality thesis is?

I think that’s the deformation of a fundamental theorem (« there exists an universal Turing machine, e.g. it can run any program  ») into a practical belief (« an intelligence can pick its value at random »), with a motte and bailey game on the meaning of can where the motte is the fundamental theorem and the bailey is the orthogonal thesis.

(thanks for the link to your own take, e.g. you think it’s the bailey that is the deformation)

Consider the sense in which humans are not aligned with ea

... (read more)

Existentially dangerous paperclip maximizers don't misunderstand human goals.

Of course they do. If they didn’t and picked their goal at random, they wouldn’t make paperclips in the first place.

There's this post from 2013 whose title became a standard refrain on this point

I wouldn’t say that’s the point I was making.

This has been hashed out more than a decade ago and no longer comes up as a point of discussion on what is reasonable to expect. Except in situations where someone new to the arguments imagines that people on LessWrong expect such unbal

... (read more)
4Vladimir_Nesov
Consider the sense in which humans are not aligned with each other. We can't formulate what "our goals" are. The question of what it even means to secure alignment is fraught with philosophical difficulties. If the oversight AI responsible for such decisions about a slightly stronger AI is not even existentially dangerous, it's likely to do a bad job of solving this problem. And so the slightly stronger AI it oversees might remain misaligned or get more misaligned while also becoming stronger. I'm not claiming sudden changes, only intractability of what we are trying to do and lack of a cosmic force that makes it impossible to eventually arrive at an end result that in caricature resembles a paperclip maximizer, clad in corruption of the oversight process, enabled by lack of understanding of what we are doing. Sure, they expect that we will know what we are doing. Within some model such expectation can be reasonable, but not if we bring in unknown unknowns outside of that model, given the general state of confusion on the topic. AI design is not yet classical mechanics. And also an aligned AI doesn't make the world safe until there is a new equilibrium of power, which is a point they don't address, but is still a major source of existential risk. For example, imagine giving multiple literal humans the power of being superintelligent AIs, with no issues of misalignment between them and their power. This is not a safe world until it settles, at which point humanity might not be there anymore. This is something that should be planned in more detail than what we get by not considering it at all. Sure, this is the way alignment might turn out fine, if it's possible to create an autonomous researcher by gradually making it more capable while maintaining alignment at all times, using existing AIs to keep upcoming AIs aligned. All significant risks are anthropogenic. If humanity can coordinate to avoid building AGI for some time, it should also be feasible to avoid ena

Perhaps the position you disagree with is that a dangerous general AI will misunderstand human goals. That position seems rather silly, and I'm not aware of reasonable arguments for it. It's clearly correct to disagree with it, you are making a valid observation in pointing this out.

Thanks! To be honest I was indeed surprised that was controversial.

But then who are the people that endorse this silly position and would benefit from noticing the error? Who are you disagreeing with, and what do you think they believe, such that you disagree with it?

Wel... (read more)

3Vladimir_Nesov
Existentially dangerous paperclip maximizers don't misunderstand human goals. They just don't pursue human goals, because that doesn't maximize paperclips. There's this post from 2013 whose title became a standard refrain on this point. Essentially nobody believes that an existentially dangerous general AI misinterprets or fails to understand human values or goals AI's designers intend the AI to pursue. This has been hashed out more than a decade ago and no longer comes up as a point of discussion on what is reasonable to expect. Except in situations where someone new to the arguments imagines that people on LessWrong expect such unbalanced AIs that selectively and unfairly understand some things but not others. If it doesn't have a motive to do that, it might do a bad job of doing that. Not because it doesn't have the capability to do a better job, but because it lacks the motive to do a better job, not having alignment and non-deceptiveness as its goals. They are the goals of its developers, not goals of the AI itself. One way AI alignment might go well or turn out to be easy is if humans can straightforwardly succeed in building AIs that do monitor such things competently, that will nudge AIs towards not having any critical alignment problems. It's unclear if this is how things work, but they might. It's still a bad idea to try with existentially dangerous AIs at the current level of understanding, because it also might fail, and then there are no second chances. Consider two AIs, an oversight AI and a new improved AI. If the oversight AI is already existentially dangerous, but we are still only starting work on aligning an AI, then we are already in trouble. If the oversight AI is not existentially dangerous, then it might indeed fail to understand human values or goals, or fail to notice that the new improved AI doesn't care about them and is instead motivated by something else.

All that's required is that we aren't able to coordinate well enough as a species to actually stop it.

Indeed I would be much more optimistic if we were better at dealing with much simpler challenges, like put a price on pollution and welcome refugees with humanity.

Thanks 👍

(noice seems to mean « nice », I assume you meant « noise »)

Assuming "their" refers to the agent and not humans,

It refers to humans, but I agree it doesn’t change the disagreement, i.e. a super AI stupid enough to not see a potential misalignment coming is as problematic as the notion of a super AI incapable of understanding human goals.

3Vladimir_Nesov
Perhaps the position you disagree with is that a dangerous general AI will misunderstand human goals. That position seems rather silly, and I'm not aware of reasonable arguments for it. It's clearly correct to disagree with it, you are making a valid observation in pointing this out. But then who are the people that endorse this silly position and would benefit from noticing the error? Who are you disagreeing with, and what do you think they believe, such that you disagree with it? Not understanding human goals is not the only reason AI might fail to adopt human goals. And it's not the expected reason for a capable AI. A dangerous AI will understand human goals very well, probably better than humans do themselves, in a sense that humans would endorse on reflection, with no misinterpretation. And at the same time is can be motivated by something else that is not human goals. There is no contradiction between these properties of an AI, it can simultaneously be capable enough to be existentially dangerous, understand human values correctly and in detail and in intended sense, and be motivated to do something else. If its designers know what they are doing, they very likely won't build an AI like that. It's not something that happens on purpose. It's something that happens if creating an AI with intended motivations is more difficult than the designers expect, so that they proceed with the project and fail. The AI itself doesn't fail, it pursues its own goals. Not pursuing human goals is not AI's failure in achieving or understanding what it wants, because human goals is not what it wants. Its designers may have intended for human goals to be what it wants, but they failed. And then the AI doesn't fail in pursuing its own goals that are different from human goals. The AI doesn't fail in understanding what human goals are, it just doesn't care to pursue them, because they are not its goals. That is the threat model, not AI failing to understand human goals.

(Epistemic fstatus: first thoughts after first reading)

Most is very standard cognitive neuroscience, although with more emphasis on some things (the subdivision of synaptic buttons into silent/modifiable/stable, notion of complex and simple cells in the visual system) than other (the critical periods, brain rhythms, iso/allo cortices, brain symetry and circuits, etc). There’s one bit or two wrong, but that’s nitpicks or my mistake.

The idea of synapses as detecting frequency code is not exactly novel (it is the usual working hypothesis for some synapses in ... (read more)

Impressively promising work, thanks & good luck! Is there anything a layperson can do to help you reach your goal?

2Lysandre Terrisse
Thank you! Personally, I think that, if a layperson were trying to help me, they could do it by trying to find flaws in the plan. I already mentioned that the One-Time Pad used to fail during WWII in an unexpected way, despite the fact that it had a proof of perfect secrecy. If someone were to find a flaw in the plan, it would help me a lot (although it would also prove that my goal is impossible).
Answer by Ilio5-3

More specifically, if the argument that we should expect a more intelligent AI we build to have a simple global utility function that isn't aligned with our own goals is valid then why won't the very same argument convince a future AI that it can't trust an even more intelligent AI it generates will share it's goals?

For the same reason that one can expect a paperclip maximizer could both be intelligent enough to defeat humans and stupid enough to misinterpret their goal, e.g. you need to believe the ability to select goals is completely separated from the ability to reach them.

(Beware it’s hard and low status to challenge that assumption on LW)

6Vladimir_Nesov
Assuming "their" refers to the agent and not humans, the issue is that a goal that's "misinterpreted" is not really a goal of the agent. It's possibly something intended by its designers to be a goal, but if it's not what ends up motivating the agent, then it's not agent's own goal. And if it's not agent's own goal, why should it care what it says, even if the agent does have the capability to interpret it correctly. That is, describing the problem as misinterpretation is noncentral. The problem is taking something other than (the intended interpretation of) the specified goal as agent's own goal, for any reason. When the agent is motivated by something else, it results in the agent not caring about the specified goal, even if the agent understands it perfectly and in accord with what its designers intended.
4Orual
This is a definitely an assumption that should be challenged more. However, I don't think that FOOM is remotely required for a lot of AI X-risk (or at least unprecedented catastrophic human death toll risk) scenarios. Something doesn't need to recursively self-improve to be a threat if it's given powerful enough ways to act on the world (and all signs point to us being exactly dumb enough to do that). All that's required is that we aren't able to coordinate well enough as a species to actually stop it. Either we don't detect the threat before it's too late or we aren't able to get someone to actually hit the "off" button (literally or figuratively) in time if the threat is detected. And if it only kills 90% of humans because of some error and doesn't tile its light cone in paperclips, that's still really, really bad from a human perspective.

Yes, that’s the crux. In my view, we can reverse…

Inability to distinguish noice and patters is true only for BBs. If we are real humans, we can percieve noice as noice with high probability.

… as « Ability to perceive noise means we’re not BB (high probability). »

Can you tell more about why we can’t use our observation to solve this?

3avturchin
Momentary BB (the ones which exist just one observer-moment) has random thought structure, so it has no causal connection between its observations and thoughts. So even if it percieve noice and think noice, it is just a random coincidence.  However, there is a dust theory. It claims that random BBs can form chains in logical space. In that case, what is noice for one BB, can be "explained" in the next observer moment - for example random perception can be explained as static on my home TV.  There is an article about about it https://arxiv.org/pdf/1712.01826.pdf 

That’s an interesting loophole in my reasoning, thanks! But isn’t that in tension with the observation that we can perceive noise as noise?

(yes humans can find spurious patterns in noise, but they never go as far as mistaking white noise for natural pictures)

-2avturchin
Inability to distinguish noice and patters is true only for BBs. If we are real humans, we can percieve noice as noice with high probability. But we don't know if we are BB or real humans, and can't use our observations about the randomness to solve this.

Yes, although I see that more as an alternative intuition pump rather than a different point.

A true Boltzmann brain may have an illusion of the order in completely random observations.

Sure, like a random screen may happen to look like a natural picture. It’s just exponentially unlikely with picture size, whereas the scenario you suggest is indeed generic in producing brains that look like they evolved from simpler brains.

3avturchin
I meant not that 'random screen may happen to look like a natural picture", but that BB will perceive random screen as if it has order, because BBs are more likely to make logical mistakes.

In other words, you escape the standard argument by adding an observation, e.g. the observation that random fluctuations should almost never make our universe looks obeying physical laws.

One alternative way to see this point is the following: if (2) our brains are random fluctuations, then they are exponentially unlikely to have been created long ago, whereas if (1) it is our observable universe itself that comes from random fluctuations, it could equally have been created 10 billions years or 10 seconds ago. Then counting makes (1) much more likely than (2).

3avturchin
A true Boltzmann brain may have an illusion of the order in completely random observations. So the fact that my observations look ordered is not evidence that they are really ordered for me as BB. In short, we should nor believe BB's thoughts. And thus I can't disprove that I am BB just looking on my observation.  But your argument may still be valid. This is because evolving fluctuations may be more probable than momentary fluctuations. For example, imagine infinite universe filled with low concentration of gas. This gas can form a brain directly for a second, it will be BB. But this gas can also form large but fuzzy blob, which will then gravitationally collapse into a group of stars, some of them will have planets with life and such planets will produce many brains.  While mass of initial gas fluctuation is many orders of magnitude larger than one of the brain, it is less ordered and thus more probable. Thus normal worlds are more probable than BBs.
2Marco Discendenti
Your point is that in the case of the low entropy universe you have much possibilities for the time to consider for its random formation compared to the single brain?

0% that the tool itself will make the situation with the current comment ordering and discourse on platforms such as Twitter, Facebook, YouTube worse.

Thanks for the detailed answer, but I’m more interested in polarization per see than in the value of comment ordering. Indeed we could imagine that your tool feels like it behaves as well as you wanted, but that’s make the memetic world less diverse then more fragile (like monocultures tend to collapse here and then). What’d be your rough range for this larger question?

2Roman Leventov
The system shall indeed create the dynamic of converging on some most reasonable positions (such as that climate change is not a hoax and is man-made, etc.), which you can read as a homogenisation of views, but also naturally keeps itself out of complete balance: when the views are sufficiently homogeneous in a community or the society at large, most of the comments will generally be low-information value to most of the readers, but in such a muted environment, any new promising theory or novel perspective will receive more attention than it would in a highly heterogeneous belief landscape. Which creates the incentive for creating such new theories or perspectives. Thus, the discourse and the belief landscape as a whole should equilibrate themselves at some "not too homogeneous, not too heterogeneous" level.

This is sort of what is customary to expect, but leaning into my optimism bias, I should plan as if this is not the case. (Otherwise, aren’t we all doomed, anyway?)

In your opinion, what are the odds that your tool would make polarization worse? (What’s wrong with keep looking for better plans?)

3Roman Leventov
0% that the tool itself will make the situation with the current comment ordering and discourse on platforms such as Twitter, Facebook, YouTube worse. It will be obvious and consistent across applications whether the tool prioritises thought-provoking, insightful, and reconciling, or bias-confirmatory, groupthink-ey, and combative comments. For example, the tool could rank the comments by the decreasing value of Expectation[user reacts "Insightful"] * Expectation[user reacts "Changed my mind"]. If the model is trained on anything than the dataset where users deliberately coordinated to abuse the "Insightful" reaction to completely reverse its semantics (i.e., they always voted "Insightful" as if it was "Combative", and vice versa), then either the ranking will not be much better than the status quo, or it will be better. Let alone if the model is trained on the LW data, which is high quality (though there are concerns whether an SSM trained on the LW reactions data can generalise beyond LW, as I noted in this comment, the worst case risk here is again uselessness, not harm). Two caveats: * You can imagine there is a "dual-use technology risk" of sorts, namely that if such an SSM proves to be trainable and gives good comment ordering, someone will give it a Waluigi spin: put out a version of the tool, an ultimate "filter bubble that works across all websites" that leverages the same SSM to prioritise the most bias-confirmatory and groupthink-ey comments. Then, the cynic projection is that people will actually flock to using that tool in large numbers, therefore accelerating polarisation. * I think the risk of this is not completely negligible, but it's a small fraction of the risk that people just won't use [BetterDiscourse] because they are mostly interested in confirming their pre-existing beliefs. And again, if the escapism proves to be so rampant, the humanity is doomed through many other AI-enabled paths, such as AI romantic partners. * It's also plausib

Nothing at all. I’m big fan of these kind of ideas and I’d love to present yours to some friends, but I’m afraid they’ll get dismissive if I can’t translate your thoughts into their usual frame of reference. But I get you didn’t work this aspect specifically, there’s many fields in cognitive sciences.

About how much specificity, it’s up to interpretation. A (1k by 1k by frame by cell type by density) tensor representing the cortical columns within the granular cortices is indeed a promising interpretation, although it’d probably be short of an extrapyramidal tensor (and maybe an agranular one).

1Bill Benzon
Well, when Walter Freeman was working on the olfactory cortex of rodents he was using a surface mounted 8x8 matrix of electrodes. I assume that measured in millimeters. In his 1999 paper Consciousness, Intentionality, and Causality (paragraphs 36 - 43) a hemisphere-wide global operator (42):  Later (43):  He goes on from there. I'm not sure whether he came back to that idea before he died in 2016. I haven't found it, didn't do an exhaustive search, but I did look.

You mean this: "We're not talking about some specific location or space in the brain; we're talking about a process."

You mean there’s some key difference in meaning between your original formulation and my reformulation? Care to elaborate and formulate some specific prediction?

As an example, I once gave a try at interpreting data from olfactory system for a friend who were wondering if we could find sign of an chaotic attractor. If you ever toy with Lorenz model, one key feature is: you either see the attractor by plotting x vs vs z, or you can see it b... (read more)

2Bill Benzon
I've lost the thread entirely. Where have I ever said or implied that odors are not location specific or that anything else is not location specific. And how specific are you about location? Are we talking about centimeters (or more), millimeters, individual cortical columns? What's so obscure about the idea that consciousness is a process that can take place pretty much anywhere, though maybe its confined to interaction within the cortex and between subcortical areas, I've not given that one much thought. BTW, I take my conception of consciousness from William Powers, who didn't speculation about its location in the brain.

Is accessing the visual cartesian theater physically different from accessing the visual cortex? Granted, there's a lot of visual cortex, and different regions seem to have different functions. Is the visual cartesian theater some specific region of visual cortex?

In my view: yes, no. To put some flesh on the bone, my working hypothesis is: what’s conscious is gamma activity within an isocortex connected to the claustrum (because that’s the information which will get selected for the next conscious frame/can be considered as in working memory)

I'm not

... (read more)
2Bill Benzon
"You said: what matters is temporal dynamics" You mean this: "We're not talking about some specific location or space in the brain; we're talking about a process." If so, all I meant was a process that can take place pretty much anywhere. Consciousness can pretty much 'float' to wherever its needed. Since you asked for more, why not this: Direct Brain-to-Brain Thought Transfer: A High Tech Fantasy that Won't Work.

I'm willing to speculate that [6 Hz to 10 Hz ]that's your 'one-shot' refresh rate.

It’s possible. I don’t think there was relevant human data in Walter Freeman time, so I’m willing to speculate that’s indeed the frame rate in mouse. But I didn’t check the literature he had access to, so just a wild guess.

the imagery of the stage 'up there' and the seating area 'back here' is not at all helpful

I agree there’s no seating area. I still find the concept of a cartesian theater useful. For exemple, it allows knowing where to plant electrodes if you want to... (read more)

2Bill Benzon
Is accessing the visual cartesian theater physically different from accessing the visual cortex? Granted, there's a lot of visual cortex, and different regions seem to have different functions. Is the visual cartesian theater some specific region of visual cortex? I'm not sure what your question about ordering in sensory areas is about. As for backprop, that gets the distribution done, but that's only part of the problem. In LLMs, for example, it seems that syntactic information is handled in the first few layers of the model. Given the way texts are structured, it makes sense that sentence-level information should be segregated from information about collections of sentences. That's the kind of structure I'm talking about. Sure, backprop is responsible for those layers, but it's responsible for all the other layers as well. Why do we seem to have different kinds of information in different layers at all? That's what interests me. Actually, it just makes sense to me that that is the case. Given that it is, what is located where? As for why things are segregated by location, that does need an answer, doesn't it. Is that what you were asking? Finally, here's an idea I've been playing around with for a long time: Neural Recognizers: Some [old] notes based on a TV tube metaphor [perceptual contact with the world].

A few comments before later. 😉

What I meant was that the connectionist alternative didn't really take off until GPUs were used, making massive parallelism possible.

Thanks for the clarification! I guess you already noticed how research centers in cognitive science seem to have a failure mode over a specific value question: Do we seek excellence at the risk of overfitting funding agency criterion, or do we seek fidelity to our interdisciplinary mission at the risk of compromising growth?

I certainly agree that, before the GPUs, the connectionist approach ... (read more)

2Bill Benzon
In a paper I wrote awhile back I cite the late Walter Freeman as arguing that "consciousness arises as discontinuous whole-hemisphere states succeeding one another at a "frame rate" of 6 Hz to 10 Hz" (p. 2). I'm willing to speculate that that's your 'one-shot' refresh rate. BTW, Freeman didn't believe in a Cartesian theater and neither do it; the imagery of the stage 'up there' and the seating area 'back here' is not at all helpful. We're not talking about some specific location or space in the brain; we're talking about a process. Well, of course, "the distributed way." But what is that? Prompt engineering is about maneuvering your way through the LLM; you're attempting to manipulate the structure inherent in those weights to produce a specific result you want. That 1978 comment of Yevick's that I quote in that blog post I mentioned somewhere up there, was in response to an article by John Haugeland evaluating cognitivism. He wondered whether or not there was an alternative and suggested holography as a possibility. He didn't make a very plausible case and few of the commentators took is as a serious alternative. People were looking for alternatives. But it took awhile for connectionism to build up a record of interesting results, on the one hand, for cognitivism to begin seeming stale on the other hand. It's the combination of the two that brought about significant intellectual change. Or that's my speculation.

When I hear « conventional, sequential, computational regime », my understanding is « the way everyone was trying before parallel computation revolutionized computer vision ». What’s your definition so that using GPU feels sequential?

2Bill Benzon
Oh, I didn't mean to say imply that using GPUs was sequential, not at all. What I meant was that the connectionist alternative didn't really take off until GPUs were used, making massive parallelism possible.  Going back to Yevick, in her 1975 paper she often refers to holographic logic as 'one-shot' logic, meaning that the whole identification process takes place in one operation, the illumination of the hologram (i.e. the holographic memory store) by the reference beam. The whole memory 'surface' is searched in one unitary operation. In an LLM, I'm thinking of the generation of a single token as such a unitary or primitive process. That is to say, I think of the LLM as a "virtual machine" (I first saw the phrase in a blog post by Chris Olah) that is running an associative memory machine. Physically, yes, we've got a massive computation involving every parameter and (I'm assuming) there's a combination of massive parallel and sequential operations taking place in the GPUs. Complete physical parallelism isn't possible (yet). But there are no logical operations taking place in this virtual operation, no transfer of control. It's one operation. Obviously, though, considered as an associative memory device, an LLM is capable of much more than passive storage and retrieval. It performs analytic and synthetic operations over the memory based on the prompt, which is just a probe ('reference beam' in holographic terms) into an associative memory. We've got to understand how the memory is structured so that that is possible. More later.

Thanks, I didn’t know this perspective on the history of our science. The stories I most heard were indeed more about HH model, Hebb rule, Kohonen map, RL, and then connexionism became deep learning..

If the object tends toward geometrical simplicity – she was using identification of visual objects as her domain – then a conventional, sequential, computational regime was most effective.

…but neural networks did refute that idea! I feel like I’m missing something here, especially since you then mention GPU. Was sequential a typo?

1Bill Benzon
How so?

Our daily whims might be a bit inconsistent, but our larger goals aren't.

It’s a key faith I used to share, but I’m now agnostic about that. To take a concrete exemple, everyone knows that blues and reds get more and more polarized. Grey type like old me would thought there must be a objective truth to extract with elements from both sides. Now I’m wondering if ethics should ends with: no truth can help deciding whether future humans should be able to live like bees or like dolphins or like the blues or like the reds, especially when living like the reds... (read more)

Fascinating paper! I wonder how much they would agree that holography means sparse tensors and convolution, or that the intuitive versus reflexive thinking basically amount to visuo-spatial versus phonological loop. Can’t wait to hear which other idea you’d like to import from this line of thought.

4Bill Benzon
Miriam Lipshutz Yevick was born in 1924 and died in 2018, so we can't ask her these questions. She fled Europe with her family inn 1940 for the same reason many Jews fled Europe and ended up in Hoboken, NJ. Seven years later she got a PhD in math from MIT; she was only the 5th woman to get that degree from MIT. But, as both a woman and a Jew, she had almost no chance of an academic post in 1947. She eventually got an academic gig, but it was at a college oriented toward adult education. Still, she managed to do some remarkable mathematical work. The two papers I mention in that blog post were written in the mid-1970s. That was the height of classic symbolic AI and the cognitive science movement more generally. Newell and Simon got their Turing Award in 1975, the year Yevick wrote that remarkable 1975 paper on holographic logic, which deserves to be more widely known. She wrote as a mathematician interested in holography (an interest she developed while corresponding with physicist David Bohm in the 1950s), not as a cognitive scientist. Of course, in arguing for holography as a model for (one kind of) thought, she was working against the tide. Very few were thinking in such terms at that time. Rosenblatt's work was in the past, and had been squashed by Minsky and Pappert, as you've noted. The West Coast connectionist work didn't jump off until the mid-1980s. So there really wasn't anyone in the cognitive science community at the time to investigate the line of thinking she initiated. While she wasn't thinking about real computation, you know, something you actually do on computers, she thought abstractly in computational terms, such as Turing and others did (though Turing also worked with actual computers). It seems to me that her contribution was to examine the relationship between a computational regime and the objects over which he was asked to compute. She's quite explicit about that. If the object tends toward geometrical simplicity – she was using identificat

I have no idea whether or not Hassibis is himself dismissive of that work

Well that’s a problem, don’t you think?

but many are.

Yes, as a cognitive neuroscientist myself, you’re right that many within my generation tend to dismiss symbolic approaches. We were students during a winter that many of us thought caused by the over promising and under delivering of the symbolic approach, with Minsky as the main reason for the slow start of neural networks. I bet you have a different perspective. What’s your three best points for changing the view of my generation?

2Bill Benzon
I'll get back to you tomorrow. I don't think it's a matter of going back to the old ways. ANNs are marvelous; they're here to stay. The issue is one of integrating some symbolic ideas. It's not at all clear how that's to be done. If you wish, take a look at this blog post: Miriam Yevick on why both symbols and networks are necessary for artificial minds.

Because I agree, and because « strangely » sounds to me like « with inconstancies ».

In other words, in my view the orthodox view on orthogonality is problematic, because it suppose that we can pick at will within the enormous space of possible functions, whereas the set of intelligent behavior that we can construct is more likely sparse and by default descriptible using game theory (think tit for tat).

3Seth Herd
I think this would be a problem if what we wanted was logically inconsistent. But it's not. Our daily whims might be a bit inconsistent, but our larger goals aren't. And we can get those goals into AI - LLMs largely understand human ethics even at this point. And what we really want, at least in the near term, is an AGI that does what I mean and checks.

This is a sort of positive nihilism. Because value is not inherent in the physical world, you can assign value to whatever you want, with no inconsistency.

Say we construct a strong AI that attributes a lot of value to a specific white noise screenshot. How would you expect it to behave?

3Seth Herd
Strangely. Why?

Your point is « Good AIs should have a working memory, a concept that comes from psychology ».

DH point is « Good AIs should have a working memory, and the way to implement it was based on concepts taken from neuroscience ».

That’s indeed orthogonal notions, if you will.

2Bill Benzon
I did a little checking. It's complicated. In 2017 Hassibis published an article entitled "Neuroscience-Inspired Artificial Intelligence" in which he attributes the concept of episodic memory to a review article that Endel Tulving published in 2002, "EPISODIC MEMORY: From Mind to Brain." That article has quite a bit to say about the brain. In the 2002 article Tulving dates the concept to an article he published in 1972. That article is entitled "Episodic and Semantic Memory." As far as I know, while there are precedents – everything can be fobbed off on Plato if you've a mind to do it, that's where the notion of episodic memory enters in to modern discussions. Why do I care about this kind of detail? First, I'm a scholar and it's my business to care about these things. Second, a lot of people in contemporary AI and ML are dismissive of symbolic AI from the 1950s through the 1980s and beyond. While Tulving was not an AI researcher, he was very much in the cognitive science movement, which included philosophy, psychology, linguistics, and AI (later on, neuroscientists would join in). I have no idea whether or not Hassibis is himself dismissive of that work, but many are. It's hypocritical to write off the body of work while using some of the ideas. These problems are too deep and difficult to write off whole bodies of research in part because they happened before you were born – FWIW Hassibis was born in 1976.

I’m a bit annoyed that Hassabis is giving neuroscience credit for the idea of episodic memory.

That’s not my understanding. To me he is giving neuroscience credit for the ideas that made possible to implement a working memory in LLM. I guess he didn’t want to use words like thalamocortical, but from a neuroscience point of view transformers indeed look inspired by the isocortex, e.g. by the idea that a general distributed architecture can process any kind of information relevant to a human cognitive architecture.

2Bill Benzon
Yeah, he's talking about neuroscience. I get that. But "episodic memory" is a term of art and the idea behind it didn't come from neuroscience. It's quite possible that he just doesn't know the intellectual history and is taking "episodic memory" as a term that's in general use, which it is. But he's also making claims about intellectual history.  Because he's using that term in that context, I don't know just what claim he's making. Is he also (implicitly) claiming that neuroscience is the source of the idea? If he thinks that, then he's wrong. If he's just saying that he got the idea from neuroscience, OK. But, the idea of a "general distributed architecture" doesn't have anything to do with the idea of episodic memory. They are orthogonal notions, if you will.

I’d be happy if you could point out a non competitive one, or explain why my proposal above does not obey your axioms. But we seem to get diminished returns to sort these questions out, so maybe it’s time to close at this point and wish you luck. Thanks for the discussion!

Saying fuck you is helpful when the aim is to exclude whoever disagree with your values. This is often instrumental to construct a social group, or to get accepted in a social group that includes high status toxic characters. I take be nice as the claim that there are always better objectives.

This is aiming at a different problem than goal agnosticism; it's trying to come up with an agent that is reasonably safe in other ways.

Well, assuming a robust implementation, I still think it obeys your criterions, but now you mention « restrictive », my understanding is that you want this expression to specifically refers to pure predictors. Correct?

If yes, I’m not sure that’s the best choice for clarity (why not « pure predictors »?) but of course that’s your choice. If not, can you give some examples of goal agnostic agents other than pure predictors?

3porby
Goal agnosticism can, in principle, apply to things which are not pure predictors, and there are things which could reasonably be called predictors which are not goal agnostic. A subset of predictors are indeed the most powerful known goal agnostic systems. I can't currently point you toward another competitive goal agnostic system (rocks are uselessly goal agnostic), but the properties of goal agnosticism do, in concept, extend beyond predictors, so I leave the door open. Also, by using the term "goal agnosticism" I try to highlight the value that arises directly from the goal-related properties, like statistical passivity and the lack of instrumental representational obfuscation. I could just try to use the more limited and implementation specific "ideal predictors" I've used before, but in order to properly specify what I mean by an "ideal" predictor, I'd need to specify goal agnosticism.

You forgot to explain why these arguments only apply to strangers. Is there a reason to think medical research and economical incentives are better when it’s a family member who need a kidney?

Nope, my social media presence is very very low. But I’m open to suggestion since I realized there’s a lot of toxic characters with high status here. Did you try EA forums? Is it better?

2Lyrongolem
Hm... pretty similar here. I also don't have much of a media presence. I haven't tried EA forums yet, mainly because I consider myself intellectually more aligned with LW, but in any case I'm open to looking. This is looking to be a more personal conversation now. Would you like to continue in direct messages? Open to hearing your suggestions, I'm just as clueless right now. 

(The actual question is about your best utilitarian model, not your strategy given my model.)

Uniform distribution of donating kidney sounds also the result when a donor is 10^19 more likely to set the example. Maybe I should precise that the donor is unlikely to take the 1% risk unless someone else is more critical to war effort.

Load More