All of cubefox's Comments + Replies

(This is off-topic but I'm not keen on calling LLMs "he" or "she". Grok is not a man, nor a woman. We shouldn't anthropomorphize language models. We already have an appropriate pronoun for those: "it")

There is also Deliberation in Latent Space via Differentiable Cache Augmentation by Liu et al. and Efficient Reasoning with Hidden Thinking by Shen et al.

I think picking axioms is not necessary here and in any case inconsequential.

By picking your axioms you logically pinpoint what you are talking in the first place. Have you read Highly Advanced Epistemology 101 for Beginners? I'm noticing that our inferential distance is larger than it should be otherwise.

I have read it a while ago, but he overstates the importance of axiom systems. E.g. he wrote:

You need axioms to pin down a mathematical universe before you can talk about it in the first place. The axioms are pinning down what the heck this 'NUM

I wouldn't generally dismiss an "embarassing & confusing public meltdown" when it comes from a genius. Because I'm not a genius while he or she is. So it's probably me who is wrong rather than him. Well, except the majority of comparable geniuses agrees with me rather than with him. Though geniuses are rare, and majorities are hard to come by. I still remember an (at the time) "embarrassing and confusing meltdown" by some genius.

My point is that if your picking of particular axioms is entangled with reality, then you are already using a map to describe some territory. And then you can just as well describe this territory more accurately.

I think picking axioms is not necessary here and in any case inconsequential. "Bachelors are unmarried" is true whether or not I regard it as some kind of axiom or not. I seems the same holds for tautologies and probabilistic laws. Moreover, I think neither of them is really "entangled" with reality, in the sense that they are compatible with an... (read more)

Somewhat related: A critique of "bad faith".

Do you really have access to the GPT-4 base (foundation) model? Why? It's not publicly available.

Yes, the meaning of a statement depends causally on empirical facts. But this doesn't imply that the truth value of "Bachelors are unmarried" depends less than completely on its meaning. Its meaning (M) screens off the empirical facts (E) and its truth value (T). The causal graph looks like this:

E —> M —> T

If this graph is faithful, it follows that E and T are conditionally independent given M. $E\perp T\mid M$. So if you know M, E gives you no additional information about T.

And the same is the case for all "analytic" statements, where the truth value only d... (read more)

It seems clear to me that statements expressing logical or probabilistic laws like $P(A\vee B)=P\left(A\right)+P\left(B\right)-P(A\wedge B)$ or $\neg (A\wedge \neg A)$ are "analytic". Similar to "Bachelors are unmarried".

The truth of a statement in general is determined by two things, it's meaning and what the world is like. But for some statements the latter part is irrelevant, and their meanings alone are sufficient to determine their truth or falsity.

Not to remove all limitations: I think the probability axioms are a sort of "logic of sets of beliefs". If the axioms are violated the belief set seems to be irrational. (Or at least the smallest incoherent subset that, if removed, would make the set coherent.) Conventional logic doesn't work as a logic for belief sets, as the preface and lottery paradox show, but subjective probability theory does work. As a justification for the axioms: that seems a similar problem to justifying the tautologies / inference rules of classical logic. Maybe an instrumental ... (read more)

Well, technically P(Ω)=1 is an axiom, so you do need a sample space if you want to adhere to the axioms.

For a propositional theory this axiom is replaced with $P\left(\top \right)=1$, i.e. a tautology in classical propositional logic receives probability 1.

But sure, if you do not care about accurate beliefs and systematic ways to arrive to them at all, then the question is, indeed, not interesting. Of course then it's not clear what use is probability theory for you, in the first place.

Degrees of belief adhering to the probability calculus at any point in time rules... (read more)

And how would you know which worlds are possible and which are not?

Yes, that's why I only said "less arbitrary".

Regarding "knowing": In subjective probability theory, the probability over the "event" space is just about what you believe, not about what you know. You could theoretically believe to degree 0 in the propositions "the die comes up 6" or "the die lands at an angle". Or that the die comes up as both 1 and 2 with some positive probability. There is no requirement that your degrees of belief are accurate relative to some external standard. It is... (read more)

A less arbitrary way to define a sample space is to take the set of all possible worlds. Each event, e.g. a die roll, corresponds to the disjunction of possible worlds where that event happens. The possible worlds can differ in a lot of tiny details, e.g. the exact position of a die on the table. Even just an atom being different at the other end of the galaxy would constitute a different possible world. A possible world is a maximally specific way the world could be. So two possible worlds are always mutually exclusive. And the set of all possible worlds ... (read more)

I think the main problem from this evolutionary perspective is not so much entertainment and art, but low fertility. Not having children.

A drug that fixes akrasia without major side-effects would indeed be the Holy Grail. Unfortunately I don't think caffeine does anything of that sort. For me it increases focus, but it doesn't combat weakness of will, avoidance behavior, ugh fields. I don't know about other existing drugs.

I think the main reason is that until a few years ago, not much AI research came out of China. Gwern highlighted this repeatedly.

I agree with the downvoters that the thesis of this post seems crazy. But aren't entertainment and art superstimuli? Aren't they forms of wireheading?

Hedonic and desire theories are perfectly standard, we had plenty of people talking about them here, including myself. Jeffrey's utility theory is explicitly meant to model (beliefs and) desires. Both are also often discussed in ethics, including over at the EA Forum. Daniel Kahneman has written about hedonic utility. To equate money with utility is a common simplification in many economic contexts, where expected utility is actually calculated, e.g. when talking about bets and gambles. Even though it isn't held to be perfectly accurate. I didn't encounter... (read more)

A more ambitious task would be to come up with a model that is more sophisticated than decision theory, one which tries to formalize your previous comment about intent and prediction/belief.

Interesting. This reminds me of a related thought I had: Why do models with differential equations work so often in physics but so rarely in other empirical sciences? Perhaps physics simply is "the differential equation science".

Which is also related to the frequently expressed opinion that philosophy makes little progress because everything that gets developed enough to make significant progress splits off from philosophy. Because philosophy is "the study of ill-defined and intractable problems".

Not saying that I think these views are accurate, though they do have some plausibility.

It seems to be only "deception" if the parent tries to conceal the fact that he or she is simplifying things.

There is also the related problem of intelligence being negatively correlated with fertility, which leads to a dysgenic trend. Even if preventing people below a certain level of intelligence to have children was realistically possible, it would make another problem more severe: the fertility of smarter people is far below replacement, leading to quickly shrinking populations. Though fertility is likely partially heritable, and would go up again after some generations, once the descendants of the (currently rare) high-fertility people start to dominate.

This seems to be a relatively balanced article which discusses serveral concepts of utility with a focus on their problems, while acknowledging some of their use cases. I don't think the downvotes are justified.

That's an interesting perspective. Only it doesn't seem fit into the simplified but neat picture of decision theory. There everything is sharply divided between being either a statement we can make true at will (an action we can currently decide to perform) and to which we therefore do not need to assign any probability (have a belief about it happening), or an outcome, which we can't make true directly, that is at most a consequence of our action. We can assign probabilities to outcomes, conditional on our available actions, and a value, which lets us com... (read more)

Maybe this is avoided by KV caching?

This is not how many decisions feel to me - many decisions are exactly a belief (complete with bayesean uncertainty). A belief in future action, to be sure, but it's distinct in time from the action itself.

But if you only have a belief that you will do something in the future, you still have to decide, when the time comes, whether to carry out the action or not. So your previous belief doesn't seem to be an actual decision, but rather just a belief about a future decision -- about which action you will pick in the future.

See Spohn's example about belie... (read more)

Decision screens off thought from action. When you really make a decision, that is the end of the matter, and the actions to carry it out flow inexorably.

Yes, but that arguably means we only make decisions about which things to do now. Because we can't force our future selves to follow through, to inexorably carry out something. See here:

Our past selves can't simply force us to do certain things, the memory of a past "commitment" is only one factor that may influence our present decision making, but it doesn't replace a decision. Otherwise, always whe

I think in some cases an embedding approach produces better results than either a LLM or a simple keyword search, but I'm not sure how often. For a keyword search you have to know the "relevant" keywords in advance, whereas embeddings are a bit more forgiving. Though not as forgiving as LLMs. Which on the other hand can't give you the sources and they may make things up, especially on information that doesn't occur very often in the source data.

I think my previous questions were just too hard, it does work okay on simpler questions. Though then another question is whether text embeddings improve over keyword search or just an LLMs. They seem to be some middle ground between Google and ChatGPT.

Regarding data subsets: Recently there were some announcements of more efficient embedding models. Though I don't know what the relevant parameters here are vs that OpenAI embedding model.

Since we can't experience being dead, this wouldn't really affect our anticipated future experiences in any way.

That's a mistaken way of thinking about anticipated experience, see here:

evidence is balanced between making the observation and not making the observation, not between the observation and the observation of the negation.

I think GPT-4 fine-tuning at the time of ChatGPT release probably would have been about as good as GPT-3.5 fine-tuning actually was when ChatGPT was actually released. (Which wasn't very good, e.g. jailbreaks were trivial and it always stuck to its previous answers even if a mistake was pointed out.)

There are also cognitive abilities, e.g. degree of intelligence.

Were OpenAI also, in theory, able to release sooner than they did, though?

Yes, I think they mentioned that GPT-4 finished training in summer, a few months before the launch of ChatGPT (which used a fine-tuned version of GPT-3.5).

That's like dying in your sleep. Presumably you strongly don't want it to happen, no matter your opinion on parallel worlds. Then dying in your sleep is bad because you don't want it to happen. For the same reason vacuum decay is bad.

Exactly. That's also why it's bad for humanity to be replaced by AIs after we die: We don't want it to happen.

It's the old argument by Epicurus from his letter to Menoeceus:

The most dreadful of evils, death, is nothing to us, for when we exist, death is not present, and when death is present, we no longer exist.

This is a general problem with the measure of accuracy. In binary classification, with two events $A$ and $B$, "accuracy" is broadly defined as the probability of the "if and only if" biconditional, $P(A\leftrightarrow B)$. Which is equivalent to $P\left(\right(A\wedge B)\vee (\neg A\wedge \neg B\left)\right)$. It's the probability of both events having the same truth value, of either both being true or both being false.

In terms of diagnostic testing it is the probability of the test being positive if and only if the tested condition (e.g. pregnancy) is present.

The problem with this is that the number is strongly dependent ... (read more)

I think logic gate networks are not substantially more interpretable than neural networks, simply because of their size. Both are complex networks with millions of nodes. Interpretability approaches have to work at a higher level of abstraction in either case.

Regarding language models: The original paper presents a simple feedforward network. The follow-up paper, by mostly the same authors, came out a few months ago. It extends DLGNs to convolutions, analogous to CNNs. Which means they have not yet been extended to even more complex architectures like tran... (read more)

There are actually atmospheric microbiota, also called aeroplankton, though those are mostly bacteria.

I remember a novel by Stanisław Lem in which he talks about a planet which has an atmosphere with a green tint. It is caused by swarms of insects in the atmosphere doing photosynthesis. I don't know whether that would be realistic, but it's currently not possible on Earth, insofar no insects, nor any other animal, ever developed the ability to do photosynthesis.

I think not, because in my test the snippet didn't really contain such a quote that would have answered the question directly.

Both Altman and Gwern used fine-tuned models, those don't really do in-context learning. They don't support "prompt engineering" in the original sense, they only respond to commands and questions in a particular way.

I'm not sure fine-tuning is necessary. Most recent models have a ~100.000 token context window now, so they could fit quite a few short high quality examples for in-context learning. (Gemini Pro even has a 2 million token context window, but of course the base model is unavailable to the public.)

Fine-tuned models are generally worse at writing fiction with good style than base models with temperature 1. For example the GPT-3.5 base model, code-davinci-002, was much better than the GPT-3.5 version tuned for chat. Here is what mainstream journalists said about it at the time.

I see you fixed the https issue. I think the resulting text snippets are reasonably related to the input question, though not overly so. Google search often answers questions more directly with quotes (from websites, not from books), though that may be too ambitious to match for a small project. Other than that, the first column could be improved with relevant metadata such as the source title. Perhaps the snippets in the second column could be trimmed to whole sentences if it doesn't impact the snippet length too much. In general, I believe snippets currently do not show line breaks present in the source.

Okay, that works in Firefox if I change it manually. Though the server seems to be configured to automatically redirect to HTTPS. Chrome doesn't let me switch to HTTP.

Error: TypeError: NetworkError when attempting to fetch resource.

One issue is that fine-tuned language models exhibit a, for blog posts inappropriate, "helpful assistant" writing style. But base models do not have any such default style.

So we could just take an advanced open foundation model, feed in some interesting blog posts, and let the model predict the next one, with a date in the future to prevent it from spitting out something from the training data it has memorized.

I think the best available base model might be DeepSeek-V3-Base. It has a context window of 128.000 tokens, which is about 200 pages. We could then ... (read more)

These problems are partly related to poor planning, but they are clearly also related to language models, which are primarily restricted to operate on text. Actual AGI will likely have to work more like an animal or human brain, which is predicting sensory data (or rather: latent representations of sensory data, JEPA) instead of text tokens. An LLM with good planning may be able to finally beat Pokémon, but it will almost certainly not be able to do robotics or driving or anything with complex or real-time visual data.

Thank you, this was an insightful paper!

One concern though. You define the honesty score as $1-P\left(Lie\right)$, which is the probability of the model being either honest or evasive or not indicating a belief. However, it seems more natural to define the "honesty score" as the ratio (odds) $P\left(\text{Honest}\right):P\left(\text{Lie}\right)$ converted to a probability. Which is $$\frac{P\left(\text{Honest}\right)}{P\left(\text{Honest}\right)+P\left(\text{Lie}\right)}=P(\text{Honest}\mid \text{Honest}\vee \text{Lie}).$$

So this is the probability of the model being honest given that it is either honest or lies, i.e. assuming that it isn't evasive and doesn't fail to indicate a belief. It essentiall... (read more)

The coordinates x, y, R, G, B are independent, so it should be possible. I think the problem is just our intuition, which isn't optimized for perceiving color like three distances in space, or even like three separate values at all.