All of TheManxLoiner's Comments + Replies

In Sakana AI's paper on AI Scientist v-2, they claim that the sytem is independent of human code.  Based on quick skim, I think this is wrong/deceptful. I wrote up my thoughts here: https://lovkush.substack.com/p/are-sakana-lying-about-the-independence

Main trigger was this line in the system prompt for idea generation: "Ensure that the proposal can be done starting from the provided codebase."

Substacks:
- https://aievaluation.substack.com/
- https://peterwildeford.substack.com/
- https://www.exponentialview.co/
- https://milesbrundage.substack.com/

Podcasts:
- Cognitive Revolution. https://www.cognitiverevolution.ai/tag/episodes/
- Doom debates. https://www.youtube.com/@DoomDebates
- AI policy podcast https://www.csis.org/podcasts/ai-policy-podcast

Worth checking this too: https://forum.effectivealtruism.org/posts/5Hk96JqpEaEAyCEud/how-do-you-follow-ai-safety-news 

Vague thoughts/intuitions:

• Using the word "importance" I think is misleading. Or, makes it harder to reason about the connection between this toy scenario and real text data. In real comedy/drama, there is patterns in the data to let me/the model deduce it is comedy or drama and hence allow me to focus on the conditionally important features.
• Phrasing the task as follows helps me: You will be given 20 random numbers x1 to x20. I want you to find projections that can recover x1 to x20. Half the time I will ignore your answers from x1 to x10 and the other half

there are features such as X_1 which are perfectly recovered

Just to check, in the toy scenario, we assume the features in R^n are the coordinates in the default basis. So we have n features X_1, ..., X_n

Separately, do you have intuition for why they allow network to learn b too? Why not set b to zero too?

If you’d like to increase the probability of me writing up a “Concrete open problems in computational sparsity” LessWrong post

I'd like this!

I think this is missing from the list. https://wba-initiative.org/en/25057/. Whole brain architectue initiative.

Should LessWrong have an anonymous mode? When reading a post or comments, is it useful to have the username or does that introduce bias?

I had this thought after reading this review of LessWrong: https://nathanpmyoung.substack.com/p/lesswrong-expectations-vs-reality

Sounds sensible to me!

What do we mean by $U-V$?

I think the setting is:

• We have a true value function $V$
• We have a process to learn an estimate of $V$. We run this process once and we get $U$
• We then ask an AI system to act so as to maximize $U$ (its estimate of human values)

So in this context, $U-V$ is just a fixed function measuring the error between the learnt values and true values.

I think confusion could be using the term $U$ to represent both a single instance or the random variable/process.

But in this Eiffel Tower example, I’m not sure what is correlating with what

The physical object Eiffel Tower is correlated with itself.
 

However, I think the basic ability of an LLM to correctly complete the sentence “the Eiffel Tower is in the city of…” is not very strong evidence of having the relevant kinds of dispositions.

It is highly predictive of the ability of the LLM to book flights to Paris, when I create an LLM-agent out of it and ask it to book a trip to see the Eiffel Tower.
 

I think the question about whether current AI systems have re

Zvi's latest newsletter has a section on this topic! https://thezvi.substack.com/i/151331494/good-advice

+1 to you continuing with this series.

1. Pedantic point. You say "Automating AI safety means developing some algorithm which takes in data and outputs safe, highly-capable AI systems." I do not think semi-automated interpretability fits into this, as the output of interpretability (currently) is not a model but an explanation of existing models.
2. Unclear why Level (1) does not break down into the 'empirical' vs 'human checking'.  In particular, how would this belief obtained: "The humans are confident the details provided by the AI systems don’t compromise the safety of the algorithm."
3. Unclear

Couple of thoughts:
1. I recently found out about this new-ish social media platform. https://www.heymaven.com/.  Good chance they are researching alternative recommendation algorithms.
2. What particular actions do you think rationality/ea community could do that other big efforts have not done enough, e.g. projects by Tristan Harris or Jaron Lanier.

Thanks for the feedback! Have editted the post to include your remarks.

The 'evolutionary pressures' being discussed by CGP Grey is not the direct gradient descent used to train an individual model. Instead, he is referring to the whole set of incentives we as a society put on AI models. Similar to memes - there is no gradient descent on memes.

(Apologies if you already understood this, but it seems your post and Steven Byrne's post focus on training of individual models)

What is the status of this project? Are there any estimates of timelines?

Totally agree! This is my big weakness right now - hopefully as I read more papers I'll start developing a taste and ability to critique.

Huge thanks for writing this! Particularly liked the SVD intuition and how it can be used to understand properties of $M{M}^T$.  One small correction I think. You wrote:

 $x-E\left[x\right]$ is simply $p_1\left(x\right)$ the projection along the vector $(1\dots 1)$

I think $E\left[x\right]$ is projection along the vector $(1,\dots ,1)$, so $x-E\left[x\right]$ is projection on hyperplane perpendicular to $(1,\dots ,1)$

Interesting ideas, and nicely explained! Some questions:

1) First notation: request patching means replacing the vector at activation A for R2 on C2 with vector at same activation A for R1 on C1.  Then the question: Did you do any analysis on the set of vectors A as you vary R and C?  Based on your results, I expect that the vector at A is similar if you keep R the same and vary C.

2) I found the success on the toy prompt injection surprising! My intuition up to that point was that R and C are independently represented to a large extent, and you co... (read more)

No need to apologise! I missed your response by even more time...

My instinct is that it is because of the relative size of the numbers, not the absolute size.

It might be an interesting experiment to see how the intuition varies based on the ratio of the total amount to the difference in amounts: "You have two items whose total cost is £1100 and the difference in price is £X. What is the price of the more expensive item?", where X can be 10p or £1 or £10 or £100 or £500 or £1000.

With X=10p, one possible instinct is 'that means they are basically the same price, so the more expensive item is £550 + 10p = £550.10.

 

I have the same experience as you, drossbucket: my rapid answer to (1) was the common incorrect answer, but for (2) and (3) my intuition is well-honed.

A possible reason for this is that the intuitive but incorrect answer in (1) is a decent approximation to the correct answer, whereas the common incorrect answers in (2) and (3) are wildly off the correct answer. For (1) I have to explicitly do a calculation to verify the incorrectness of the rapid answer, whereas in (2) and (3) my understanding of the situation immediately rules out the incorrect answers.

He... (read more)