All of rotatingpaguro's Comments + Replies

Isn't it normal in startup world to make bets and not make money for many years? I am not familiar with the field so I don't have intuitions for how much money/how many years would make sense, so I don't know if OpenAI is doing something normal, or something wild.

During our evaluations we noticed that Claude 3.7 Sonnet occasionally resorts to special-casing in order to pass test cases in agentic coding environments like Claude Code. Most often this takes the form of directly returning expected test values rather than implementing general solutions, but also includes modifying the problematic tests themselves to match the code’s output.

These behaviors typically emerge after multiple failed attempts to develop a general solution, particularly when:

• The model struggles to devise a comprehensive solution

• Test cases p

Economy can be positive-sum, i.e., the more people work, the more everyone gets. Do you think the UK in particular is in a situation where instead if you work more, you are just lowering wages without getting more done?

My current thinking is that

1. relying on the CoT staying legible because it's English, and
2. hoping the (racing) labs do not drop human language when it becomes economically convenient to do so,

were hopes to be destroyed as quickly as possible. (This is not a confident opinion, it originates from 15 minutes of vague thoughts.)

To be clear, I don't think that in general it is right to say "Doing the right thing is hopeless because no one else is doing it", I typically prefer to rather "do the thing that if everyone did that, the world would be better". My intuitio... (read more)

I wonder whether stuff like "turn off the wifi" is about costly signals? (My first-order opinion is still that it's dumb.)

I started reading, but I can't understand what the parity problem is, in the section that ought to define it.

I guess, the parity problem is finding the set S given black-box access to the function, is it?

I think I prefer Claude's attitude as assistant. The other two look too greedy to be wise.

Referring to the section "What is Intelligence Even, Anyway?":

I think AIXI is fairly described as a search over the space of Turing machines. Why do you think otherwise? Or maybe are you making a distinction at a more granular level?

When you say "true probability", what do you mean?

The current hypotheses I have about what you mean are (in part non-exclusive):

1. You think some notion of objective, non-observer dependent probability makes sense, and that's the true probability.
2. You do not think "true probability" exists, you are referencing to it to say the market price is not anything like that.
3. You define "true probability" a probability that observers contextually agree on (like a coin flip observed by humans who don't know the thrower).

Anton Leicht says evals are in trouble as something one could use in a regulation or law. Why? He lists four factors. Marius Hobbhahn of Apollo also has thoughts. I’m going to post a lot of disagreement and pushback, but I thank Anton for the exercise, which I believe is highly useful.

I think there's one important factor missing: if you really used evals for regulation, then they would be gamed. I trust more the eval when the company is not actually at stake on it. If it was, there would be a natural tendence for evals to slide towards empty box-checking.

I sometimes wonder about this. This post does pose the question, but I don't think it gives an analysis that could make me change my mind on anything, it's too shallow and not adversarial.

I read part of the paper. That there's a cultural difference north-south about honesty and willingness to break the rules matches my experience on the ground.

I find this intellectually stimulating, but it does not look useful in practice, because with repeated i.i.d. data the information in the data is much higher than the prior if the prior is diffuse/universal/ignorance.

Italians over time sorted themselves geographically by honesty, which is both weird and damn cool, and also makes a lot of sense. There are multiple equilibria, so let everyone find the one that suits them. We need to use this more in logic puzzles. In one Italian villa everyone tells the truth, in the other…

I can't get access to the paper, anyone has a tip on this?

I agree with whay you say about how to maximize what you get out of an interview. I also agree about that discussion vs. debate distinction you make, and I wasn't specifically trying to go there when I used the word "debate", I was just sloppy with words.

I guess you agree that it is friction to create a social norm that you should do a read up of the other person material before engaging in public. I expect less discussions would happen. There is not a clear threshold at how much you should be prepared.

I guess we disagree about how much value do we lose du... (read more)

I see your proposed condition for meaningful debate as bureaucracy that adds friction rather than value.

I somewhat disagree with Tenobrus' commentary about Wolfram.

I watched the full podcast, and my impression was that Wolfram uses a "scientific hat", of which he is well aware of, which comes with a certain ritual and method for looking at new things and learning them. Wolfram is doing the ritual of understanding what Yudkowsky says, which involves picking at the details of everything.

Wolfram often recognizes that maybe he feels like agreeing with something, but "scientifically" he has a duty to pick it apart. I think this has to be understood as a learning process rather than as a state of belief.

So, should the restrictions on gambling be based on feedback loop length? Should sport betting be broadly legal when about the far enough future?

current inference scaling methods tend to be tied to CoT and the like, which are quite transparent

Aschenbrenner in Situational Awareness predicts illegible chains of thought are going to prevail because they are more efficient. I know of one developer claiming to do this (https://platonicresearch.com/) but I guess there must be many.

Related, I have a vague understanding on how product safety certification works in EU, and there are multiple private companies doing the certification in every state.

Half-informed take on "the SNPs explain a small part of the genetic variance": maybe the regression methods are bad?

Not sure if I missed something because I read quickly, but: all these are purely correlational studies, without causal inference, right?

OpenAI is recklessly scaling AI. Besides accelerating "progress" toward mass extinction, it causes increasing harms. Many communities are now speaking up. In my circles only, I count seven new books critiquing AI corps. It’s what happens when you scrape everyone's personal data to train inscrutable models (computed by polluting data centers) used to cheaply automate out professionals and spread disinformation and deepfakes.

Could you justify that it causes increasing harms? My intuition is that OpenAI is currently net-positive without taking into account fu... (read more)

Ok, that. China seems less interventionist, and to use more soft power. The US is more willing to go to war. But is that because the US is more powerful than China, or because Chinese culture is intrinsically more peaceful? If China made the killer robots first, would they say "MUA-HA-HA actually we always wanted to shoot people for no good reason like in yankee movies! Go and kill!"

Since politics is a default-no on lesswrong, I'll try to muddle the waters by making a distracting unserious figurative narration.

Americans maybe have more of a culture of "if ... (read more)

[Alert: political content]

About the US vs. China argument: have any proponent made a case that the Americans are the good guys here?

My vague perspective as someone not in China neither in the US, is that the US is overall more violent and reckless than China. My personal cultural preference is for US, but when I think about the future of humanity, I try to set aside what I like for myself.

So far the US is screaming "US or China!" while creating the problem in the first place all along. It could be true that if China developed AGI it would be worse, but tha... (read more)

I agree it's not a flaw in the grand scheme of things. It's a flaw for using it for consensus for reasoning.

I start with a very low prior of AGI doom (for the purpose of this discussion, assume I defer to consensus).

You link to a prediction market (Manifold's "Will AI wipe out humanity before the year 2100", curretly at 13%).

Problems I see with using it for this question, in random order:

1. It ends in 2100 so the incentive is effectively about what people will believe a few years from now, not about the question. It is a Keynesian beauty contest. (Better than nothing.)
2. Even with the stated question, you win only if it resolves NO, so it is strategically correct to b

This type of issue is a huge effective blocker for people with my level of skills. I find myself excited to write actual code that does the things, but the thought of having to set everything up to get to that point fills with dread – I just know that the AI is going to get something stupid wrong, and everything’s going to be screwed up, and it’s going to be hours trying to figure it out and so on, and maybe I’ll just work on something else. Sigh. At some point I need to power through.

Reminds me of this 2009 kalzumeus quote:

I want to quote a real customer

Ah, sorry for being so cursory.

A common trope about mathematicians vs. other math users is that mathematicians are paranoid persnickety truth-seekers, they want everything to be exactly correct down to every detail. Thus engineers and physicists often perceive mathematicians as a sort of fact-checker caste.

As you say, in some sense mathematicians deal with made-up stuff and engineers with real stuff. But from the engineer's point of view, they deal with mathematicians when writing math, not when screwing bolts, and so perceive mathematicians as "the annoyi... (read more)

The analogy with mathematicians is very stretched.

• Include, in the cue to each note, a hint as to its content, besides just the ordinal pointer. A one-letter abbreviation, standardised thruout the work, may work well, e.g.:
  • "c" for citation supporting the marked claim
  • "d" for a definition of the marked term
  • "f" for further, niche information extending the marked section
  • "t" for a pedantic detail or technicality modifying the marked clause
• Commit to only use notes for one purpose — say, only definitions, or only citations. State this commitment to the reader.

These don't look like good solutions to me. Just a first impression.

I don't make eye contact while speaking but fix people while in silence. Were there people like me? Did they managed to reverse this? The way I feel inside is more like I can't think both about the face of someone and what I am saying at once, too many things to keep track of.

Is champerty legal in California?

I take this as a fun occasion to lose some of my karma in a silly way to remind myself lesswrong karma is not important.

A very interesting problem is measuring something like general intelligence. I’m not going to delve deeply into this topic but simply want to draw attention to an idea that is often implied, though rarely expressed, in the framing of such a problem: the assumption that an "intelligence level," whatever it may be, corresponds to some inherent properties of a person and can be measured through their manifestations. Moreover, we often talk about measurements with a precision of a few percentage points, which suggests that, in theory, the measurement should be

Still 20$/usd

Does it imply that normally everyone would receive as many spam calls, but the more expensive companies are spending a lot of their budget to actively fight against the spammers?

Yeah, they said this is what happens.

"things that mysteriously don't work in USA despite working more or less okay in most developed countries"

Let my try:

• expensive telephone (despite the infamous bell breakup)
• super-expensive health
• super-expensive college
• no universally free or dirty cheap wire transfers between any bank
• difficult to find a girlfriend
• no gun control (disputed)
• crap trai

I don't really know about this specific proposal to deter spam calls, but speaking in general: I'm from another large first world country, and when staying in the US a striking difference was receiving on average 4 spam calls per day. My american friends told me it was because my phone company was low-cost, but it was O(10) more expensive (per unit data) than what I had back home, with about O(1) spam calls per year.

So I expect that it is totally possible to solve this problem without doing something too fancy, even if I don't know how it's solved where I am from.

I'm reminded of the recent review of How Language Began on ACX: the missionary linguist becomes an atheist because in the local very weird language they have declinations to indicate the source of what you are saying, and saying things about Jesus just doesn't click.

I still don't understand your "infinite limit" idea. If in your post I drop the following paragraph:

A way to think about the proposition $A_p$ is as a kind of limit. When we have little evidence, each bit of evidence has a potentially big impact on our overall probability of a given proposition. But each incremental bit of evidence shifts our beliefs less and less. The proposition $A_p$ can be thought of a shorthand for an infinite collection of evidences $F_i$ where the collection leads to an overall probability of $p$ given t

I happened to have the same doubt as you. A deeper analysis of the sacred texts shows how your interpretation of the Golden Rule is amiss. You say:

“Do unto others as you would have them do unto you.” (Matthew 7:12)

But the correct version is:

Therefore whatever you desire for men to do to you, you shall also do to them; for this is the law and the prophets.

The verse speaks specifically of men, not generically of others. So if you are straight, it does not compel you to sexual acts on women, while if you are gay, you shall try to hit on all the men to your he... (read more)

There are too many nonpolar bears in the US to keep up the lie.

I guess the point of the official party line is to avoid kids going and trying to scare polar bears.

I don't think this concept is useful.

What you are showing with the coin is a hierarchical model over multiple coin flips, and doesn't need new probability concepts. Let $F_i$ be the flips. All you need in life is the distribution $P(F_1,F_2,\dots )$. You can decide to restrict yourself to distributions of the form ${\int }_0^{1}d{p}_{\text{coin}}P(F,G|p_{\text{coin}})p\left(p_{\text{coin}}\right)$. In practice, you start out thinking about $p_{\text{coin}}$ as a variable atop all the $F_i$ in a graph, and then think in terms of $P(F,G|p_{\text{coin}})$ and $p\left(p_{\text{coin}}\right)$ separately, because that... (read more)

A group of MR links led to a group of links that led to this list of Obvious Travel Advice. It seems like very good Obvious Travel Advice, and I endorse almost all points.

> A place that has staff trying to flag down customers walking past is almost certainly pursuing a get people in the door strategy.

To my great surprise, I found this to be false in Pisa (n_restaurant = 2).

Timothy Bates: The more things change, the more they stay the same: 1943 paper shows that a mechanical prediction of admissions greatly out predicts the decisions from administrators asked to add their subjective judgement :-(excellent talk from Nathan Kuncel !)

Nick Brown: I would bet that if you asked those subjective evaluators, they would say “We know the grades are the best predictor on average, but ‘sometimes’ they don’t tell the whole story”. People want to double-dip: Use the method most of the time, but add their own “special expertise”.

Timothy Bat

Unpolished first thoughts:

1. Selection effect: people who go to a blog to read bc they like reading, not doing
2. Concrete things are hard reads, math-heavy posts, doesn't feel ok to vote when you don't actually understand
3. In general easier things have wider audience
4. Making someone change their mind is more valuable to them than saying you did something?
5. There are many small targets and few big ideas/frames, votes are distributed proportionally

Talking 1-1 with music is so difficult to me that I don't enjoy a place if there's music. I expect many people on/towards the spectrum could be similar.