All of β-redex's Comments + Replies

Conditional on LLMs scaling to AGI, I feel like it's a contradiction to say that "LLMs offer little or negative utility AND it's going to stay this way". My model is that we are either dying in a couple years to LLMs getting us to AGI, and we are going to have a year or two or of AIs that can provide incredible utility, or we are not dying to LLMs and the timelines are longer.

I think I read somewhere that you don't believe LLMs will get us to AGI, so this might already be implicit in your model? I personally am putting at least some credence on the ai-2027... (read more)

Out of curiosity, why do you post on Twitter? Is it network effects, or does it just have such a unique culture that's not available anywhere else? (Or something else?) Do you not feel any kind of aversion towards the platform, which would potentially discourage you from interacting? (I don't mean this to sound accusatory, if your position is that "yes Twitter is awesome and more rationalists should join" I would also like to hear about that.)

2briangriffinlover420
Sometimes one wants to play around in the mud. I don't think I can provide a better reason for using Twitter than that.
4habryka
I mostly reply to people on Twitter. Also, the shape of my reply tends to be definitely much more oriented around "convincing the audience" or "correcting the record" than to have a conversation with the specific person I am replying to. This makes me a bit sad, but I think it overall makes public discourse better. 

As you highlight, asking everyone to set up their own cross-posting solution is probably not viable. But if there was some service run by the LW team that had a simple guide for setting it up (e.g. go to your LW account, get your Twitter API key, copy it here, grant permission, done.) and it took ~5 minutes, that would lower the barrier to entry a lot and would be a huge step forward.

Not even fitting for Quick Takes? We could have a "Quicker Quick Takes" or "LW Shitposts" section for all I care. (More seriously if you really wanted some separate category for this you could name it something like "LW Microblog".)

Also a lot of Eliezer's tweets are quite high effort, to the point where some of them get cross-posted as top level posts. (E.g. https://www.lesswrong.com/posts/fPvssZk3AoDzXwfwJ/universal-basic-income-and-poverty )

2habryka
I post a good amount on Twitter (mostly replying to other people) and think it wouldn't make any sense to post approximately any of that to LW. They are just very different platforms with very different cultures.

My strictly personal hot take is that Quick Takes became a place of hiding from responsibility of top level posts. People sometimes write very large Quick Takes which certainly would be top level posts, but, in my model of these people, top level posts have stricter moderation and cause harsher perception and scarier overall, so people opt out for Quick Takes. I'm afraid if we create even less strict section of LW, people are going to migrate there entirely.

β-redex3627

Why is a significant amount of content by some rationality adjacent people only posted on X/Twitter?

I hope I don't have to explain why some people would rather not go near X/Twitter with a ten foot pole.

The most obvious example is Eliezer, who is much more active on Twitter than LW. I "follow" some people on Twitter by reading their posts using Nitter (e.g. xcancel.com ).

What triggered me to post this today is that it seems @Aella set her account to followers-only (I assume due to some recent controversy), so now the only way for me to read her tweets woul... (read more)

5quetzal_rainbow
I think it's because these people don't consider their Twitter content fitting for LW? LW has strict requirements for signal-to-noise ratio and a lot of Twitter stuff is just shitposting.

I hope I don't have to explain why some people would rather not go near X/Twitter with a ten foot pole.

Right. So for me the even bigger question is "why is someone like Eliezer on twitter at all?" If politics is the mind killer, twitter is the mind mass-murderer.

It delivers you the tribal fights most tailored to trigger you based on your particular interests. Are you a grey tribe who can rise above the usual culture war battles? Well then here's an article about NIMBYs or the state of American education that's guaranteed to make you seethe with anger at pe... (read more)

1CstineSublime
I feel like this is almost identical to the question of "why don't people migrate to [insert social network/platform]?". I also wonder, provided that you got full consent or permission[1] from a certain number of Rationality Adjacent people, what otherwise is stopping you from setting up a scraper or bot to automate so that they don't have to lift a finger and yet their content is still available elsewhere? 1. ^ It may go without saying but: if someone has set their account to followers-only then I would presume that increases the likelihood of their reluctance to allow their content to be mirrored elsewhere. Even if whomever is administering the mirror is well meaning and trustworthy. It would be an interesting exercise to see how many people would or wouldn't consent to such a project since it promises that they wouldn't need to lift a finger.

A guide to taking the perfect dating app photo. This area of your life is important, so if you intend to take dating apps seriously then you should take photo optimization seriously, and of course you can then also use the photos for other things.


It has been ~1 year since this was posted, and photorealistic image generation has now gone mainstream with e.g. ChatGPT introducing it. People can now generate "improved" photos of themselves.

How has this affected dating apps? Could anyone actively using them weigh in on this?

I imagine the equilibrium to be every... (read more)

β-redex100

What? I am telling you it is.

and preferentially read things that take less cognitive effort (all else equal, of course)

Sorry, no offense meant, I am just genuinely surprised. But I believe you now, I guess our experiences are just very different in this regard.

that sentence is semantically dense and grammatically complicated. I have to put in some work to break it down into noun phrases and such and figure out how it fits together. requiring cognitive work of potential readers before they've even decided if they want to read your thing is extremely anti-memetic

Sorry, but I call bullshit on this being a problem for you, or any other LW reader.

Now you are probably right that if you take the general population, for a significant number of people parsing anything but the simplest grammatical structures is going t... (read more)

5tcheasdfjkl
  What? I am telling you it is. Not in the sense that I can't parse it, but in the sense that I notice the cognitive effort involved, and preferentially read things that take less cognitive effort (all else equal, of course). If I'm skimming a bunch of titles and trying to pick which thing to read, the difference between titles that lodge their meaning into my brain as soon as my eyes fall on them vs. titles that take an extra couple seconds to parse is going to matter. (Similarly, I prefer Cooking For Engineers recipe layouts to recipe blogs that require me to extract the instructions from longer-form text - not that I can't do that, but I don't prefer to.) Maybe this means you don't want me to read your post! But I don't think that's right. Titles are usually the most optimized-for-memeticness part of a post; I typically assume that the rest of it will be denser, and that's fine - if I'm reading your post I am probably sold on being interested in what you're saying. (Still better all things equal to make stuff easier to parse, when that doesn't trade off against other desiderata.)
β-redex7-3

I'm worried about Chicago

In what world is this a good title? It basically gives zero information about the topic of the article, it is exactly the kind of clickbait title that purposefully omits any relevant information from the title to try to make people click. I personally associate such clickbait titles with terrible content, and they make me much less likely to click.[1]

What would be wrong with just using the subtitle as the actual title? It's much more informative:

Slowing national population growth plus remote work spell big trouble for Midweste

... (read more)
3tcheasdfjkl
Using that subtitle as the title would make it work worse for me because:  * that title would be too long. many of the interfaces that show me titles would simply cut it off and I would not see the back half of it * that sentence is semantically dense and grammatically complicated. I have to put in some work to break it down into noun phrases and such and figure out how it fits together. requiring cognitive work of potential readers before they've even decided if they want to read your thing is extremely anti-memetic * the tone of it sounds like a dry academic paper. those are typically not very fun to read. it signals that this will also not be fun to read Whereas the actual title has none of those problems. It's short, it's easy to parse, it's conversational with a clear and compelling valence. It's true it does not already explain why the author is worried about Chicago, but I think that's fine - when the explanation is complex enough you should need click on the article to find it out. Clickbait is when the answer could easily fit into the title and the author chose not to do that. Here I don't think it can.

Interesting. While the post resonates with me, I feel like I am trying to go in the opposite direction right now, trying to avoid getting nerd sniped by all the various fields I could be getting into, and instead strategically choosing the skills so that they are the most useful for solving the bottlenecks for my other goals that are not "learning cool technical things".

Which is interesting, because so far based on your posts you struck me as the kind of person I am trying to be more like in this regard, being more strategic about my goals. So maybe the pe... (read more)

5StartAtTheEnd
Same here. I think it's because working on myself made me neglect my connection to everyday life. Working too much on yourself, in your own way, makes you a little bit incompatible with everything and everyone else. Wizard power (as I see it: A powerful mentality, independence, the ability to go ahead of everyone else) is best developed in isolation or smaller groups, but social connections are important for success in life. The feeling which caused me to switch was that of a pyramid being upside down. If you work on top of the needs hierarchy while neglecting the bottom like I did, you're putting yourself at risk, and this creates anxiety which is justified and thus hard to shake off again.  In my way of looking at this, Einstein had a bit of wizard power, and he had trouble getting recognized for it once he made his discoveries. Most people did not have much reason to believe that he was a genius, as he didn't have a lot of social proof. Tesla also leaned too much towards his intellectual pursuits, I think. He didn't have much in terms of money and friends, and this seemed to cause him difficulties even though he was such a competent person. An alternative route I've thought of is becoming a social wizard. Those people who have an unnatural amount of charisma and the ability to read people like books. About being nerd sniped - I think many things do bring some benefits. The problem is, even though you can do anything, you can not do everything. There simply isn't enough time. I like this quote, attributed to Leonardo da Vinci: "As every divided kingdom falls, so every mind divided between many studies confounds and saps itself." A thing which I can't put into words well but only warn about vaguely is the consequence of learning many new things too fast. When I do this, I face the same thing as OP does - I basically become another person. If I want to get back, I have to reverse my mindset, attitude, values, cognitive models, priorities, etc. and not just reme
7johnswentworth
I do think one needs to be strategic in choosing which wizard powers to acquire! Unlike king power, wizard powers aren't automatically very fungible/flexible, so it's all the more important to pick carefully. I do think there are more-general/flexible forms of wizard power, and it makes a lot of sense to specialize in those. For instance, CAD skills and knowing how to design for various production technologies (like e.g. CNC vs injection molding vs 3D printing) seems more flexibly-valuable than knowing how to operate an injection molding device oneself.
β-redex121

Wonder if correctness proofs (checked by some proof assistant) can help with this.[1]

I think the main bottleneck in the past for correctness proofs was that it takes much more effort to write the proofs than it takes to write the programs themselves, and current automated theorem provers are nowhere near good enough.

Writing machine checked proofs is a prime RL target, since proof assistant kernels should be adversarially robust. We have already seen great results from stuff like AlphaProof.


  1. One counterargument I could see is that writing the correctness

... (read more)

Just to add another data point, as a software engineer, I also find it hard to extract utility from LLMs. (And this has not been for a lack of trying, e.g. at work we are being pushed to use LLM enabled IDEs.) I am constantly surprised to hear when people on the internet say that LLMs are a significant productivity boost for them.

My current model is that LLMs are better if you are working on some mainstream problem domain using a mainstream tech stack (language, library, etc.). This is approximately JavaScript React frontend development in my mind, and as ... (read more)

3Viliam
Yes, I have the same impression. Generating Java or Python code using popular libraries: mostly okay. Generating Lean code: does not compile even after several attempts to fix the code by feeding the compiler errors to LLM.

I would definitely be interested if you found a way to self-review recordings of your social interactions for improvement. The main roadblock I see is that either you tell the other parties that you are recording, which will probably influence their behavior a lot and erase most of the signal you were looking for in the first place, or you don't, which does feel a bit unethical.

3niplav
Ethical concerns here are not critical imho, especially if one only listens to the recording oneself and deletes them afterwards. People will be mad if you don't tell them, but if you actually don't share it and delete it after a short time afterwards I don't think you'd be doing anything wrong.
3ProgramCrafter
Most cases, you could build a mini-quadcopter, "teach" it some tricks and try showcasing it, having video as a side effect!

What does "obscure" mean here? (If you label the above "obscure", I feel like every query I consider "non-trivial" could be labeled obscure.)

I don't think Lean is obscure, it's one of the most popular proof assistants nowadays. The whole Lean codebase should be in the AIs training corpus (in fact that's why I deliberately made sure to specify an older version, since I happen to know that the olean header changed recently.) If you have access to the codebase, and you understand the object representation, the solution is not too hard.

Here is the solution I w... (read more)

2Yair Halberstadt
The quantity of lean code in the world is orders of magnitude smaller than the quantity of python code. I imagine that most people reporting good results are using very popular languages. My experience using Gemini 2.5 to write lean was poor.

Yep, that "standard library" part sure seems problematic, I am not sure if an algorithm for listing primes is shorter than just the above lookup table.

Just to give an example, here is the kind of prompt I am thinking of. I am being very specific about what I want, I think there is very little room for misunderstanding about how I expect the program to behave:

Write a Python program that reads a .olean file (Lean v4.13.0), and outputs the names of the constants defined in the file. The program has to be standalone and only use modules from the python standard library, you cannot assume Lean to be available in the environment.

o3-mini gives pure garbage hallucination for me on this one, like it's not even close.

8Yair Halberstadt
That seems like an example of C (obscure technology)

If your answer to question A is "a specific thing," and your answer to B is "yes, I'm very clear on what I want," then just explain it thoroughly, and you're likely to get satisfying results. Impressive examples like "rewrite this large complex thing that particular way" fall into this category.

Disagree. It sounds like by "being specific" you mean that you explain how you want the task to be done to the AI, which in my opinion can only be mildly useful.

When I am specific to an AI about what I want, I usually still get buggy results unless the solution i... (read more)

1β-redex
Just to give an example, here is the kind of prompt I am thinking of. I am being very specific about what I want, I think there is very little room for misunderstanding about how I expect the program to behave: o3-mini gives pure garbage hallucination for me on this one, like it's not even close.

I guess the reasoning for why the solution given in the post is more "valid" than this one is "something something Occam's razor" or that it is "more elegant" (whatever "elegant" means), but if someone can make a more precise argument I would be interested to hear. (In particular, in my mind Occam's razor is something to do with empiricism, while what we are doing here is pure logic, so not sure how it exactly applies?)

2Yair Halberstadt
Kolmogorov complexity? Your solution takes more bits to specify than the one in the solution (at least if you've already defined a standard library with concepts like primes)?

Unfortunately no, I don't think any contradictions can be derived from the examples given in the post if we assume -E and [E] unary, and E E binary operators. Here are some example assignments for these operators that satisfy (AFAICT) the examples from the post (assuming left associativity for juxtaposition, and that the precedence of - is lower, so that -E E is interpreted as -(E E) in the last example):

Definitions for [E]:

1 => 2
2 => 4
-1 => 1/2
-2 => 1/4
1/2 => sqrt(2)
1001 => 1001
1004 => 1004

Definitions for E E:

1 2 => 1001
... (read more)
3β-redex
I guess the reasoning for why the solution given in the post is more "valid" than this one is "something something Occam's razor" or that it is "more elegant" (whatever "elegant" means), but if someone can make a more precise argument I would be interested to hear. (In particular, in my mind Occam's razor is something to do with empiricism, while what we are doing here is pure logic, so not sure how it exactly applies?)

I fairly quickly figured out that the grammar is something like E ::= "[]" | "[" E "]" | "-" E | "E E", and that eval([E]) = 2^eval(E) (and eval(-E) = -eval(E)), and then went down the rabbit hole of trying to come up with some f eval(E1 E2) = f(eval(E1), eval(E2)) for juxtaposition, and thinking about whether it's left or right associative. I was also thinking that maybe it's n-ary rather than binary so that associativity does not matter.

Anyway, I think where I went wrong is that I decided that [E] is a unary operator by itself, and did not reconsider thi... (read more)

1Zygi Straznickas
I did exactly the same thing. I'm curious, is there a way to systematically solve problems of this type? E.g. if I made an assumption that this is an algebra with one unary and one binary operation, can that be strictly refuted just from the examples given?

Pull requests. Useful to group a bunch of messy commits into a single high-level purpose and commit that to main. Makes your commit history easier to read.

You can also squash multiple commits without using PRs. In fact, if someone meticulously edited their commit history for a PR to be easy-to-follow and the changes in each commit are grouped based on them being some higher level logical single unit of change, squashing their commits can be actively bad, since now you are destroying the structure and making the history less readable by making a single m... (read more)

2Daniel Tan
IMO it's mainly useful when collaborating with people on critical code, since it helps you clearly communicate the intent of the changes. Also you can separate out anything which wasn't strictly necessary. And having it in a PR to main makes it easy to revert later if the change turned out to be bad.  If you're working by yourself or if the code you're changing isn't very critical, it's probably not as important

Does Overleaf have such AI integration that can get "accidentally" activated, or are you using some other AI plugin?

Either way, this sounds concerning to me, we are so bad at AI boxing that it doesn't even have to break out, we just "accidentally" hand it edit access to random documents. (And especially an AI safety research paper is not something I would want a misaligned AI editing without close oversight.)

Could someone explain the joke to me? If I take the above statement literally, some change made it into your document, which nobody with access claims to have put there. You must have some sort of revision control, so you should at least know exactly who and when made that edit, which should already narrow it down a lot?

5johnswentworth
The joke is that Claude somehow got activated on the editor, and added a line thanking itself for editing despite us not wanting it to edit anything and (as far as we've noticed) not editing anything else besides that one line.

I am not a 100% convinced by the comparison, because technically LLMs are only "reading" a bunch of source code, they are never given access to a compiler/interpreter. IMO actually running the code one has written is a very important part of learning, and I think it would be a much more difficult task for a human to learn to code just by reading a bunch of books/code, but never actually trying to write & run their own code.[1]

Also, in the video linked earlier in the thread, the girlfriend playing Terraria is deliberately not given access to the wiki, a... (read more)

1ErickBall
I kind of see your point about having all the game wikis, but I think I disagree about learning to code being necessarily interactive. Think about what feedback the compiler provides you: it tells you if you made a mistake, and sometimes what the mistake was. In cases where it runs but doesn't do what you wanted, it might "show" you what the mistake was instead. You can learn programming just fine by reading and writing code but never running it, if you also have somebody knowledgeable checking what you wrote and explaining your mistakes. LLMs have tons of examples of that kind of thing in their training data.

And as a separate note, I'm not sure what the appropriate human reference class for game-playing AIs is, but I challenge the assumption that it should be people who are familiar with games. Rather than, say, people picked at random from anywhere on earth.

If you did that for programming, AIs would already be considered strongly superhuman. Just like we compare AI's coding knowledge to programmers, I think it's perfectly fair to compare their gaming abilities to people who play video games.

1ErickBall
Yeah but we train AIs on coding before we make that comparison. And we know that if you train an AI on a videogame it can often get superhuman performance. Here we're trying to look at pure transfer learning, so I think it would be pretty fair to compare to someone who is generally competent but has never played videogames. Another interesting question is to what extent you can train an AI system on a variety of videogames and then have it take on a new one with no game-specific training. I don't know if anyone has tried that with LLMs yet.
4MondSemmel
By this I was mainly arguing against claims like that this performance is "worse than a human 6-year-old".

Notably, this was exactly the sort of belief I was trying to show is false

Please point out if there is a specific claim I made in my comment that you believe to be false. I said that "I don't think a TC computer can ever be built in our universe.", which you don't seem to argue with? (If we assume that we can only ever get access to a finite number of atoms. If you dispute this I won't argue with that, neither of us has a Theory of Everything to say for certain.)

Just to make precise why I was making that claim and what it was trying to argue against, ta... (read more)

3Noosphere89
I think I understand the question now. I actually agree that if we assume that there's a finite maximum of atoms, we could in principle reformulate the universal computer as a finite state automaton, and if we were willing to accept the non-scalability of a finite state automaton, this could actually work. The fundamental problem is that now we would have software that only works up to a specified memory limit, because we essentially burned the software into the hardware of the finite automaton and if you are ever uncertain of how much memory or time a problem requires, or more worryingly if we were ever uncertain about how much resources we could actually use, then our "software" for the finite automaton is no longer usable and we'd have to throw it away and recreate a new computer for every input length. Turing Machine models automatically handle arbitrarily large inputs without having to throw away expensive work on developing the software. So in essence, if you want to handle the most general case, or believe unbounded atoms are possible, like me, then you really want the universal computer architecture of modern computers. The key property of real computers that makes them Turing Complete in theory is that they can scale with more memory and time arbitrarily without changing the system descriptior/code. More below: https://www.dwarkesh.com/p/adam-brown

I don't think a TC computer can ever be built in our universe. The observable universe has a finite number of atoms, I have seen numbers around thrown around. Even if you can build a RAM where each atom stores 1 bit,[1] this is still very much finite.

I think a much more interesting question is why TC machines are — despite only existing in theory — such useful models for thinking about real-world computers. There is obviously some approximation going on here, where for the vast majority of real-world problems, you can write them in such a way that the... (read more)

-2Noosphere89
Notably, this was exactly the sort of belief I was trying to show is false, and your observation about the physical universe does not matter for the argument I made here, because the question is whether with say 2^1000000 atoms, you can solve larger problem sizes with the same code, and Turing-complete systems say yes to the question. In essence, it's a question of whether we can scale our computers with more memory and time without having to change the code/algorithms, and basically all modern computers can do this in theory. Short version, because you can always extend them with more memory and time, and it really matters a lot in practical computing if you can get a general coding solution that is also cheap to work with, because it can handle upgrades to it's memory very well. In essence, the system descriptor/code doesn't have to change if the memory and time increases (for a finite state machine or look-up table, they would have to change.

Let be the state space of our finite/physical computer, where is the number of bits of state the computer has. This can include RAM, non-volatile storage, CPU registers, cache, GPU RAM, etc... just add up all the bits.

The stateless parts of the computer can be modeled as a state transition function , which is applied at every time step to produce the next state. (And let's suppose that there is some special halting state .)

This is clearly a FSM with states, and not TC. The halting problem can be trivially solved for it: it is guarante... (read more)

2Noosphere89
Notably, this is why we focus on the arbitrarily large memory and time case, where we can assume that the machine has arbitrarily large memory and time to work with. The key question here is whether a finite physical computer can always be extended with more memory and time without requiring us to recode the machine into a different program/computer, and most modern computers can do this (modulo physical issues of how you integrate more memory and time). In essence, the key property of modern computers is that the code/systems descriptor doesn't change if we add more memory and time, and this is the thing that leads to Turing-completeness if we allow unbounded memory and time.

I think the reacts being semantic instead of being random emojis is what makes this so much better.

I wish other platforms experimented with semantic reacts as well, instead of just letting people react with any emoji of their choosing, and making you guess whether e.g. "thumbs up" means agreement, acknowledgement, or endorsement, etc.

This was my first time taking this, looking forward to the results!

5Jacob G-W
Same!
Answer by β-redex305

I know of Robert Miles, and Writer, who does Rational Animations. (In fact Robert Miles' channel is the primary reason I discovered LessWrong :) )

Don't leave me hanging like this, does the movie you are describing exist? (Though I guess your description is a major spoiler, you would need to go in without knowing whether there will be anything supernatural.)

4Ben
Many of the feature length Scooby Doo cartoon films I watched as a child do a big reveal where, actually this time (because its a longer story) some aspect of the magic or monster turns out to be real. I think one actually has both the fake and real monster.
1Ninety-Three
There is a narrative-driven videogame that does exactly this, but unfortunately I found the execution mediocre. I can't get spoilers to work in comments or I'd name it. Edit: It's
2Raemon
Oh, no I just made it up, alas. This is a sketch of how someone else should construct such a movie. (Maybe the movie exists but yeah if it did it'd unfortunately now be a spoiler in this context. :P)
Answer by β-redex120
  1. The Thing: classic
  2. Eden Lake
  3. Misery
  4. 10 Cloverfield Lane
  5. Gone Girl: not horror, but I specifically like it because of how agentic the protagonist is

2., 3. and 4. have in common that there is some sort of abusive relationship that develops, and I think this adds another layer of horror. (A person/group of people gain some power over the protagonist(s), and they slowly grow more abusive with this power.)

5TekhneMakre
I was going to say The Thing. https://en.wikipedia.org/wiki/The_Thing_(1982_film)

Somewhat related: does anyone else strongly dislike supernatural elements in horror movies?

It's not that I have anything against a movie exploring the idea of "what if we suddenly discovered that we live in a universe where supernatural thing X exist", but the characters just accept this without much evidence at all.

I would love a movie though where they explore the more likely alternate hypotheses first (mental issues, some weird optical/acoustic phenomenon, or just someone playing a super elaborate prank), but then the evidence starts mounding, and eventually they are forced to accept that "supernatural thing X actually exists" is really the most likely hypothesis.

3Elizabeth
I saw a review of a reality show like this. The contractor agent was just one "your door hinges were improperly installed" after another. One couple saw a scary shadow face on their wall and it turned out to a patch job on the paint that matched perfectly in daylight but looked slightly different in low light. Oculus isn't perfect for this, she starts out already believing in the haunted house due to past experience, but has a very scientific approach to proving it.
3Raemon
I think my favorite [edit: ideal hope] for this sort of thing has 2-3 killers/antogonists/scary-phenomena, and one of them turns out to be natural, and one supernatural. So the audience actually has to opportunity to figure it out, rather than just be genre savvy and know it'll eventually turn out to be supernatural.
β-redex1420

These examples show that, at least in this lower-stakes setting, OpenAI’s current cybersecurity measures on an already-deployed model are insufficient to stop a moderately determined red-teamer.

I... don't actually see any non-trivial vulnerabilities here? Like, these are stuff you can do on any cloud VM you rent?

Cool exploration though, and it's certainly interesting that OpenAI is giving you such a powerful VM for free (well actually not because you already pay for GPT-4 I guess?), but I have to agree with their assessment which you found that "it's expected that you can see and modify files on this system".

The malware is embedded in multiple mods, some of which were added to highly popular modpacks.

Any info on how this happened? This seems like a fairly serious supply chain attack. I have heard of incidents with individual malicious packages on npm or PyPI, but not one where multiple high profile packages in a software repository were infected in a coordinated manner.

Uhh this first happening in 2023 was the exact prediction Gary Marcus made last year: https://www.wired.co.uk/article/artificial-intelligence-language

Not sure whether this instance is a capability or alignment issue though. Is the LLM just too unreliable, as Gary Marcus is saying? Or is it perfectly capable, and just misaligned?

I don't see why communicating with an AI through a BCI is necessarily better than through a keyboard+screen. Just because a BCI is more ergonomic and the AI might feel more like "a part of you", it won't magically be better aligned.

In fact the BCI option seems way scarier to me. An AI that can read my thoughts at any time and stimulate random neurons in my brain at will? No, thanks. This scenario just feels like you are handing it the "breaking out of the box" option on a silver platter.

3RussellThor
The idea is that the BCI is added slowly and you integrate the new neurons into you in a continuous identity preserving way., the AI thinks your thoughts.

Why is this being downvoted?

From what I am seeing people here are focusing way too much on having a precisely calibrated P(doom) value.

It seems that even if P(doom) is 1% the doom scenario should be taken very seriously and alignment research pursued to the furthest extent possible.

The probability that after much careful calibration and research you would come up with a P(doom) value less than 1% seems very unlikely to me. So why invest time into refining your estimate?

5the gears to ascension
because it fails to engage with the key point: that the low predictiveness of the dynamics of ai risk makes it hard for people to believe there's a significant risk at all. I happen to think there is; that's why I clicked agree vote. but I clicked karma downvote because of failing to engage with the key epistemic issue at hand.

There was a recent post estimating that GTP-3 is equivalent to about 175 bees. There is also a comment there asserting that a human is about 140k bees.

I would be very interested if someone could explain where this huge discrepancy comes from. (One estimate is equating synapses with parameters, while this one is based on FLOPS. But there shouldn't be such a huge difference.)

2LawrenceC
Author of the post here — I don’t think there’s a huge discrepancy here, 140k/175 is clearly within the range of uncertainty of the estimates here! That being said the Bee post really shouldn’t be taken too seriously. 1 synapse is not exactly one float 16 or int8 parameter, etc
2the gears to ascension
I didn't read that post. should I? is it more than a joke? edit: I read it. it was a lot shorter than I expected, sometimes I'm a dumbass about reading posts and forget to check length. it's a really simple point, made in the first words, and I figured there would be more to it than that for some reason. there isn't.
6benjamincosman
10^15 - 10^30 is not at all a narrow range! So depending on what the 'real' answer is, there could be as little as zero discrepancy between the ratios implied by these two posts, or a huge amount. If we decide that GPT-3 uses 10^15 FLOPS (the inference amount) and meanwhile the first "decent" simulation of the human brain is the "Spiking neural network" (10^18 FLOPS according to the table), then the human-to-GPT ratio is 10^18 / 10^15 which is almost exactly 140k / 175. Whereas if you actually need the single molecules version of the brain (10^43 FLOPS), there's suddenly an extra factor of ten septillion lying around.
1RomanS
I wouldn't call it a huge discrepancy. If both values are correct, it means the human brain requires only 10² - 10³ more compute than GPT-3.  The difference could've been in dozens or even hundreds of OOMs, but it's only 2 - 3, which is quite interesting. Why the difference in compute is so small, if the nature of the two systems is so different? 

Indeed (as other commenters also pointed out) the ability to sexually reproduce seems to be much more prevalent than I originally thought when writing the above comment. (I thought that eukaryotes only capable of asexual reproduction were relatively common, but it seems that there may only be a very few special cases like that.)

I still disagree with you dismissing the importance of mitochondria though. (I don't think the OP is saying that mitochondria alone are sufficient for larger genomes, but the argument for why they are at least necessary is convincing to me.)

I disagree with English (in principle at least) being inadequate for software specification.

For any commercial software, the specification basically is just "make profit for this company". The rest is implementation detail.

(Obviously this is an absurd example, but it illustrates how you can express abstractions in English that you can't in C++.)

I don't think the comparison of giving a LLM instructions and expecting correct code to be output is fair. You are vastly overestimating the competence of human programmers: when was the last time you wrote perfectly correct code on the very first try?

Giving the LLM the ability to run its code and modify it until it thinks its right would be a much fairer comparison. And if, as you say, writing unit tests is easy for a LLM, wouldn't that just make this trial-and-error loop trivial? You can just bang the LLM against the problem until the unit tests pass.

(And this process obviously won't produce bug-free code, but humans don't do that in the first place either.)

Not all eukaryotes employ sexual reproduction. Also prokaryotes do have some mechanisms for DNA exchange as well, so copying errors are not their only chance for evolution either.

But I do agree that it's probably no coincidence that the most complex life forms are sexually reproducing eukaryotes.

1DPiepgrass
I see that someone strongly-disagreed with me on this. But are there any eukyrotes that cannot reproduce sexually (and are not very-recently-decended from sexual-reproducers) but still maintain size or complexity levels commonly associated with eukyrotes?
5Steven Byrnes
I’m pretty sure that I read (in Nick Lane’s The Vital Question) that all eukaryotes employ sexual reproduction at least sometimes. It’s true that they might reproduce asexually for a bunch of generations between sexual reproduction events. (It’s possible that other people disagree with Nick Lane on this, I dunno.)

I barely registered the difference between small talk and big talk

I am still confused about what "small talk" is after reading this post.

Sure, talking about the weather is definitely small talk. But if I want to get to know somebody, weather talk can't possibly last for more than 30 seconds. After that, both parties have demonstrated the necessary conversational skills to move on to more interesting topics. And the "getting to know each other" phase is really just a spectrum between surface level stuff and your deepest personal secrets, so I don't reall... (read more)

3[anonymous]
You can show disinterest or make up excuses to leave the social interaction. These behaviors are a lot worse, IMO, than just telling them straight up, but these are the socially accepted methods. Talking about sociopathic tendencies in the general public...

It was actually this post about nootropics that got me curious about this. Apparently (based on self reported data) weightlifting is just straight up better than most other nootropics?

Anyway, thank you for referencing some opposing evidence on the topic as well, I might try to look into it more at some point.

(Unfortunately, the thing that I actually care about - whether it has cognitive benefits for me - seems hard to test, since you can't blind yourself to whether you exercised.)

I think this is (and your other post about exercise) are good practical examples of situations where rational thinking makes you worse off (at least for a while).

If you had shown this post to me as a kid, my youth would probably have been better. Unfortunately no one around me was able to make a sufficiently compelling argument for caring about physical appearance. It wasn't until much later that I was able to deduce the arguments for myself. If I just blindly "tried to fit in with the cool kids, and do what is trendy", I would have been better off.

I wonde... (read more)

This alone trumps any other argument mentioned in the post. None of the other arguments seem universal and can be argued with on an individual basis.

I actually like doing things with my body. I like hiking and kayaking and mountain climbing and dancing.

As some other commenters noted, what if you just don't?

I think it would be valuable if someone made a post just focused on collecting all the evidence for the positive cognitive effects of exercise. If the evidence is indeed strong, no other argument in favor of exercise should really matter.

Bezzi107

Well, I've always been quite skeptical about the supposed huge mental benefits of exercising. I surely don't feel immediate mental benefits while exercising, and the first time I heard someone else claiming this I seriously thought it was a joke (it must be one of those universal human experiences that I am missing).

Anyway, I can offer one reference digged up from SSC

Although the role of poor diet/exercise in physical illness is beyond questioning, its role in mental illness is more anecdotal and harder to pin down. Don’t get me wrong, there are lot

... (read more)

FWIW I don't think that matters, in my experience interactions like this arise naturally as well, and humans usually perform similarly to how Friend did here.

In particular it seems that here ChatGPT completely fails at tracking the competence of its interlocutor in the domain at hand. If you asked a human with no context at first they might give you the complete recipe just like ChatGPT tried, but any follow up question immediately would indicate to them that more hand-holding is necessary. (And ChatGPT was asked to "walk me through one step at a time", which should be blatantly obvious and no human would just repeat the instructions again in answer to this.)

Cool! (Nitpick: You should probably mention that you are deviating from the naming in the HoTT book. AFAIK usually and types are called Pi and Sigma types respectively, while the words "product" and "sum" (or "coproduct" in the HoTT book) are reserved for and .)

I am especially looking forward to discussion on how MLTT relates to alignment research and how it can be used for informal reasoning as Alignment Research Field Guide mentions.

I always get confused when the term "type signature" is used in text unrelated to type theory. Like what do peop... (read more)

Load More