Customize

Quick Takes

Ryan Meservey, Eli Tyre, and 4 more

Notes on living semi-frugally in the Bay Area. I live in the Bay Area, but my cost of living is pretty low: roughly $30k/year. I think I live an extremely comfortable life. I try to be fairly frugal, both so I don't end up dependent on jobs with high salaries and so that I can donate a lot of my income, but it doesn't feel like much of a sacrifice. Often when I tell people how little I spend, they're shocked. I think people conceive of the Bay as exorbitantly expensive, and it can be, but it doesn't have to be. Rent: I pay ~$850 a month for my room. It's a small room in a fairly large group house I live in with nine friends. It's a nice space with plenty of common areas and a big backyard. I know of a few other places like this (including in even pricier areas like Palo Alto). You just need to know where to look and to be willing to live with friends. On top of rent I pay ~$200/month (edit: I was missing one expense, it's more like $300) for things like utilities, repairs on the house, and keeping the house tidy. I pool the grocery bill with my housemates so we can optimize where we shop a little. We also often cook for each other (notably most of us, including myself, also get free meals on weekdays in the offices we work from, though I don't think my cost of living was much higher when I was cooking for myself each day not that long ago). It works out to ~$200/month. I don't buy that much stuff. I thrift most of my clothes, but I buy myself nice items when it matters (for example comfy, somewhat-expensive socks really do make my day better when I wear them). I have a bunch of miscellaneous small expenses like my Claude subscription, toothpaste, etc, but they don't add up to much. I don't have a car, a child, or a pet (but my housemate has a cat, which is almost the same thing). I try to avoid meal delivery and Ubers, though I use them in a pinch. Public transportation costs aren't nothing, but they're quite manageable. I actually have a PA who helps me with

jacquesthibs21h3820

Jesper L., Raemon, and 2 more

Building an AI safety business that tackles the core challenges of the alignment problem is hard. Epistemic status: uncertain; trying to articulate my cruxes. Please excuse the scattered nature of these thoughts, I’m still trying to make sense of all of it. You can build a guardrails or evals platform, but if your main threat model involves misalignment via internal deployment with self-improving AI (potentially stemming from something like online learning on hard problems like alignment which leads to AI safety sabotage), it is so tied to capabilities that you will likely never have the ability to influence the process. You can build reliability-as-a-business but this probably speeds up timelines via second-order effects and doesn’t really matter for superintelligence. I guess you can hone in on the types of problems where Goodharting is an obvious problem and you are building reliable detectors to help reduce it. Maybe you can find companies that would value that as a feature and you can relate it to the alignment-relevant situations. You can build RL environments, sell evals or sell training data, but you still seemingly end up too far removed from what is happening internally. You could choose a high-stakes vertical you can make money with as a test-bed for alignment and build tooling/techniques that ensure a high-level of guarantees. If you have a theory of change, it will likely need to be some technical alignment breakthrough you make legible and low-friction to incorporate or some open source infra the labs can leverage. You can build ControlArena or Inspect, open-source it, and then try to make a business around it, but of course you are not tackling the core alignment challenges. Unless your entire theory of change is building infrastructure the labs will port into their local Frankenstein infra and that Control ends up being the only thing the labs needed for solving alignment with AIs. And I guess from a startup perspective, you recognize that bu

GradientDissenter3d9317

gjm, noahzuniga, and 3 more

Here's my attempt at a neutral look at Prop 50, which people in California can vote on Tuesday (Nov 4th). The bill seems like a case-study in high-stakes game theory and when to cooperate or defect. The bill would allow the CA legislature to re-write the congressional district maps until 2030 (when district-drawing would go back to normal). Currently, the district maps are drawn by an independent body designed to be politically neutral. In essence, this would allow the CA legislature to gerrymander California. That would probably give Democrats an extra 3-5 seats in Congress. It seems like there's a ~17% chance that it swings the House in the midterms. Gerrymandering is generally agreed to be a bad thing, since it means elections are determined on the margin more by the map makers and less by the people. The proponents of this bill don't seem to think otherwise. They argue the bill is in response to Texas passing a similar bill to redistrict in a way that is predicted to give Republicans 5 new house seats (not to mention similar bills in North Carolina and Missouri that would give republicans an additional 2 seats). Trump specifically urged Texas, North Carolina, and Missouri to pass their bills, and the rationale was straightforwardly to give Republicans a greater chance at winning the midterms. For example, Rep. Todd Hunter, the author of Texas's redistricting bill, said "The underlying goal of this plan is straightforward, [to] improve Republican political performance". Notably some Republicans have also tried to argue that the Texas bill is in response to Democrats gerrymandering and obstructionism, but this doesn't match how Trump seems to have described the rationale originally.[1] The opponents of Prop 50 don't seem to challenge the notion that the Republican redistricting was bad.[2] They just argue that gerrymandering is bad for all the standard reasons. So, it's an iterated prisoners' dilemma! Gerrymandering is bad, but the Republicans did it, maybe

Tomás B.3d6353

oligo, Mateusz Bagiński, and 15 more

Tallness is zero sum. But I suspect beauty isn't. If everyone was more beautiful but the relative differences remained, I think people would be happier. Am I wrong in this? This has policy implications, as once genetic engineering gets better taxing height is likely wise to avoid red-queens races into unhealthy phenotypes. Taxing beauty seems very horrible to me. As beauty is quite beautiful.

Daniel Tan5h40

Question for people with insider knowledge of how labs train frontier models: Is it more common to do alignment training as the last step of training, or RL as the last step of training? * Edit: I'm mainly referring to on-policy RL, e.g. the type of RL that is used to induce new capabilities like coding / reasoning / math / tool use. I'm excluding RLHF because I think it's pretty disanalogous (though I also welcome disagreement / takes on this point.) Naively I'd expect we want alignment to happen last. But I have a sense that usually RL happens last - why is this the case? Is it because RL capabilities are too brittle to subsequent finetuning?

Simon Lermen15h100

I ran a small experiment to discover preferences in LLMs. I asked the models directly if they had a preferences and then put the same models into a small role playing game where they could choose between different tasks. Models massively prefer creative work across model families and hate repetitive work. https://substack.com/home/post/p-178237064 This is still preliminary work.

romeostevensit3d4820

Amalthea, Viliam

One of the most important things I've learned from therapy and contemplative practice for pragmatic interactions is that people will usually dig their heels in if they detect you have a bottom line written first. This effect is so strong that it activates even when people agree with you on the bottom line (but they have any level of conflict about it). I think this is one of the overwhelming considerations for why Eliezer gets the results he does. It cuts against another effect public intellectuals are subject to, which is that the easiest way to get popular is to become 'The X Guy' where X is a particular thing that the public has room in their heads for. But if you're the X guy people also associate you with a particular bottom line, so good luck getting any truth finding interactions. Instead, you wind up as a piece that gets moved around the board for conflict/spectacle purposes.

Popular Comments

Raemon3d*8845

Heroic Responsibility

I think this part of Heroic Responsibility isn't too surprising/novel to people. Obviously the business owner has responsibility for the business. The part that's novel is more like: If I'm some guy working in legal, and I notice this hot potato going around, and it's explicitly not my job to deal with it, I might nonetheless say "ugh, the CEO is too busy to deal with this today and it's not anyone else's job. I will deal with it." Then you go to each department head, even if you're not even a department head you're a lowly intern (say), and say "guys, I think we need to decide who's going to deal with this." And if their ego won't let them take advice from an intern, you might also take it as your responsibility to figure out how to navigate their ego – maybe by making them feel like it was their own idea, or by threatening to escalate to the CEO if they don't get to it themselves, or by appealing to their sense of duty. A great example of this, staying with them realm of "random Bureaucracy", I got from @Elizabeth: E. D. Morel was a random bureaucrat at a shipping company in 1891. He noticed that his company was shipping guns and manacles into the Congo, and shipping rubber and other resources back out to Britain. It was not Morel's job to notice that this was a bit weird. It was not Morel's job to notice that that weirdness was a clue, and look into those clues. And then find out that what was happening was, weapons were being sent to the Congo to forcibly steal resources at gunpoint. It was not his job to make it his mission to raise awareness of the Congo abuses and stop them. But he did. ... P.S. A failure mode of rationalists is to try to take Heroic responsibility for everything, esp. in a sort of angsty way that is counterproductive and exhausting. It's also a failure mode to act as if only you can possibly take Heroic responsibility, rather than trying to model the ecosystem around you and the other actors (some of whom might be Live Players who are also taking Heroic Responsibility, some of whom might be sort of local actors following normal incentives but are still, like, part of the solution) There is nuance to when and how to do Heroic Responsibility well.

niplav1d*360

People Seem Funny In The Head About Subtle Signals

Hm, I am unsure how much to believe this, even though my intuitions go the same way as yours. As a correlational datapoint, I tracked my success from cold approach and the time I've spent meditating (including a 2-month period of usually ~2 hours of meditation/day), and don't see any measurable improvement in my success rate from cold approach: (Note that the linked analysis also includes a linear regression of slope -6.35e-08, but with p=0.936, so could be random.) In cases where meditation does stuff to your vibe-reading of other people, I would guess that I'd approach women who are more open to being approached. I haven't dug deeper into my fairly rich data on this, and the data doesn't include much post-retreat approaches, but I still find the data I currently have instructive. I wish more people tracked and analyzed this kind of data, but I seem alone in this so far. I do feel some annoyance at everyone (the, ah, "cool people"?) in this area making big claims (and sometimes money off of those claims) without even trying to track any data and analyze it, leaving it basically to me to scramble together some DataFrames and effect sizes next to my dayjob.[1] > So start meditating for an hour a day for 3 months using the mind illuminated as an experiment (getting some of the cool skills mentioned in Kaj Sotala's sequence?) and see what happens? Do you have any concrete measurable predictions for what would happen in that case? ---------------------------------------- 1. I often wonder if empiricism is just incredibly unintuitive for humans in general, and experimentation and measurement even more so. Outside the laboratory very few people do it, and see e.g. Aristotle's claims about the number of women's teeth or his theory of ballistics, which went un(con)tested for almost 2000 years? What is going on here? Is empiricism really that hard? Is it about what people bother to look at? Is making shit up just so much easier so that everyone keeps in that mode, which is a stable equilibrium? ↩︎

Raemon4d*10880

The Tale of the Top-Tier Intellect

Serious question: (well, it'll start as "more of a comment, really", but, at the end I do have a question) The comment: I think the world is bottlenecked on people understanding the sort of concepts in posts like these. I don't think the world is particularly bottlenecked on current-gen-Yudkowsky-shaped dialogue essays about it. They appeal to a small set of people. My guess is you write them anyway because they are pretty easy to write in your default style and maybe just mostly fun-for-their-own-sake. And when you're in higher-effort modes, you do also write things like If Anyone Builds It, that are shaped pretty different. And, probably these essays still help some people, and maybe they help workshop new analogies that eventually can be refined into If Anyone style books or podcast interviews. But, that said, my questions are: * How much have you experimented with finding low-energy-but-different formats, that might reroll on who finds them compelling? * (I'm particularly interested in if there turns out to be anything short in this reference class) * How much have you (or, anyone else, this doesn't have to be you) systematically thought about how to improve the distribution channel of this sort of essay so it reaches more people? Both of these are presumably high effort. I'm not sure if the the first one is a better use of your high-effort time than other things, or how likely it is to work out. But, wondering if this is an area you think you've already checked for low or mid-hanging fruit it. (Also, having now thought about it for 5 min, I think this sort of thing would actually make a good youtube video that the Rational Animations people might do. That could be mostly outsourced)