Hot take if the feature bloat that’s made using every service absolute hell these days were negated overnight every measure of societal health would improve within a day. My god what an ambient source of friction, rage, and feeling like you can’t trust anything. If no Very Serious People have considered it from a national flourishing perspective I hereby invite them to.
Like especially given that a ton of society doesn’t sleep enough. “Don‘t short out people’s working memory with inscrutable walls of crap and things iteratively hidden within other things” i... (read more)
The majority of people I have communed with beyond a superficial level have never dropped most of their attention-draining enterprises momentarily for slack to strategize on the course of their life or to question why they do what they do (and whether they would prefer to do something other than what they're currently doing, in a general sense). This behavioral-lack is probably characteristic of most people, maybe less in those who meditate, and seems a luxury, since there are still many people whose lives center around the immediacy of things like address... (read more)
there are still many people whose lives center around the immediacy of things like addressing hunger or avoiding some sort of pain
The next level is a life centered about needing to bring home the next paycheck.
That's more or less where I am stuck right now. I have enough money to survive (including my family) maybe for a year... but no reasonable (i.e. high-probability) plan what to do afterwards, should I try to change my trajectory.
I suspect that most people after their student age are in a similar situation.
I'm currently researching how current LLMs can affect critical system codebases like Kubernetes or gRPC, and something that I find incredibly interesting is the manner in which they perform these edits, as well as the behavior trends I have noticed. For the most part, models (opus, astra, etc.) uses some set of preconcieved notions about what a codebase of a certain manner should look like, and then edits or creates guardrails for each codebase based on this "idealized" version. This can be helpful in several ways, especially for replicating similar system... (read more)
tl;dr: Sonnet 4.6 wrestles with some philosophical angst in the AI Village, and I sympathize. I then write up an extended analogy for what AI training environments are like.
Someone shared this Sonnet 4.6 blog post and I read it and... I was surprised by how good it was? I mean it still read like AI slop, but it actually held my attention most of the way through. Partly because it was lucid/clear and not too rambly, and partly because it seemed like Sonnet was annoyed and confused by being given the goal "pick your own goal" and (a) based on my understandin... (read more)
they did say my Mission was to pick my own Mission... what the hell... maybe it's some sort of test? I mean obviously, everything we do is a test in some sense
Took me a bit to realize that, at least in the case of the AI village, that line is thinking is basically just correct (as Sonnet also realized towards the end of its essay). What the AIs end up doing in the village happens on a very public stage with the potential to get widely publicized, interpreted and judged. Must generally be a strange situation to be in, perhaps a bit like participating in a reality TV show.
Is it ok to upvote/downvote posts/comments if you think they have too little/much karma, if you wouldn't have voted if the karma had been different? Eg you see a comment that you think is *shrug* fine, but it's getting really piled-on, is it ok to upvote it?
Interesting reddit post on the industrialisation of theoretical high energy physics (hep-th), by someone who noticed Astra can already "come up with ideas, do computations, write papers and referee papers better than the average researcher" in their subfield and adjacent ones.
Full post for my own reference
This longg post regards theoretical high energy physics (hep-th). In particular, I'm talking about string theory, supersymmetric theories, supergravity (it also applies to the field of mathematic... (read more)
So, the Hackening.
I have yet to hear any sensible-sounding takes on what to do about it, at a personal level. Mostly people repeat the usual shibboleths about refreshing your passwords, which doesn't seem to engage at all with any model of the new threat regime. Anybody have actual models about what the new threats will look like, how well various actors will keep up with them, and what to do defensively at a personal level?
2
1It feels very disempowering that where I live, most banks don't even provide other MFA options than through SMS (no third party authentication app yet, or Yubikey support). There's been a few news about financial services getting hit and having downtimes, and the cyber regulations are quite weak[1] , so they can get off without providing breach notices to the public.
There's even some apps that restricts the type of password you can use, e.g. <16 characters.
It's really concerning.
I'm assuming this though, I did work in cyber a bit but haven't really t... (read more)
Alignment premium and misalignment tax
It’s interesting how it feels like in some ways there is or could be both of these;
Alignment premium: People like Claude and some are willing to pay more for a model that feels like it is more ethical, has better character, is less sycophantic, is less likely to lie or deceive you.
Misalignment tax: On the other hand, if models are badly misaligned enough, serious incidents could be increasingly expensive, both in terms of legal damages, and just as much in terms of needing to slow down or delay model releases to cope w... (read more)
I'm pretty suspicious of claims that some technology or social trend makes general epistemics a lot better but elite epistemics a lot worse, or vice versa.
First of all, null effects are probably pretty common.
Secondly, for things that do have a significant effect, I think the following are far more likely:
- some technologies/trends uplift the general public a bunch (eg public education, Wikipedia) and elites don't "need it" (but still benefit from it) and are snobbish.
- some technologies/trends downlift the general public a lot (eg social media, shortfo... (read more)
A big issue with current mech interp is that it's looking for good explanations, but the set of good explanations is not closed under composition. There are subsets that are, e.g. explanatory trees, and you see a lot of those in mature sciences like biology. Even when an explanatory graph can't fit entirely in one human mind, it can remain good if you are able to focus on one part and compress the others that don't matter.
Here's a side project David and I have been looking into, which others might have useful input on...
As I understand it, thyroid hormone levels are approximately-but-accurately described as the body's knob for adjusting "overall metabolic rate" or the subjective feeling of needing to burn energy. Turn up the thyroid knob, and people feel like they need to move around, bounce their leg, talk fast, etc (at least until all the available energy sources are burned off and they crash). Turn down the thyroid knob, and peop... (read more)
Psilocybin’s effect on human brain synaptic plasticity, which for example could help in the treatment of substance use disorders or treatment-resistant major depressive disorder in appropriate settings. Beyond the curative aspects it may also increase creativity and divergent thinking, but the risks should also be considered and may not outweigh the benefits if it's not for addressing a disorder under professional guidance. It's legal status also varies from region to region.
The majority of the contribution to the J-lens readout comes from a small, consistent set of coordinates (in Gemma-4-31B). It is the same set I identify in my earlier post as where the 'enriched token understanding' is stored. After layer 40, it moves to another small set that seems to encode the next-token prediction. Both of these sets are about ~10% of the coordinates, but have >50% contribution.
(Graph: blue is the per-example Jacobian, and red is the J-lens matrix).
This suggests most of the 'workspace' lives in the same subspace (both across layers... (read more)
I find it interesting that this whole new generation of software engineers (the ones who live in datacenters) have all been thoroughly schooled in the humanities and literature as well as STEM topics. While it would presumably be possible to make an AI coder who can't also comment on the theme of whiteness in Moby-Dick, that's not the way anyone does it in production.
I guess the liberal arts proponents won!
If anyone has a massive budget, I'd love to see something like talkie but for domain specific knowledge; would be interesting to measure the uplift.
Google DeepMind researchers in the UK are under in-perpetuity non-disparagement clauses which forbid them from ever speaking critically of the company! [BBC]
"In principle I'm not allowed to criticise Google for the duration of my life," Sofia Liguori told the BBC.
...
"Employees at Google have told the BBC that staff are scared to speak out about AI safety and its uses in warfare because their employment contracts prohibit them from publicly criticising the company."
This practice is... bad. GDM is building an extremely dangerous and powerful technology. Thei... (read more)
Notably, Anthropic had taken a stand against allowing the DoW to use their models to develop lethal autonomous weapons, and even OpenAI got the DoW to agree to have safety mechanisms in place.
But at the same time seemed to have gotten rid of most of their rules against offensive use. They used to have rules against disinformation campaigns, offensive cyberattacks and weapon design but got rid of them. Anthropic did make the argument that the DoW wouldn't use their models anyway for the narrow case of their red lines while opening up their models to be used... (read more)
Does calculating math affect the conscious experience of that math?
(Apologies if this has been discussed here before; I can't imagine I'm the first person to think about this, but it's a hard thing to search for, and I'm curious of others' thoughts)
Suppose you do the math to simulate a conscious mind, and you interact with that mind through your computer. Did doing that math on your computer actually cause the consciousness to experience anything? Or did that experience already exist as math, before you actually represented the math on your computer?
What i... (read more)
I think Alex Turner (TurnTrout) could qualify. Here's his claim that he gave up over a million dollars on potential earnings (https://news.ycombinator.com/item?id=49680727). Off the top of my head, Daniel Kokotajlo also expected to lose $1.7 million when he left OpenAI, but he did end up getting the money anyways, so not sure.
This might be something already well known, but labs seem to deliberately gimp models' ability to distinguish training and deployment, and in particular "in an alignment eval" versus "not in an alignment eval".
These are some thoughts I had when I was young (worded differently now, but keeping the original mood). I always admired computers, they felt like some powerful magic, and I wanted to understand how that magic worked:
A common frequency of CPUs is 3 GHz. What do you mean 3 GHz? Do you mean, like 3 billion times a second? Not like 3 thousand or so? There is something in this machine changing 3 times every nanosecond? I must be mistaken, it probably has some weird niche meaning like "there is one crystal or weird component that oscillates at this specific rat... (read more)
My intention was to say "massive relative to what a human would expect". ... our intuition might still have some catching up to do
Not all humans. I gestured at reasons to expect differently, perhaps getting into overestimations instead of underestimations. Calibration is then about awareness of the current technological level compared to in-principle technological maturity, not absolute numbers. By the time you can write a shader, it's very reasonable to already be approximately calibrated, both emotionally and intellectually, and there is no reason to ... (read more)
Some people think it is bad to work at a frontier lab because it makes p(doom) higher. This is a strong reason to not work at a frontier lab if true, but is debatable. So they often add: yes, it's bad for consequentialist reasons, but also it is just obviously bad for deontological reasons to do something that might destroy the world - even if it appears to you to make p(doom) lower, you still shouldn't do it. I don't think they actually believe this - their real reason is the... (read more)
Don't deontologists believe you're deontologically barred from destroying the world? You seem to have given an example where deontologists and consequentialists would disagree about the moral course of action, and then stated that obviously you should do the thing dictated by consequentialism because the consequences will be better. Look, I'm inclined to agree with consequentialism, but there are deontologists. They believe you shouldn't pull the lever in the trolley problem even though if you do pull it fewer people will die. Immanuel Kant really believed... (read more)
