In general, the hacking capabilities of state actors and the likely involvement of national security when we get closer to AGI feel like significant blind spots of Lesswrong discourse.

(The Hacker and The State by Ben Buchanan is a great book to learn about the former)

If you are very good at cyber and extremely smart, you can hide vulnerabilities in 10k-lines programs in a way that less smart specialists will have trouble discovering even after days of examination - code generation/analysis is not really defense favored

I think the first part of the sentence is true, but "not defense favored" isn't a clear conclusion to me. I think that backdoors work well in closed-source code, but are really hard in open-source widely used code − just look at the amount of effort that went into the recent xz / liblzma backdoor, and the fact that we don't know of any other backdoor in widely used OSS.

The main effect of a market being underground is not making transactions harder (people find ways to exchange money for vulnerabilities by building trust), but making it much harder to figure out what the market price is and reducing the effectiveness of the overall market

Note this doesn't apply to all types of underground markets: the ones that regularly get shut down (like darknet drug markets) do have a big issue with trust.

Being the target of an autocratic government is an awful experience, and you have to be extremely careful if you put anything they dislike on a computer. And because of the zero-day market, you can't assume your government will suck at hacking you just because it's a small country

This is correct. As a matter of personal policy, I assume that everything I write down somewhere will get leaked at some point (with a few exceptions, like − hopefully − disappearing signal messages).

The link of "this is a linkpost for" is not the correct one

Here are the same two GIFs but with a consistent speed (30ms/frame) and an infinite loop, in case anyone else is interested for e.g. presentations:

Another mitigation strategy against the "LLM learns steganography from other LLM's output" problem is to add some noise to the training data (performing a number of transformations that preserve meaning) prior to training, to break the embedded steganography schemes. I feel like this should be relatively easy and effective?

Note that the loss would obviously be worse on the outputs of the previous LLM.

Which makes me think that comparing losses could be a way to detect that some text contains steganography, even if we can't pinpoint how exactly (but our transformations break it).

EDIT: in case of disagreement, I can try being more specific about my claim that this should be relatively easy and effective, using concrete examples. Basically I believe that after scrambling, there can only remain a small number of bits of steganographic information per token, even for a steganographic encoding that originally used the null space of natural language optimally.

EDIT

Thanks for the replies and sorry for the inaccuracies. I initially reported 4,331 blog posts and 890k words; the real results are that Robin wrote 3,302 blog posts (thanks DominikPeters for pointing this out, and for finding these better urls) and 1.5M words.

(4,331 blog posts corresponds to all authors on overcomingbias. 890k words doesn't represent anything, because the posts were truncated when accessed from the monthly archive urls.)

# Get the real number of words from Robin
$ n_current_pages=331
$ echo https://www.overcomingbias.com/author/robin-hanson > /tmp/page_urls
$ for i in $(seq 2 $n_current_pages); do echo https://www.overcomingbias.com/author/robin-hanson/page/$i >> /tmp/page_urls; done
$ getwords() { curl $1 | pup '#content' | html2text --ignore-links | wc -w; }
$ export -f getwords
$  parallel getwords < /tmp/page_urls > /tmp/words_by_page
$ awk '{sum += $1} END {print sum}' /tmp/words_by_page
1481344

ORIGINAL

Scoping: 4331 blog posts and 890k words (for overcomingbias only).

# Number of blog posts
$ curl https://www.overcomingbias.com/archives | pup '#monthly-archives' | rg '\(\d+\)' | tr -d ' ()' | awk '{sum += $1} END {print sum}'
4331

# Rough number of words (bash)
$ curl https://www.overcomingbias.com/archives | pup '#monthly-archives a attr{href}' > /tmp/urls_monthly_archives
$ getwords() { curl $1 | pup '#content' | html2text --ignore-links | wc -w; }
$ export -f getwords
$ parallel getwords < /tmp/urls_monthly_archives > /tmp/words_per_month
$ awk '{sum += $1} END {print sum}' /tmp/words_per_month
891666

Interesting... Can you tell more about how your self-control training looked like? Like when in the day, how long, how hard, what tasks, etc? Was the most productive period in your life during or after this training? Why did you stop?

To carry on with the strength training comparison, we're usually trying to achieve a maximum deployed strength over our lifetime. Perhaps we're already deploying as much strength as we can every day for useful tasks, so that adding strength training on pointless tasks would remove strength from the other tasks?

In my opinion, the 4 killer features of vim beyond moving around and basic editing are:

• macros
• the s command used with regular expressions
• the g command (see e.g. Power of g)
• the ability to run text through any Unix utility (Vim and the shell)

If you know your Unix utilities (I often use awk inside of vim; it's also great outside of it), your regular expressions, and these features, you can sculpt any text as you wish. What I mean is that you can take the output of pretty much anything, in a messed up format, and transform it into the content and format you want. This is supposed to be inspiring but I'm not sure how good a job I'm doing.

Also, if anyone's interested, here are my current vim Anki cards. I use Anki for keyboard shortcuts which are supposed to be muscle memory, AMA