All of Sune's Comments + Replies

I use ChatGPT and Claude to try to learn Macedonian, because there is only very little learning material available for that language. For example, they can (with a few errors sometimes) explain grammatical concepts or give me sentences to translate. I have not found a good way of storing a description of my abilities and weaknesses across conversations, but within a conversation they are good at adapting the difficulty of the questions to the quality of my answers. 
Unfortunately I’m not aware of any tools that can pronounce or transcribe Macedonian.&n... (read more)

What is the definition of a Dyson Swarm? Is it really easier to define, or just easier to see that we are not there, only because we are not close yet?

That is assuming you live sufficiently long. The point in life insurance is to make sure you leave something for your kids/spouse if you die soon.

Why does the plot start at 3x3 instead of 2x2? Of course, it is not common to have games with only one choice, but for Chicken that is what you end up with when removong one option. You could even start the investigation at 2x1 options.

Retracting the comment because I have seen a couple of couterexamples, including myself!

The alcor-page was not updated since 15th December 2022, where a person who died in August 2022 (as well as later data) was added, so if he was signed up there, we should not expect it too be mentioned yet. For CI latest update was for a patient dying 29th February 2024, but I can’t see any indication of when that post was made.

My point is that potential parents often care about non-existing people: their potential kids. And once they bring these potential kids into existence, those kids might start caring about a next generation. Simularly, some people/minds will want to expand because that is what their company does, or they would like the experience of exploring a new planet/solar system/galaxy or would like the status of being the first to settle there.

Which non-existing person are you refering to?

Beyond a certain point, I doubt that the content of the additional minds will be interestingly novel.

Somehow people keep finding meaning in failling in love and starting a family, even when billions of people have already done that before. We also find meaning in doing careers that are very similar to what million of people have done before or traveling to destination that has been visited by millions of turist. The more similar an activity is to something our ancestors did, the more meaningful it seems.

From the outside, all this looks grabby, but from the inside it feels meaningful.

There has been enough discussion about timelines that it doesn’t make sense to provide evidence about it in a post like this. Most people on this site has already formed views about timelines, and for many, these are much shorter than 30 years. Hopefully, readers of this site are ready to change their views if strong evidence in either direction appears, but I dont think it is fair to expect a post like this to also include evidence about timelines.

There is a huge amount of computation going on in this story and as far as I can tell not even a single experiment. The end hints that there might be some learning from the protagonists experince, at least it is telling it story many times. But I would expect a lot more experimenting, for example with different probe designs and with how much posthumans like different possible negotiated results.

I can see in the story that it make sense not to experiment with posthumans reactions to scenarios, since it might take a long time to send them to the fronter and... (read more)

An alternative reason for building telescopes would be to recieve updates and more efficient strategies for expanding found after the probe was send out.

How did this happen?! I guess not by rationalists directly trying to influence the pope? But I’m curious to know the process leading up to this.

What does respect mean in this case? That is a word I don’t really understand and seems to be a combination of many different concepts being mixed together.

This is also just another way of saying “willing to be vulnerable” (from my answer below) or maybe “decision to be vulnerable”. Many of these answers are just saying the same thing in different words.

My favourite definition of trust is “willingness to be vulnerable” and I think this answers most of the questions in the post. For example it explains why trust is a decision that can exist independently from your beliefs: if you think someone is genuinely on your side with probability 95%, you can choose to trust them, by doing something that benefit you in 95% of cases and hurt you on the 5% of cases, or you can decide not to, by taking actions that are better in the 5% of cases. Similar for trusting a statement about the world.

I think this definition co... (read more)

whilst the Jews (usually) bought their land fair and square, the owners of the land were very rarely the ones who lived and worked on it.

I have heard this before but never understood what it meant. Did the people who worked the land respect the ownership of the previous owners, for example by paying rest or by being employed by the previous owners, but they just did not respect the sale? Or did the people who worked the land consider themselves to be owners or didn’t have the same concept of ownership as we do today?

The heading of this question is misleading, but I assume I should answer the question and ignore the heading

P(Global catastrophic risk) What is the probability that the human race will make it to 2100 without any catastrophe that wipes out more than 90% of humanity?

You don’t really need the producers to be “idle”, you just have to ensure that if something important shows up, they are ready to work on that. Instead of having idle producers, you can just have them work on lower priority tasks. Has this also been modelled in queueing theory?

I have a question that tricks GPT-4, but if I post it I’m afraid it’s going to end up in the training data for GPT-5. I might post it once there is a GPT-n that solves it.

The corporate structure of OpenAI was set up as an answer to concerns (about AGI and control over AGIs) which were raised by rationalists. But I don’t think rationalists believed that this structure was a sufficient solution to the problem, anymore than non-rationalists believed it. The rationalists that I have been speaking to were generally mostly sceptical about OpenAI.

They were not loyal to the board, but it is not clear if they were loyal to The Charter since they were not given any concrete evidence of a conflict between Sam and the Charter.

I don’t understand how this is a meaningful attitude to your own private economy. But want to donate to someone who needs it more is also a way to spend your money. This would be charity, possibly EA.

I have noticed a separate disagreement about what capitalism means, between me and a family member.

I used to think of it as how you handle your private economy. If you are a capitalist, it means that when you have surplus, you save it up and use it (as capital) to improve your future, i.e. you invest it. The main alternative is to be a consumer, who simply spend it all.

My family member sees capitalism as something like big corporations that advertise and make you spend money on things you don’t need. She sees consumerism and capitalism as basically the same thing, while I see them as complete opposites.

Ok, looks like he was invited in to OpenAIs office for some reason at least https://twitter.com/sama/status/1726345564059832609

It seems the sources are supporters of Sam Altman. I have not seen any indication of this from the boards side.

It seems this was a surprise to almost everyone even at OpenAI, so I don’t think it is evidence that there isn’t much information flow between LW and OpenAI.

There seems to be an edit error after “If I just stepped forward privately, I tell the people I”. If this post wasn’t about the bystander effect, I would just have hoped someone else would have pointed it out!

Corollary: don’t trust yourself!

Most cryptocurrencies have slow transactions. For AI, who think and react much faster than humans the latency would be more of a problem, so I would expect AIs to find a better solution than current cryptocurrencies.

I don’t find it intuitive at all. It would be intuitive if you started by telling a story describing the situation and asked the LLM to continue the story, and you then sampled randomly from the continuations and counted how many of the continuations would lead to a positive resolution of the question. This should be well-calibrated, (assuming the details included in the prompt were representative and that there isn’t a bias of which types of ending the stories are in the training data for the LLM). But this is not what is happing. Instead the model outpu... (read more)

Two possible variations of the game that might be worth experimenting with:

1. Let the adversaries have access to a powerful chess engine. That might make it a better test for what malicious AIs are capable of.
2. Make the randomisation such that there might not be an honest C. For example, if there is 1/4 chance that no player C is honest, each adversary would still think that one of the other adversaries might be honest, so they would want to gain player A’s trust, and hence end up being helpful. I think the player Cs might improve player A’s chances of winni

Why select a deterministic game with complete information for this? I suspect games like poker or backgammon would be easier for the adversarial advisors to fool the player and that these games are a better model of the real world scenario.

This seems like the kind of research that can have a huge impact on capabilities, and much less and indirect impact on alignment/safety. What is your reason for doing it and publishing it?

How about “prediction sites”? Although that could include other things like 538. Not sure if you want to exclude them.

In case you didn’t see the author’s comment below: there is now a patreon button!

Sorry my last comments wasn’t very constructive. I was also confusing two different critisisms:

1. that some changes in predicted probabilities are due to the deadline getting closer and you need to make sure not to claim that as news, and
2. that deadlines are not the in headlines and not always in the graphs either.

About 2): I don’t actually think this is much of a problem, if you ensure that the headline is not misleading and that the information about deadlines is easily available. However if the headline does not contain a deadline, and the deadline is r... (read more)

I think this is a great project! Have you considered adding a donation button or using Patreon to allow readers to support the project?

I do have one big issue with the current way the information is presented: one of the most important things to take into account when making and interpreting predictions is the timeframe of the question. For example, if your are asking about the probability that Putin losses power, they the probability would likely be twice as high if you consider a 2 year timeframe compared to a 1 year time frame, assuming the probability ... (read more)

Shouldn’t you get notification when there are reactions to your post? At least in the batched notification. The urgency/importance of reactions are somewhere between replies, where you get the notification immediately and karma changed, were the default is that it is batched.

Can you only react with -1 of a reaction if someone else has already reacted with the +1 version of the reaction?

Most of the reactions are either positive of negative, but if a comment has several reactions, I find it difficult to see immediately which are positive and which are negative. I’m not sure if this is a disadvantage, because it is slightly harder to get peoples overall valuation of the comment, or if it actually an advantage because you can’t get the pleasure/pain of learning the overall reaction to your comment without first learning the specific reasons for it.

Another issue, if we (as readers of the reactions) tend to group reaction into positive and neg... (read more)

Testing comment. Feel free to react to this however you like, I won’t intrepret the reactions as giving feedback to the comment.

I don't follow the construction. Alice don't know x and S when choosing f. If she is taking the preimage for all 2^n values of x, each with a random S, she will have many overlapping preimages.

I tried and failed to formalize this. Let me sketch the argument, to show where I ran into problems.

Considering a code $c:\{0,1{\}}^k\to \{0,1{\}}^n$ with a corresponding decoding function $d:\{0,1{\}}^n\to \{0,1{\}}^k$, and assume that $m\ll 2^{k/2}$  .

For any function $g:\{0,1{\}}^k\to \{0,1\}$ we can define $f=g\circ d$.  We then choose $g$ randomly from the $2^{2^k}$ such functions. We want to code to be such that for random  $x\in \{0,1{\}}^n$ and random $S\subset \{1,\dots ,n\}$ the information $(x_S,S\}$ is enough to deduce $d\left(x\right)\in \{0,1{\}}^k$, with hi... (read more)

This question is non-trivial even for $s=0$. Here it becomes: let Alice choose a probability $p$ (which has to be on the form $p=\frac{k}{2^n}$ but this is irrelevant for large $n$) and Bob observes the binomially distributed number $B(m,p)$. With which distribution should Alice choose $p$ to maximize the capacity of this channel.  

"STEM-level" is a type error: STEM is not a level, it is a domain. Do you mean STEM at highschool-level? At PhD-level? At the level of all of humanity put together but at 100x speed? 

Seems difficult to mark answers to this question.

The type of replies you get, and the skills you are testing, would also depend how long the subject is spending on the test. Did you have a particular time limit in mind?

This seems to be a copy of an existing one month old post: https://www.lesswrong.com/posts/CvfZrrEokjCu3XHXp/ai-practical-advice-for-the-worried