All of ZY's Comments + Replies

Out of curiosity - what was the time span for this raise that achieved this goal/when did first start again? Was it 2 months ago?

3habryka
Yep, when the fundraising post went live, i.e. November 29th. 

A few thoughts from my political science classes and experience - 

when people value authority more than arguments

It's probably less about "authority", but more about the desperate hope to reach stability, and the belief of unstable governments leading to instability, after many years of being colonized on the coasts, and war (ww 2 + civil war).

"Societies can be different" 

is a way too compressed term to summarize the points you made. Some of them are political ideology issues, and others are resource issues, but not related to "culture" as could ... (read more)

For "prison sentencing" here, do you mean some time in prison, but not life sentencing? Also instead of prison sentencing, after increasing "reliability of being caught", would you propose alternative form of sentencing?

Some parts of 1) and most of 2) made me feel educating people on the clear consequences of the crime is important.

For people who frequently go in and out of prison - I would guess most legal systems already make it more severe than previous offenses typically, but for small crimes they may not be.

I do think other types of punishments that you have listed there (physical pain, training programs, etc) would be interesting depending on the crime.

how to punish fewer people in the first place

This seems to be hard when actual crimes (murder, violent crimes, etc.) are committed; seems to be good to figure out why they commit the crimes, and reducing that reason in the first place is more fundamental.

A side note - 

We don’t own slaves, women can drive, while they couldn’t in Ancient Rome, and so on.

Seems to be a very low bar for being "civilized"

focusing less on intent and more on patterns of harm

In a general context, understanding intent though will help to solve the issue fundamentally. There might be two general reasons behind harmful behaviors: 1.do not know this will cause harm, or how not to cause harm, aka uneducated on this behavior/being ignorant, 2.do know this will cause harm, and still decided to do so. There might be more nuances but these two are probably the two high level categories. Knowing what the intent is helps to create strategies to address the issue - 1.more education? 2.more punishments/legal actions?

In my opinion, theoretically, the key to have "safe" humans and "safe" models, is "to do no harm" under any circumstances, even when they have power. This is roughly what law is about, and what moral values should be about (in my opinion)

Yeah nice; I heard youtube also has something similar for checking videos as well

It is interesting; I am only a half musician but I wonder what a true musician think about the music generation quality generally; also this reminds me of the Silicon Valley show's music similarity tool to check for copyright issues; that might be really useful nowadays lmao

1yams
A great many tools like this already exist and are contracted by the major labels. When you post a song to streaming services, it’s checked against the entire major label catalog before actually listing on the service (the technical process is almost certainly not literally this, but it’s something like this, and they’re very secretive about what’s actually happening under the hood).

On the side - could you elaborate why you think "relu better than sigmoid" is a "weird trick", if that is implied by this question?

The reason that I thought to be commonly agreed is that it helps with the vanishing gradient problem (this could be shown from the graphs).

I personally agree with your reflection on suffering risks (including factory farming, systemic injustices, and wars) and the approach to donating to different cause areas. My (maybe unpopular under "prioritizing only 1" type of mindset) thought is: maybe we should avoid prioritizing only one single area (especially collectively), but recognize that in reality there are always multiple issues we need to fight about/solve. Personally we could focus professionally on one issue, and volunteer for/donate to another cause area, depending on our knowledge, inter... (read more)

Yeah that makes sense; the knowledge should still be there, just need to re-shift the distribution "back"

Haven't looked too closely at this, but my initial two thoughts:

  1. child consent is tricky.
  2. likely many are foreign children, which may or may not be in the 75 million statistic

It is good to think critically, but I think it would be beneficial to present more evidence before making the claim or conclusion

This is very interesting, and thanks for sharing. 

  • One thing that jumps out at me is they used an instruction format to prompt base models, which isn't typically the way to evaluate base models. It should be reformatted to a completion type of task. If this is redone, I wonder if the performance of the base model will also increase, and maybe that could isolate the effect further to just RLHF.
  • I wonder if this has anything to do with also the number of datasets added on by RLHF (assuming a model go through supervised/instruction finetuning first, and th
... (read more)
2p.b.
There was one comment on twitter that the RLHF-finetuned models also still have the ability to play chess pretty well, just their input/output-formatting made it impossible for them to access this ability (or something along these lines). But apparently it can be recovered with a little finetuning. 

I find it useful sometimes to think about "how to differentiate this term" when defining a term. In this case, in my mind it would be thinking about "reasoning", vs "general reasoning" vs "generalization".

  • Reasoning: narrower than general reasoning, probably would be your first two bullet points combined in my opinion
  • Generalization: even more general than general reasoning (does not need to be focused on reasoning). Seems could be the last two bullet points you have, particularly the third
  • General reasoning (this is not fully thought through): Now that we ta
... (read more)
3eggsyntax
Interesting approach, thanks!

In my observation (trying to avoid I think!), "I think" is intended to (or actually should have been used to) point out perspective differences (which helps to lead to more accurate conclusions, including collaborative and effective communication), rather than confidence. In the latter case of misuse, it would be good if people clarify "this term is about confidence, not perspective in my sentence". 

True. I wonder for the average people, if being self-aware would at least unconsciously be a partial "blocker" on the next malevolence action they might do, and that may evolve across time too (even if it may take a bit longer than a mostly-good)

I highly agree with almost all of these points, and those are very consistent with my observation. As I am still relatively new to lesswrong, one big observation (based on my experience) I still see today, is disconnected concepts, definitions, and or terminologies with the academic language. Sometimes I see terminology that already exists in academia and introducing new concepts with the same name may be confusing without using channels academics are used to. There are some terms that I try to search on google for example, but the only relevant ones are f... (read more)

What would be some concrete examples/areas to work on for human flourishing? (Just saw a similar question on the definition; I wonder what could be some concrete areas or examples)

True; and they would only need to merge up to they reach a "swing state" type of voting distribution.

That would be interesting; on the other hand, why not just merge all the states? I guess it would be a more dramatic change and may be harder to execute and unnecessary in this case.

7Adam Scherlis
It stops being in the interests of CATXOKLA to invite more states once they're already big enough to dominate national electoral politics.

Yes, what I meant is exactly "there is no must, but only want". But it feels like a "must" in some context that I am seeing, but I do not recall exactly where. And yeah true, there may be some survival bias.

I agree it is tragedy from human race's perspective, but I think what I meant is from a non-human perspective to view this problem. For example, to an alien who is observing earth, human is just another species that rise up as a dominant species, as a thought experiment. 

(On humans prefer to be childless - actually this already slowed down in many countries due to cost of raising a child etc, but yeah this is a digress on my part.)

My two cents:

  1. The system has a fixed goal that it capably works towards across all contexts.
  2. The system is able to capably work towards goals, but which it does, if any, may depend on the context.

From these two above, seems it would be good for you to define/clarify what exactly you mean by "goals". I can see two definitions: 1. goals as in a loss function or objective that the algorithm is optimizing towards, 2. task specific goals like summarize an article, planning. There may be some other goals that I am unaware of, or this is obvious elsewhere in some c... (read more)

I think that is probably not a good reason to be libertarian in my opinion? Could you also share maybe how much older were your than you siblings? If you are not that far apart, you and your siblings came from the same starting line, distributing is not going to happen in real life economically nor socially even if not libertarian (in real life, where we need equity is when the starting line is not the same and is not able to be changed by choice. A more similar analogy might be some kids are born with large ears, and large ears are favored by the society, and the large eared kids always get more candy). If you are ages apart with you being a lot older, it may make some limited sense to for your parents to re-distribute.

3johnswentworth
To be clear, I don't really think of myself as libertarian these days, though I guess it'd probably look that way if you just gave me a political alignment quiz. To answer your question: I'm two years older than my brother, who is two years older than my sister.

I am not quite sure about the writing/examples in computational kindness and responsibility offloading, but I think I feel the general idea. 

For computational kindness, I think it is just really the difference in how people prefer to communicate, or making plans it seems, with the example on trip planning. I, for example, personally prefer being offered with their true thoughts - if they are okay with just really anything, or not. Anything is fine as long as that is what they really think or prefer (side talk: I generally think communicating real pref... (read more)

Ah thanks. Do you know why these former rationalists were "more accepting" of irrational thinking? And to be extremely clear, does "irrational" here mean not following one's preference with their actions, and not truth seeking when forming beliefs?

I don't understand either. If it is meant what it meant, this is a very biased perception and not very rational (truth seeking or causality seeking). There should be better education systems to fix that.

On what evidence do I conclude what I think is know is correct/factual/true and how strong is that evidence? To what extent have I verified that view and just how extensively should I verify the evidence?


For this, aside from traditional paper reading from credible sources, one good approach in my opinion is to actively seek evidence/arguments from, or initiate conversations with people who have a different perspective with me (on both side of the spectrum if the conclusion space is continuous). 

I am interested in learning more about this, but not sure what "woo" means; after googling, is it right to interpret as "unconventional beliefs" of some sort?

4gilch
It's short for "woo-woo", a derogatory term skeptics use for magical thinking. I think the word originates as onomatopoeia from the haunting woo-woo Theremin sounds played in black-and-white horror films when the ghost was about to appear. It's what the "supernatural" sounds like, I guess. It's not about the belief being unconventional as much as it being irrational. Just because we don't understand how something works doesn't mean it doesn't work (it just probably doesn't), but we can still call your reasons for thinking so invalid. A classic skeptic might dismiss anything associated categorically, but rationalists judge by the preponderance of the evidence. Some superstitions are valid. Prescientific cultures may still have learned true things, even if they can't express them well to outsiders.

I personally agree with you on the importance of these problems. But I myself might also be a more general responsible/trustworthy AI person, and I care about other issues outside of AI too, so not sure about a more specific community, or what the definition is for "AI Safety" people.

For funding, I am not very familiar and want to ask for some clarification: by "(especially cyber-and bio-)security", do you mean generally, or "(especially cyber-and bio-)security" caused by AI specifically?

Does "highest status" here mean highest expertise in a domain generally agreed by people in that domain, and/or education level, and/or privileged schools, and/or from more economically powerful countries etc? It is also good to note that sometimes the "status" is dynamic, and may or may not imply anything causal with their decision making or choice on priorities.

One scenario is "higher status" might correlates with better resources to achieve those statuses, and a possibility is as a result they haven't experienced or they are not subject to many near-ter... (read more)

1davekasten
I mean, functionally all of those things.  (Well, minus the country dynamic.  Everyone at this event I talked to was US, UK, or Canadian, which is all sorta one team for purposes of status dynamics at that event)

Could you define what you mean by "correctness" in this context? I think there might be some nuances into this, in terms of what "correct" means, and under what context

Based on the words from this post alone -

I think that would depend on what the situation is; in the scenario of price increases, if the business is a monopoly or have very high market power, and the increase is significant (and may even potentially cause harm), then anger would make sense. 

Thanks! I think the term duration is interesting and creative. 

Do you think for the short-term ones there might be pre-studies they need to do for the exact topics they need to learn on? Or maybe could design the short-term ones for topics that can be learnt quickly and solved quickly? I am a little worried about the consistency in policy as well (for example even with work, when a person on a project take vacation, and someone need to cover for them, there are a lot of onboarding docs, and prior knowledge to transfer), but could not find a good way just yet. I will think more about these.

Amazingly detailed article covering malevolence, interaction with power, and the other nuances! Have been thinking of exploring similar topics, and found this very helpful. Besides the identified research questions, some of which I highly agree with, one additional question I was wondering is: do self-awareness of one's own malevolence factors help one to limit the malevolence factors? if so how effective would that be? how would this change when they have power? 

3Viliam
Probably the effect would be nonlinear, like the evil people would just laugh, the average might get depressed and give up, and the mostly-good would strive to achieve perfection (or conclude that they are already good enough compared to others, and relax their efforts?).

Interesting idea, and I think there is a possibility that the responsibility will make the "normal people" make better choices or learn more even though they do not know policy, etc in the first place. 

A few questions:

  • Do you think there is a situation where selected random people do not want to be in office/leadership and want to pursue their own passion/career and thus due to this reason may do a bad job? Is this mandatory?
  • What are some nuances about population and diversity? (I am not sure yet)
2John Huang
>Do you think there is a situation where selected random people do not want to be in office/leadership and want to pursue their own passion/career and thus due to this reason may do a bad job? Is this mandatory? I think a robust way to design the assembly (or multiple assemblies like with Bouricius's model) is to have many different people serving different term lengths. Some people may serve a term of only a couple days or weeks. Others might serve for years.  For short-term service, I would make that mandatory. Everyone is required to come.  For long term service, maybe those should be voluntary.  As far as incentives go, there's a range of enforcement options for "mandatory" service. Perhaps you can just pay a big fine, as a percentage of your income, as an alternative to service. There probably ought to be mechanisms to defer service so you can time things a bit better with your life circumstances.  The typical Citizens' Assembly will also offer benefits such as child care, parental care.  A high paying salary will encourage the lower and middle class to participate.  I have trouble coming up with ways to help small business owners to participate though. Could a small business owner drop their work for an entire year, even if it was well paid -- especially if the small business is so small there are no managers to cover their role? Perhaps there could be alternatives for them, such as part time work coupled with work-from-home.    >What are some nuances about population and diversity? (I am not sure yet)   I have yet to hear about a case where Deliberative decision making techniques were tried and failed due to excessive diversity or cultural factors. I'm not an expert on the latest and greatest research here so I may be wrong. I do know that deliberation experiments have been performed all around the world, including East Asia, Africa, and India.  An example deliberative poll was performed in Uganda, paper linked here: https://direct.mit.edu/daed/a

Could you maybe elaborate on "long term academic performance"?

Agree with this, and wanted to add that I am also not completely sure if mechanistic interpretability is a good "commercial bet" yet based on my experience and understanding, with my definition of commercial bet being materialization of revenue or simply revenue generating. 

One revenue generating path I can see for LLMs is the company uses them to identify data that are most effective for particular benchmarks, but my current understanding (correct me if I am wrong) is that it is relatively costly to first research a reliable method, and then run inte... (read more)

Would agree with most of the posts; To me, humans have some general shared experiences that may activate empathy related to those experiences, but the the numerous small differences in experience make it very hard to know exactly what the others would think/feel, even if in exactly the same situations.  We could never really model the entire learning/experience history from another person. 

My belief/additional point I want to add/urge is that this should not be interpreted as say empathy is not needed because we don't get it right anyways (I saw ... (read more)

I think I observe this generally a lot: "as soon as those implications do not personally benefit them", and even more so when this comes with a cost/conflict of interest.

On rationality on decision making (not the seeking truth part on belief forming I guess) - I thought it is more like being consistent with their own preference and values (if we are constraining to the definition on lesswrong/sequence ish)? I have a hot take that:

  1. If the action space of commit to a belief is a binary choice, then when people do not commit to a belief, the degree they believ
... (read more)

I think the title could be a bit more specific like - "involving political party in science discussions might not be productive", or something similar.  If using the word "politics", it would be crucial to define what "politics" here mean or refer to. The reason I say this is "politics" might not be just about actual political party's power dynamics, but also includes general policy making, strategies, and history that aim to help individuals in the society, and many other aspects. These other types of things included in the word "politics" is crucial... (read more)

I think by winning, he meant: "art of choosing actions that lead to outcomes ranked higher in your preferences", though I don't completely agree with this word choice of "winning" which could be ambiguous/causing confusion.

A bit unrelated, but more of a general comment on this - in my belief, I think people generally have unconscious preferences, and knowing/acknowledging these before weighing out preferences are very important, even if some preferences are short term.

I also had similar feelings on the simplicity part, and also how theory/idealized situation and execution could be very different.  Also agree on the conflict part (and to me many different type of conflicts).  And, I super super strongly support the section on The humans behind the numbers.
(These thoughts still persist after taking intro to EA courses). 

I think EA's big overall intentions are good to me and I am happy/energized by see how passionate people are comparing to no altruism at all at least; but the details/execution are not quite there to me.

I have been having some similar thoughts on the main points here for a while and thanks for this.

I guess to me what needs attention is when people do things along the lines of "benefit themselves and harm other people". That harm has a pretty strict definition,  though I know we may always be able to give borderline examples. This definitely includes the abuse of power in our current society and culture, and any current risks etc. (For example, if we are constraining to just AI with warning on content, https://www.iwf.org.uk/media/q4zll2ya/iwf-ai-csam... (read more)

This is about basic human dignity and respect to other humans, and has nothing to do with politics.

Answer by ZY61

Oxford languages (or really just after googling) says "rational" is "based on or in accordance with reason or logic."

I think there are a lot of other types of definitions (I think lesswrong mentioned it is related to the process of finding truth). For me, first of all it is useful to break this down into two parts: 1) observation and information analysis, and 2) decision making.

For 1): Truth, but also particularly causality finding. (Very close to the first one you bolded, and I somehow feel many other ones are just derived from this one. I added causality... (read more)

From my perspective - would say it's 7 and 9.

For 7: One AI risk controversy is we do not know/see existing model that pose that risk yet. But there might be models that the frontier companies such as Google may be developing privately, and Hinton maybe saw more there.

For 9: Expert opinions are important and adds credibility generally as the question of how/why AI risks can emerge is by root highly technical. It is important to understand the fundamentals of the learning algorithms. Additionally they might have seen more algorithms. This is important to me ... (read more)

(Like the answer on declarative vs procedural). Additionally, reflecting on practicing Hanon for piano (which is almost a pure finger strength/flexibility type of practice) - might be also for physical muscle development and control.

Agree with a lot of the things in this post, including "But implicit in that is the assumption that all DALYs are equal, or that disability or health effects are the only factors that we need to adjust for while assessing the value of a life year. However, If DALYs vary significantly in quality (as I’ll argue and GiveWell acknowledges we have substantial evidence for), then simply minimizing the cost of buying a DALY risks adverse selection. "

Had the same question/thoughts when I did the Introduction to Effective Altruism course as well. 

Load More