Good arguments - notes on Craft of Research chapter 7

Arguments take place in 5 parts.

1. Claim: What do you want me to believe?
2. Reasons: Why should I agree?
3. Evidence: How do you know? Can you back it up?
4. Acknowledgment and Response: But what about ... ?
5. Warrant: How does that follow?

This can be modeled as a conversation with readers, where the reader prompts the writer to taking the next step on the list.

Claim ought to be supported with reasons. Reasons ought to be based on evidence. Arguments are recursive: a part of an argument is an acknowledgment of an anticipated response, and another argument addresses that response. Finally, when the distance between a claim and a reason grows large, we draw connections with something called warrants.

The logic of warrants proceeds in generalities and instances. A general circumstance predictably leads to a general consequence, and if you have an instance of the circumstance you can infer an instance of the consequence.

Arguing in real life papers is complexified from the 5 steps, because

• Claims should be supported by two or more reasons
• A writer can anticipate and address numerous responses. As I mentioned, arguments are recursive, especi

Thinking about a top-level post on FOMO and research taste

• Fear of missing out defined as inability to execute on a project cuz there's a cooler project if you pivot
• but it also gestures at more of a strict negative, where you think your project sucks before you finish it, so you never execute
• was discussing this with a friend: "yeah I mean lesswrong is pretty egregious cuz it sorta promotes this idea of research taste as the ability to tear things down, which can be done armchair"
• I've developed strategies to beat this FOMO and gain more depth and detail with projects (too recent to see returns yet, but getting there) but I also suspect it was nutritious of me to develop discernment about what projects are valuable or not valuable for various threat models and theories of change (in such a way that being a phd student off of lesswrong wouldn't have been as good in crucial ways, tho way better in other ways).
  • but I think the point is you have to turn off this discernment sometimes, unless you want to specialize in telling people why their plans won't work, which I'm more dubious on the value of than I used to be

Idk maybe this shortform is most of the value of the top level post

thoughts on chapter 9 of Craft of Research

Getting the easy things right shows respect for your readers and is the best training for dealing with the hard things.

If they don't believe the evidence, they'll reject the reasons and, with them, your claim.

We saw previously that claims ought to be supported with reasons, and reasons ought to be based on evidence. Now we will look closer at reasons and evidence.

Reasons must be in a clear, logical order. Atomically, readers need to buy each of your reasons, but compositionally they need to buy your logic. Storyboarding is a useful technique for arranging reasons into a logical order: physical arrangements of index cards, or some DAG-like syntax. Here, you can list evidence you have for each reason or, if you're speculating, list the kind of evidence you would need.

When storyboarding, you want to read out the top level reasons as a composite entity without looking at the details (evidence), because you want to make sure the high-level logic makes sense.

Readers will not accept a reason until they see it anchored in what they consider to be a bedrock of established fact. ... To count as evidence, a statement must report something tha

Sources - notes on Craft of Research chapters 5 and 6

Primary, secondary, and tertiary sources

Primary sources provide you with the "raw data" or evidence you will use to develop, test, and ultimately justify your hypothesis or claim. Secondary sources are books, articles, or reports that are based on primary sources and are intended for scholarly or professional audiences. Tertiary sources are books and articles that synthesize and report on secondary sources for general readers, such as textbooks, articles in encyclopedias, and articles in mass-circulation publications.

The distinction between primary and secondary sources comes from 19th century historians, and the idea of tertiary sources came later. The boundaries can be fuzzy, and are certainly dependent on the task at hand.

I want to reason about what these distinctions look like in the alignment community, and whether or not they're important.

The rest of chapter five is about how to use libraries and information technologies, and evaluating sources for relevance and reliability.

Chapter 6 starts off with the kind of thing you should be looking for while you read

Look for creative agreement

• Offer additional support. You can

Yesterday I quit my job for direct work on epistemic public goods! Day one of direct work trial offer is April 4th, and it'll take 6 weeks after that to know if I'm a fulltime hire.

I'm turning down

• raise to 200k/yr usd
• building lots of skills and career capital that would give me immense job security in worlds where investment into one particular blockchain doesn't go entirely to zero
• having fun on the technical challenges

for

• confluence of my skillset and a theory of change that could pay huge dividends in the epistemic public goods space
• 0.35x paycut

did anyone draw up an estimate of how much the proportion of code written by LLMs will increase? or even what the proportion is today

I used to think "community builder" was a personality trait I couldn't switch off, but once I moved to the bay I realized that I was just desperate for serendipity and knew how to take it from 0% to 1%. Since the bay is constantly humming at 70-90% serendipity, I simply lost the urge to contribute.

Benefactors are so over / beneficiaries are so back / etc.

I'm excited for language model interpretability to teach us about the difference between compilers and simulations of compilers. In the sense that chatgpt and I can both predict what a compiler of a suitably popular programming language will do on some input, what's going on there---- surely we're not reimplementing the compiler on our substrate, even in the limit of perfect prediction? Will be an opportunity for a programming language theorist in another year or two of interp progress

Proof cert memo

In the Safeguarded AI programme thesis^[1], proof certificates or certifying algorithms are relied upon in the theory of change. Let's discuss!

From the thesis:

Proof certificates are a quite broad concept, introduced by[33] : a certifying algorithm is defined as one that produces enough metadata about its answer that the answer's correctness can be checked by an algorithm which is so simple that it is easy to understand and to formally verify by hand.

The abstract of citation 33 (McConnell et al)^[2]

A certifying algorithm is an algorithm

Any tips for getting out of a "rise to the occasion mindset" and into a "sink to your training" mindset?

I'm usually optimizing for getting the most out of my A-game bursts. I want to start optimizing for my baseline habits, instead. I should cover the B-, C-, and Z-game; the A-game will cover itself.

Mathaphorically, "rising to the occasion" is taking a max of a max, whereas "sinking to the level of your habits" looks like a greatest lower bound.

Yall, this is a rant. It will be sloppy.

I'm really tired of high functioning super smart "autism" like ok we all have madeup diagnoses--- anyone with a IQ slightly above 90 knows that they can learn the slogans to manipulate gatekeepers to get performance enhancement, and they decide not to if they think theyre performing well enough already. That doesn't mean "ADHD" describes something in the world. Similarly, there's this drift of "autism" getting more and more popular. It's obnoxious because labels and identities are obnoxious, but i only find it repuls... (read more)

For the record, to mods: I waited till after petrov day to answer the poll because my first guess upon receiving a message on petrov day asking me to click something is that I'm being socially engineered. Clicking the next day felt pretty safe.

messy, jotting down notes:

• I saw this thread https://twitter.com/alexschbrt/status/1666114027305725953 which my housemate had been warning me about for years.
• failure mode can be understood as trying to aristotle the problem, lack of experimentation
• thinking about the nanotech ASI threat model, where it solves nanotech overnight and deploys adversarial proteins in all the bloodstreams of all the lifeforms.
• These are sometimes justified by Drexler's inside view of boundary conditions and physical limits.
• But to dodge the aristotle problem, there would have

Methods, famously, includes the line "I am a descendant of the line of Bacon", tracing empiricism to either Roger (13th century) or Francis (16th century) (unclear which).

Though a cursory wikiing shows an 11th century figure providing precedents for empiricism! Alhazen or Ibn al-Haytham worked mostly optics apparently but had some meta-level writings about the scientific method itself. I found this shockingly excellent quote

The duty of the man who investigates the writings of scientists, if learning the truth is his goal, is to make himself an enemy of a

New discord server dedicated to multi-multi delegation research

DM me for invite if you're at all interested in multipolar scenarios, cooperative AI, ARCHES, social applications & governance, computational social choice, heterogeneous takeoff, etc.

(side note I'm also working on figuring out what unipolar worlds and/or homogeneous takeoff worlds imply for MMD research).

Questions and Problems - thoughts on chapter 4 of Craft of Doing Research

Last time we discussed the difference between information and a question or a problem, and I suggested that the novelty-satisfied mode of information presentation isn't as good as addressing actual questions or problems. In chapter 3 which I have not typed up thoughts about, A three step procedure is introduced

1. Topic: "I am studying ..."
2. Question: "... because I want to find out what/why/how ..."
3. Significance: "... to help my reader understand ..." As we elaborate on the different k

talk to friends as a half measure

When it comes to your internal track record, it is often said that finding what you wrote at time t-k beats trying to remember what you thought at t-k. However, the activation energy to keep such a journal is kinda a hurdle (which is why products like https://fatebook.io are so good!).

I find that a nice midpoint between the full and correct internal track record practices (rigorous journaling) and completely winging it (leaving yourself open to mistakes and self delusion) is talking to friends, because I think my memory of... (read more)

Guaranteed Safe AI paper club meets again this thursday

Event for the paper club: https://calendar.app.google/2a11YNXUFwzHbT3TA

blurb about the paper in last month's newsletter:

... If you’re wondering why you just read all that, here’s the juice: often in GSAI position papers there’ll be some reference to expectations that capture “harm” or “safety”. Preexpectations and postexpectations with respect to particular pairs of programs could be a great way to cash this out, cuz we could look at programs as interventions and simulate RCTs (labeling one program

Yoshua Bengio is giving a talk online tomorrow https://lu.ma/4ylbvs75

GSAI paper club is tomorrow (gcal ticket), summary (by me) and discussion of this paper

i'm getting back into composing and arranging. send me rat poems to set to music!

preorders as the barest vocabulary for emergence

We can say "a monotonic map, $\Phi \in mono\left(Q^P\right)$ is a phenomenon of $P$ as observed by $Q$", then, emergence is simply the impreservation of joins.

Given preorders $(P,{\le }_P)$ and $(Q,{\le }_Q)$, we say a map in $mono\left(Q^P\right)$ "preserves" joins (which, recall, are least upper bounds) iff $\forall ab\in P,\Phi a{\vee }_Q\Phi b=\Phi \left(a{\vee }_{P}b\right)$ where by "$x=y$" we mean $x\le y\wedge y\le x$.

Suppose $\Phi$ is a measurement taken from a particle. We would like for our measurement system to be robust against emergence, which is literally operationalized by measuring one particle, measuring another, t... (read more)

Jotted down some notes about the law of mad science on the EA Forum. Looks like some pretty interesting open problems in the global priorities, xrisk strategy space. https://forum.effectivealtruism.org/posts/r5GbSZ7dcb6nbuWch/quinn-s-shortform?commentId=DqSh6ifdXpwHgXnCG

Ambition, romance, kids

Two premises of mine are that I'm more ambitious than nearly everyone I meet in meatspace and normal distributions. This implies that in any relationship, I should expect to be the more ambitious one.

I do aspire to be a nagging voice increasing the ambitions of all my friends. I literally break the ice with acquaintances by asking "how's your master plan going?" because I try to create vibes like we're having coffee in the hallway of a supervillain conference, and I like to also ask "what harder project is your current project a war... (read more)

Positive and negative longtermism

I'm not aware of a literature or a dialogue on what I think is a very crucial divide in longtermism.

In this shortform, I'm going to take a polarity approach. I'm going to bring each pole to it's extreme, probably each beyond positions that are actually held, because I think median longtermism or the longtermism described in the Precipice is a kind of average of the two.

Negative longtermism is saying "let's not let some bad stuff happen", namely extinction. It wants to preserve. If nothing gets better for the poor or the an... (read more)

The audience models of research - thoughts on Craft of Doing Research chapter 2

Writers can't avoid creating some role for themselves and their readers, planned or not

Before considering the role you're creating for your reader, consider the role you're creating for yourself. Your broad options are the following

1. I've found some new and interesting information - I have information for you
2. I've found a solution to an important practical problem - I can help you fix a problem
3. I've found an answer to an important question - I can help you understand somethi

He had become so caught up in building sentences that he had almost forgotten the barbaric days when thinking was like a splash of color landing on a page.

Edward St Aubyn

claude and chatgpt are pretty good at ingesting textbooks and papers and making org-drill cards.

here's my system prompt https://chat.openai.com/g/g-rgeaNP1lO-org-drill-card-creator though i usually tune it a little further per session.

Here are takes on the idea from the anki ecosystem

• https://ankiweb.net/shared/info/1915225457
• https://ankigpt.help/

I tried a little ankigpt and it was fine, i haven't tried the direct plugin from ankiweb. I'm opting for org-drill here cuz I really like plaintext.

consider how our nonconstructive existence proof of nash equilibria creates an algorithmic search problem, which we then study with computational complexity. For example, 2-player 0-sum games are P but for three or more players general sum games are NP-hard. I wonder if every nonconstructive existence proof is like this? In the sense of inducing a computational complexity exercise to find what class it's in, before coming up with greedy heuristics to accomplish an approximate example in practice.

I like thinking about "what it feels like to write computer programs if you're a transformer".

Does anyone have a sense of how to benchmark or falsify Nostalgebraist's post on the subject?

Quick version of conversations I keep having, might be worth a top level effortpost.

A prediction market platform giving granular permission systems would open up many use cases for many people

whistleblower protections at large firms, dating, project management and internal company politics--- all userbases with underserved opinions about transparency. Manifold could pivot to this but have a lot of other stuff they could do instead.

Think about slack admins are confused about how to prevent some usergroups from @channel and discord admins aren't.

what are your obnoxious price systems for tutoring?

There's a somewhat niche CS subtopic that a friend wants to learn, I'm really well positioned to teach her. More discussion on the manifold bounty:

There's a remarkable TNG episode about enfeeblement and paul-based threatmodels, if I recall correctly.

There's a post-scarcity planet with some sort of Engine of Prosperity in the townsquare, and it doesn't require maintenance for enough generations that engineering itself is a lost oral tradition. Then it starts showing signs of wear and tear...

If paul was writing this story, they would die. I think in the actual episode, there's a disagreeable autistic teenager who expresses curiosity about the Engine mechanisms, and the grownups basically shame him, lik... (read more)

We need a cool one-word snappy thing to say for "just what do you think you know and how do you think you know it" or like "I'm requesting more background about this belief you've stated, if you have time".

I want something that has the same mouthfeel as "roll to disbelieve" for this.

Is there an EV monad? I'm inclined to think there is not, because EV(EV(X)) is a way simpler structure than a "flatmap" analogue.

critiques and complaints

I think one of the most crucial meta skills i've developed is honing my sense of who's criticizing me vs. who's complaining.

A criticism is actionable, implicitly often it's from someone who wants you to win. A complaint is when you can't figure out how you'd actionably fix something or improve based on what you're being told.

This simple binary story is problematic. It can empower you to ignore criticism you don't like by providing a set of excuses, if you're not careful. Sometimes it's operationally impossible to parse out a critic... (read more)

hmu for a haskell job in decentralized finance. Super fun zero knowledge proof stuff, great earning to give opportunity.

capabilities-prone research.

I come to you with a dollar I want to spend on AI. You can allocate p pennies to go to capabilities and 100-p pennies to go to alignment, but only if you know of a project that realizes that allocation. For example, we might think that GAN research sets p = 98 (providing 2 cents to alignment) while interpretability research sets p = 10 (providing 90 cents to alignment).

Is this remotely useful? This is a really rough model (you might think it's more of a venn diagram and that this model doesn't provide a way of reasoning about t... (read more)

Question your argument as your readers will - thoughts on chapter 10 of Craft of Research

Three predictable disagreements are

• There are causes in addition to the one you claim
• What about these counterexamples?
• I don't define X as you do, to me X means...

There are roughly two kinds of queries readers will have about your argument

1. intrinsic soundness - "challenging the clarity of a claim, relevance of reasons, or quality of evidence"
2. extrinsic soundness - "different ways of framing the problem, evidence you've overlooked, or what others have written on t