Jessica Taylor. CS undergrad and Master's at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
Ah sorry. Edited.
With 10K lines it's not impressive to write 10K lines. I'm thinking of writing software that would be 10K lines if done by humans.
"bug-free" & "reliably" is a high bar!
I think the idea of paying attention to "human values" and "alignment" was based on a philosophical assumption that moral realism would fail, that value is relative / two-place predicate. I am thinking perhaps a more disjunctive approach would be to imagine an AI that in the case of moral realism does the normative thing and in the case of moral non-realism is aligned with humans / humanity / something Earth-local. This assumes there is a fact of the matter about whether moral realism is true. Also, not everyone will like it even if moral realism is true (though not everyone liking something is a usual state of affairs). Even framing it disjunctively like this is perhaps philosophically loaded / hard to define without better concepts. See also Moral Reality Check short story about what if AIs believe moral realism.
Really appreciate the exchange, helped me clarify my thoughts.
As a brief note for potential future exploration (no need to reply if you don't want to):
A crux seems to be that with chroma world, I think experience is of difference between chroma, assuming chromatic realism (real quiddites). Whereas the Russelian monist wants the experience to be of identity of internal chroma. This is perhaps a disagreement on the phenomenology / epistemology relationship.
I think the analogy with Newtonian mechanics (if absolute positions are real, then experience is of relative positions, even if located at absolute positions) is pretty strong; chroma world "just" has different physical symmetries.
On my end I think I could continue this line of exploration by studying Deleuze (difference as more ontologically/metaphysically primary than identity), Dennett (more clarity on whether I agree/disagree with his basic picture), and Russell as a primary source.
Yeah I think it might be more productive, if I wanted to make progress on this, to look at the math rather than the metaphysical back-and-forth.
I feel like the single particle universe clearly instantiates the logical system: ∃x (particle(x)).
Nope, because they didn't write down the system. (This might be quibbling. I mean it doesn't write down that formal system anywhere.)
Let me move to a different example, why do you think our universe instantiates PA
Because mathematicians have written down the rules of PA. And they have programmed computers to use PA.
Do you think that our universe then either instantiates Con(ZFC) or doesn't?
Our universe instantiates formal system such as ZFC + Con(ZFC), because mathematicians can write them down. (It also instantiates inconsistent formal systems such as "ZFC + axiom of choice is false".)
Backtracking a bit. I reject axiom 6 (which is not to say I actively disbelieve it, but that I don't actively believe it) and I'm giving an idea of why. If you could logically prove axiom 6, you wouldn't need it to be an axiom. So you're going to have some trouble convincing me of it on logical grounds. Rather it's more metaphysical and so on.
Let us say that PA consists of a formal system with exactly the standard axioms of PA. There is some quibbling about which "paper copies" work or which "translations" work. But PA only exists in a universe if it is instantiated as physical information. Instantiating large numbers does not instantiate PA.
Okay, noting that you made a stronger claim, that there is a universe where there is no sound logical system (i.e. P and not P are true in some possible universe), but I'm happy to move to this, sure.
Not what I meant. Two interpretations (a) in the single-particle universe, no logical system, sound or unsound, is instantiated, because no non-trivial physical info is instantiated. (b) Perhaps we could say that PA's prefect-generating property is stronger than soundness. It is soundness plus "having provably distinct terms that number in the countable infinity".
Imagine a universe that consists of a single particle. The law is that the particle stays in the same state always. Actually, the particle only has 1 possible state in this universe. So it's just always in the same state as a matter of math not just physics.
There is no way for PA to exist in this universe.
"PA is possible" sure ok.
"PA has a perfect generating property" let is suppose this. for example, maybe there is some list of perfect properties of PA, and they all follow from some property of it.
"It is possible for that perfect-generating property to be instantiated (it is instantiated in Peano Arithmetic)" sure, ok
"Therefore it is possible for something to have the property of perfect-essential necessity." absolutely wild claim.
"We say that something has the property of “perfect-essential necessity” if (and only if) whenever there is a perfect-generating property for that thing, it implies that there necessarily must exist something (i.e. in every world) which instantiates that perfect-generating property."
Seems like a wild claim. For PA to have this property would mean that its perfect-generating property (perhaps logical consistency, or soundness?) has a corresponding object having that property (e.g. soundness) in every possible world. But that's a wild claim. Maybe there's a possible universe with so little information that no sound logical system exists there.
Here's an evalution claiming it isn't even true at Anthropic in terms of lines of code (I haven't checked independently)
The kind of thing I don't want to count is that, for example, it would be easy to write a script that generates many many lines of code. It's just not interesting to count "how are most lines of code written" globally. "Code actually merged at particular big tech companies" is much closer and I expect that was less than 90 percent at the time.