All of Aaro Salosensaari's Comments + Replies

>It turns out that using Transformers in the autoregressive mode (with output tokens being added back to the input by concatenating the previous input and the new output token, and sending the new versions of the input through the model again and again) results in them emulating dynamics of recurrent neural networks, and that clarifies things a lot...

I'll bite: Could you dumb down the implications of the paper a little bit, what is the difference between a Transformer emulating a RNN and  some pre-Transformer RNNs and/or not-RNN?

My much more novice... (read more)

Epistemic status: I am probably misunderstanding some critical parts of the theory, and I am quite ignorant on technical implementation of prediction markets. But posting this could be useful for my and others' learning. 

First question. Am I understanding correctly how the market would function. Taking your IRT probit market example, here is what I gather:

(1) I want to make a bet on the conditional market P(X_i | Y). I have a visual UI where I slide bars to make a bet on parameters a and b; (dropping subscript i) however, internally this is represente... (read more)

>Glancing back and forth, I keep changing my mind about whether or not I think the messy empirical data is close enough to the prediction from the normal distribution to accept your conclusion, or whether that elbow feature around 1976-80 seems compelling.

I realize you two had a long discussion about this, but my few cents: This kind of situation (eyeballing is not enough to resolve which of two models fit the data better) is exactly the kind of situation for which a concept of statistical inference is very useful.

I'm a bit too busy right now &nbs... (read more)

Hyperbole aside, how many of those experts linked (and/or contributing to the 10% / 2% estimate) have arrived to their conclusion with a thought process that is "downstream" from the thoughtspace the parent commenter thinks suspect? Then it would not qualify as independent evidence or rebuttal, as it is included as the target of criticism.

Thanks. I had read it years ago, but didn't remember that he had many more points than O(n^3.5 log(1/h)) scale and provides useful references (other than Red Plenty).

(I initially thought it would be better not to mention the context of the question as it might bias the responses. OTOH the context could make the marginal LW poster more interested in providing answers, so I here it is:)

It came up in an argument that the difficulty of economic calculation problem could be a difficult to a hypothetical singleton, insomuch a singleton agent needs certain amount of compute relative to the economy in question. My intuition consists two related hypotheses: First, during any transition period where any agent participates in glo... (read more)

Can anyone recommend good reading material on economic calculation problem? 

I found this interesting. Finnish is also language of about 5 million speakers, but we have a commonly used natural translation of "economies of scale" (mittakaavaetu, "benefit of scale"). Any commonplace obvious translation for "Single point of failure" didn't strike my mind, so I googled, and found engineering MSc thesis works and similar documents: the words they choose to use included yksittäinen kriittinen prosessi ("single critical process", most natural one IMO), yksittäinen vikaantumispiste ("single point of failure", literal translation and a bit ... (read more)

"if I were an AGI, then I'd be able to solve this problem" "I can easily imagine"

Doesn't this way of analysis come with a ton of other assumptions left unstated? 

Suppose "I" am an AGI  running on a data center and I can modeled as an agent with some objective function that manifest as desires and I know my instantiation needs electricity and GPUs to continue running. Creating another copy of "I" running in the same data center will use the same resources. Creating another copy in some other data center requires some other data center. ... (read more)

Why wonder when you can think: What is the substantial difference in MuZero (as described in [1]) that makes the algorithm to consider interruptions?

Maybe I show some great ignorance of MDPs, but naively I don't see how an interrupted game could come into play as a signal in the specified implementations of MuZero:

Explicit signals I can't see, because the explicitly specified reward u seems contingent ultimately only on the game state / win condition. 

One can hypothesize an implicit signal could be introduced if algorithm learns to "avoid game states ... (read more)

a backdrop of decades of mistreatment of the Japanese by Western countries.

I find this a bit difficult to take seriously. The WW2 in the Pacific didn't start with well-treatment of China and other countries by Japan, either. Naturally Japanese didn't care about that part of the story, but hey had plenty of other options how they could have responded their the UK or the US trade policy instead of invading Manchuria.

making Ukraine a country with a similar international status to Austria or Finland during the Cold War would be one immediate solution.

This is n... (read more)

First, avoiding arguments from the "other side" on the basis that they might convince you of false things assumes that the other side's belief are in fact false.  

I believe it is less about true/false, but whether you believe the "other side" is making a well-intentioned effort at obtaining and sharing accurate maps of reality. On practical level, I think it is unlikely studying Russian media in detail is useful and cost-effective for a modal LWer. 

Propaganda during wartime, especially during total war, is a prima facia example of situation where... (read more)

Open thread is presumably the best place for a low-effort questions, so here goes: 

I came across this post from 2012: Thoughts on the Singularity Institute (SI) by Holden Karnofsky (then-Co-Executive Director of GiveWell). Interestingly enough, some of the object-level objections (under subtitle "objections") Karnofsky raises[1] are similar to some points that were came up in the Yudkowsky/chathamroom.com discussion and Ngo/Yudkowsky dialogue I read the other day (or rather, read parts of, because they were quite long).

What are people's thought about ... (read more)

Yeah, random internet forum users emailing eminent mathematician en masse would be strange enough to be non-productive. I for one wasn't thinking anyone would to, I don't think it was what OP suggested. To anyone contemplating sending one, the task is best delegated to someone who not only can write coherent research proposals that sound relevant to the person approached, but can write the best one.

Mathematicians receive occasional crank emails about solutions to P ?= NP, so anyone doing the reaching needs to be reputable to get past their crank filters.

A reply to comments showing skepticism about how mathematical skills of someone like Tao could be relevant:

Last time I thought I would understood anything of Tao's blog was around ~2019. Then he was working on curious stuff, like whether he could prove there can be finite-time blow-up singularities in Navier-Stokes fluid equations (coincidentally, solving the famous Millenium prize problem showing non-smooth solution) by constructing a fluid state that both obeys Navier-Stokes and also is Turing complete and ... ugh, maybe I quote the man himself:

[...] one

I have not read Irving either but he is relatively "world-famous" 1970s-1980s author. (In case it helps you to calibrate, his novel The World According To Garp is the kind of book that was published in translation in the prestigious Keltainen Kirjasto series by Finnish publisher Tammi.)

However, I would like make an opposing point about literature and fiction. I was surprised that post author mentioned a work of fiction as a positive example that demonstrates how some commonly argued option is a fabricated one. I'd think literature would at least as often (... (read more)

The picture looks like evidence there is something very weird going on that is not reflected in the numbers or arguments provided. There are homeless encampments in many countries around the world, but very rarely 20 min walk from anyone's office.

From what I remember form my history of Finland classes, the 19th/early 20th century state project to build a compulsory school system met some not insignificant opposition from parents. They liked having the kids working instead going to school, especially in agrarian households.

Now, I don't want to get into debate whether schooling is useful or not (and for whom, and for what purpose, and if the usefulness has changed over time), but there is something illustrative in the opposition: children rarely are independent agents to the extent adults are. If the... (read more)

Genetic algorithms are an old and classic staple of LW. [1]

Genetic algorithms (as used in optimization problems) traditionally assume "full connectivity", that is any two candidates can mate. In other words, population network is assumed to be complete and potential mate is randomly sampled from the population.

Aymeric Vié has a paper out showing (numerical experiments) that some less dense but low average shortest path length network structures appear to result in better optimization results: https://doi.org/10.1145/3449726.3463134

Maybe this isn't news for... (read more)

My take is that the scientific concept of "heritability" has some problems in its construction: the exact definition (Var(genotype)/Var(phenotype)), while useful in some regard, does not match the intuition of the word. 

Maybe the quantity should be called "relative heritability", "heritability relative to population" or "proportion of population variance explained", like many other quantities that similarly have form A/B where both A and B are (population) parameters or their estimates.

Addendum 1.

"Heritable variance"? See also Feldman, Lewontin 1975 https://scholar.google.com/scholar?cluster=10462607332604262282

The smartest people tend to be ambitious.

If this is anecdotal, wouldn't it be easily explained by some sort of selection bias? Smart ambitious people are much visible than smart, definitely-not-ambitious people (and by definition of "smart", they have probably better chances at succeeding in their ambitions than equally ambitious less smart people).

Anecdotally, I have met some relatively smart people who are not very ambitious, and I can imagine there could be much smarter people one does not meet except by random chance, because they do not have muc... (read more)

What is the correct amount of self praise? Do you have reasons to believe Isusr has made an incorrect evaluation regarding their aptitude? Do you believe that even if the evaluation is correct that the post is still harmful?

I don't know if the post is harmful, but in general, "too much self-praise" can be a  failure mode that makes argumentative writing less likely to succeed at convincing readers of its arguments.

The following blog post might be of interest to anyone who either claims Dunning-Kruger means that low-skill people think they are highly skilled or claims Dunning-Kruger is not real: http://haines-lab.com/post/2021-01-10-modeling-classic-effects-dunning-kruger/

The author presents the case how D-K is misunderstood, then why one might suspect it is a mathematical artifact from measurement error, but then shows with a model that there is some evidence for Dunning-Kruger effect, as some observed data are reliably explained with an additive perception bias + n... (read more)

Agreed. The difference is more pronounced in live social situations, and quite easy to quantify in situation such as a proof-heavy mathematics class in college. Many students who have done their work on the problem sets can present a correct solution and if not, usually follow the presented solution. For some, completing the problem sets took more time. Likewise, some people get more out of any spontaneous discussion of the problems. Some relatively rare people would pull out the proofs and points seemingly from thin air: look at the assignment, made some brief notes, and then present their solution intelligibly while talking about it.

However, European Commission seems to defy that rule. The members are nominated by the national governments, yet, they seem not to give unfair advantage to their native countries.

I am uncertain if this is true, or at least, it can be debated. There have been numerous and many complaints of Commission producing decisions and policies that favor some countries.However, such failure mode, if true, is not of the form where individual comissioners favor their native countries, but where the commission as a body adopts stances compatible with overall political p... (read more)

>So the context of this post is less about religion itself, and more about an overall cluster of ways that rationalists/skeptics/etc could still use to improve their own thinking.

At best, this line sounds like arguing that this thing that looks like fish is not a fish because of its evolutionary history, method of giving birth, and it has this funny nose on top of its head through with it breathes makes it a mammal, thus not fish -- in a world where the most salient definition of fish is functional one, "it is a sea-creature that lives in water and we n... (read more)

>And as Duncan is getting at, employment has changed a lot since the term was coined and there's now a lot more opportunity for jobs and work to be aligned with a person's personal goals.

I can agree, I am skeptical that this ...integratedness(?) is actually a good thing for everyone. From point of view of the old "work vs life" people who valued the life part, it probably looks like them losing if what they get is "your work is supposed to integral part of what you choose to do with your life" but the options of where and what kind of work to do are not... (read more)

>The Church of England still has bishops that vote in the house of lords. 

That is argument for particular church-state relationship. The original claim spoke of entanglement (in the present tense!). For reference, the archbishop of Evangelical-Lutheran Church in Finland has always been appointed by whomever is the head of state since Gustav I Vasa embraced the Protestantism and the church was until recently an official state apparatus and to some extent still is. The Holy See has had negligible effect here since centuries, and some historians maint... (read more)

I sort of believe in something like this, except without the magical bits. It motivates me to vote in elections and follow the laws also when there is no effective enforcement. Maybe it is a consequence of reading Pratchett's Discworld novels when I was in impressionable age. 

My mundane explanation (or rationalization) is a bit difficult to write, but I believe it is because of:

>It gets in people's minds.

When people believe something, it affects their behavior. Thus memetic phenomena can have real effects.

As an example I feel is related to this, I ... (read more)

I agree with Phil that this sounds very ... counterintuitive. Usually nothing is free, and even with free things there is consequences or some sort of externality.

However, I recently read an argument by a Finnish finance podcaster, who argued while the intuition might be true and government debt system probably is not sustainable and is going to have some kind of messup in long term, not participating may put your country at disadvantage compared to countries who take the "free" money and invest it, and thus have more assets when it all falls down.

I realize this is a 3mo old comment.

>Nor does China entangle religion with politics to the same extent you find in the Christian and Islamic worlds. This makes it easier to think about conflicts. I feel it produces a better understanding of political theory and strategy.

Does not entangle? I thought China is the only country of note around that enforces their version of Catholic church with Chinese characteristics (the translation used by Wikipedia is "Chinese Patriotic Catholic Church", apparently excommunicated by the pope in Rome). One can discuss how... (read more)

Sure, but statements like

>ANNs are built out of neurons. BNNs are built out of neurons too.

are imprecise and possibly imprecise enough to be also incorrect if it turns out that biological neurons do something different than perceptrons that is important. Without making the exact arguments and presenting evidence in what respects the perceptron model is useful, it is quite easy to bake in conclusions along the lines of "this algorithm for ANNs is a good model of biology" in the assumptions "both are built out of neurons".

Home delivery is way cheaper than it used to be.

I am going to push back a little on this one, and ask for context and numbers? 

As some of my older relatives commented when Wolt became popular here, before people started going to supermarkets, it was common for shops to have a delivery / errand boy (this would have been 1950s, and more prevalent before the WW2). It is one thing that strikes out reading biographies; teenage Harpo Marx dropped out from school and did odd jobs as an errand boy; they are ubiquitous part of the background in Anne Fran... (read more)

Thanks for writing this, the power to weight statistics are quite interesting. I have an another, longer reply with my own take (edit. comments about the graph, that is) in the works, but while writing it, I started to wonder about a tangential question:

I am saying that many common anti-short-timelines arguments are bogus. They need to do much more than just appeal to the complexity/mysteriousness/efficiency of the brain; they need to argue that some property X is both necessary for TAI and not about to be figured out for AI anytime soon, not even after th

Eventually, yes, it is related to arguments concerning people. But I was curious about what aesthetics remain after I try to abstract away the messy details. 

>Is this a closed environment, that supports 100000 cell-generations?

Good question! No. I was envisioning it as a system where a constant population of 100 000 would be viable. (RA pipettes in a constant amount of nutritional fluid every day or something).  Now that you asked the question, it might make more sense to investigate this assumption more.

I have a small intuition pump I am working on, and thought maybe others would find it interesting.

Consider a habitat (say, a Petri dish) that in any given moment has maximum carrying capacity for supporting 100 000 units of life (say, cells), and two alternative scenarios.

Scenario A. Initial population of 2 cells grows exponentially, one cell dying but producing two descendants each generation. After the 16th generation, the habitat overflows, and all cells die in overpopulation. The population experienced a total of 262 142 units of flourishing.

Scenario B... (read more)

I agree the non-IID result is quite surprising. Careful reading of the Berry-Esseen gives some insight on the limit behavior. In the IID case, the approximation error is bounded by constants / $\sqrt{n}$ (where constants are proportional to third moment / $\sigma^3$.

The not-IID generalization for n distinct distribution has the bound more or less sum of third moments divided by (sum of sigma^2)^(3/2) times (sum of third moments), which is surprisingly similar to IID special case. My reading of it suggests that if the sigmas / third moments of all n distrib... (read more)

It gets worse. This isn't a randomly selected example - it's specifically selected as a case where reason would have a hard time noticing when and how it's making things worse.

Well, the history of bringing manioc to Africa is not the only example. Scientific understanding of human nutrition (alongside with disease) had several similar hiccups along the way, several which have been covered in SSC (can't remember the post titles where):

There was a time when Japanese army lost many lives to beriberi during Russo-Japanese war, thinking it was a transmissible d... (read more)

(Reply to gwern's comment but not only addressing gwern.)

Concerning the planning question:

I agree that next-token prediction is consistent with some sort of implicit planning of multiple tokens ahead. I would phrase it a bit differently. Also, "implicit" is doing lot of work here

(Please someone correct me if I say something obviously wrong or silly; I do not know how GPT-3 works, but I will try to say something about how it works after reading some sources [1].)

The bigger point about planning, though, is that the GPTs are getting feedback on

I contend it is not an *implementation* in a meaningful sense of the word. It is more a prose elaboration / expansion of the first generated bullet point list (an inaccurate one: "plan" mentions chopping vegetables, putting them in a fridge and cooking meat; prose version tells of chopping a set of vegetables, skips the fridge and then cooks beef, and then tells an irrelevant story where you go to sleep early and find it is a Sunday and no school).

Mind, substituting abstract category words with sensible more specific ones (vegetables -> carro... (read more)

At the risk of stating very much the very obvious:

Trolley problem (or the fat man variant) is a wrong metaphor for near any ethical decision, anyway, as there are very few real life ethical dilemmas that are as visceral and require immediate action from very few limited set of options and whose consequences are nevertheless as clear.

Here is a couple of a bit more realistic matter of life and death. There are many stories (probably I could find factual accounts, but I am too lazy to search for sources) of soldiers who make the snap decision to save the live... (read more)

"Non-identifiability", by the way, is the search term that does the trick and finds something useful. Please see: Daly et al. [1], section 3. They study indentifiability characteristics of logistic sigmoid (that has rate r and goes from zero to carrying capacity K at t=0..30) via Fisher information matrix (FIM). Quote:

When measurements are taken at times t ≤ 10, the singular vector (which is also the eigenvector corresponding to the single non-zero eigenvalue of the FIM) is oriented in the direction of the growth rate r in parameter spac

Was momentarily confused what is k (sometimes denotes carrying capacity in the logistic population growth model), but apparently it is the step size (in numerical integrator)?

I have not enough expertise here to speak like an expert, but it seems that stiffness would be related in a roundabout way. It seems to describe difficulties of some numerical integrators with systems like this: the integrator can veer much off of true logistic curve with insufficiently small steps because the differential changes fast.

The phenomenon seems to be more about non-sensiti... (read more)

I was going to suggest that maybe it could be a known and published result in dynamical systems / population dynamics literature, but I am unable to find anything with Google, and textbooks I have at hand, while plenty mentions of logistic growth models, do not discuss prediction from partial data before inflection point.

On the other hand, it is fundamentally a variation on the themes of difficulty in model selection with partial data and dangers of extrapolation, which are common in many numerical textbooks.

If anyone wishes to flesh it out, I believe this... (read more)

I am happy that you mention Gelman's book (I am studying it right now). I think lots of "naive strong bayesianists" would improve from a thoughtful study of the BDA book (there are lots of worked out demos and exercises available for it) and maybe some practical application of Bayesian modelling to some real-world statistical problems. The practice of "Bayesian way of life" of "updating my priors" sounds always a bit too easy in contrast to doing a genuine statistical inference.

For example, a couple of puzzles I am still ... (read more)

Howdy. I came across Ole Peters' "ergodicity economics" some time ago, and was interested to see what LW made of it. Apparently one set of skeptical journal club meetup notes: https://www.lesswrong.com/posts/gptXmhJxFiEwuPN98/meetup-notes-ole-peters-on-ergodicity

I am not sure what to make of criticisms of Seattle meetups (they appear correct, but I am not sure if they are relevant; see my comment there).

Not planning to write a proper post, but here is an example blog post of Peters which I found illustrative and demonstrates why I think the ... (read more)

Peters' December 2019 Nature Physics paper (https://www.nature.com/articles/s41567-019-0732-0 ) provides some perspective on 0.6/1.5x coin flip example and other conclusions of the above discussion. (If Peters' claims have changed along the way, I wouldn't know.)

In my reading, there Peters' basic claim is not that ergodicity economics can solve the coin flip game in a way that classical economics can not (because it can, by switching to expected log wealth utility instead of expected wealth), but the utility functions as originally pres... (read more)