Title: A Particular Equilibrium Date: 2023-07-02 19:39 Modified: 2023-02-08 12:00 Category: Maths Authors: Algon Summary: Normally Sampled Normal Lead to A Dirac Delta Distribution? Part 1

I had a dumb idea while I was looking at how to model SGD with a SDE. If you randomly initliaze matrix weights according to some Gaussian, , where I is an identity matrix, then the first gradient updates should be normally distributed to, at least for loss.

So what if that kept happening? Like, you sample from one normal distribution to form another normal distribution. And so on. This is plausibly something which can occur, like difference equations with I.I.D. brownian noise. Though the exact form I'm thinking of is kind of dumb.

Suppose you have a sequence of random variables. We define

Though really, will be sampled from a half-Gaussian. But I don't want to write that out, so whatever.

How does a sequence of jumps evolve? Well, intuitively there is a ~2/3 chance of being smaller than . If this occurs infinitely many times, we might think the "event" of the variance growing smaller will occur infinitely many times. So if you're doing the obvious thing if you're not a mathematician and ask "what's the stationary distribution", you might suspect we'll get a Dirac Delta.

Yet the "event" of the variance growing larger must also occur infinitely many times in any sequence. So, sadly, we need more than infinitely many shrinks to solve our problem. Well, let's just do a quick numerical check that eq. 1 leads to an infinitely shrinking sequence.

Running a bit of code, we get a sequence like

  • 0.06619689276821521,
  • -0.004180368466071496,
  • -3.632035781799425e-06,
  • -1.780514134591378e-11,
  • -3.809091727146634e-22,
  • -3.858858262617346e-44,
  • 1.3234421571769518e-87,
  • 7.107311345616059e-175,
  • 0.0,
  • 0.0

So it looks like we're on the right track.

What now? Well, since we're interested in growth, we might reasonably ask what is. A quick calculation shows

So in expectation, is smaller than . What do we do for ? Well, we have If we could somehow break that product of factors of successive into a sum, then we'd be closer to re-using eq. 4. Luckily, the arithmetic-geometric mean inequality, and Jensen's inequality, will let us do that[1].

It seems pretty likely Generalizing the above arguement to show is left as an exercise to the reader.

Eq. 12 implies our sequence is bounded above almost surely. Moreover, if there is some kind of limiting distribution for as , it would have to have mean 0. But since we've restricted to be in $ [ 0, \infty )$, it would have to have zero measure over the positive reals.

That doesn't imply, though, that the operator K (eq. 13) doesn't have a fixed point Or that it does. Only that, if this equilibrium is the limit of some jump process, it must be a Dirac Delta.


  1. Don't worry about the . It just means whatever mathematical theory we're working in, plus some assumed equations, can deduce the statement to the right. ↩︎

New Comment