I have originally developed a machine learning notion which I call an LSRDR ($L_{2,d}$

-spectral radius dimensionality reduction), and LSRDRs (and similar machine learning models) behave mathematically and they have a high level of interpretability which should be good for AI safety. Here, I am giving an example of how LSRDRs behave mathematically and how one can get the most out of interpreting an LSRDR.

Suppose that $n$ is a natural number. Let $N$ denote the quantum channel that takes an $n$ qubit quantum state and selects one of those qubits at random and send that qubit through the completely depolarizing channel (the completely depolarizing channel takes a state as input and returns the completely mixed state as an output).

If $A_1,\dots ,A_r,B_1,\dots ,B_r$ are complex matrices, then define superoperators $\Phi (A_1,\dots ,A_r)$ and $\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r)$ by setting

$\Phi (A_1,\dots ,A_r)\left(X\right)={\sum }_{k=1}^r{A}_{k}X{A}_k^{\ast }$ and $\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r)={\sum }_{k=1}^r{A}_{k}X{B}_k^{\ast }$ for all $X$. 

Given tuples of matrices $(A_1,\dots ,A_r),(B_1,\dots ,B_r)$, define the L_2-spectral radius similarity between these tuples of matrices by setting

$\Vert (A_1,\dots ,A_r)\simeq (B_1,\dots ,B_r){\parallel }_2$

$=\frac{\rho \left(\Gamma \right(A_1,\dots ,A_r;B_1,\dots ,B_r\left)\right)}{\rho \left(\Phi \right(A_1,\dots ,A_r\left){)}^{1/2}\rho \right(\Phi (B_1,\dots ,B_r){)}^{1/2}}$.

Suppose now that $A_1,\dots ,A_{4n}$ are matrices where $N=\Phi (A_1,\dots ,A_{4n})$. Let $1\le d\le n$. We say that a tuple of complex $d$ by $d$ matrices $(X_1,\dots ,X_{4n})$ is an LSRDR of $A_1,\dots ,A_{4n}$ if the quantity $\parallel (A_1,\dots ,A_{4n})\simeq (X_1,\dots ,X_{4n}){\parallel }_2$ is locally maximized.

Suppose now that $X_1,\dots ,X_{4n}$ are complex $2\times 2$-matrices and $(X_1,\dots ,X_{4n})$ is an LSRDR of $A_1,\dots ,A_{4n}$. Then my computer experiments indicate that there will be some constant $\lambda$ where $\lambda \Gamma (A_1,\dots ,A_{4n};X_1,\dots ,X_{4n})$ is similar to a positive semidefinite operator with eigenvalues $\{0,\dots ,n+1\}$ and where the eigenvalue $j$ has multiplicity $3\cdot C(n-1,k)+C(n-1,k-2)$ where $C(\cdot ,\cdot )$ denotes the binomial coefficient. I have not had a chance to try to mathematically prove this. Hooray. We have interpreted the LSRDR $(X_1,\dots ,X_{4n})$ of $(A_1,\dots ,A_{4n})$, and I have plenty of other examples of interpreted LSRDRs.

We also have a similar pattern for the spectrum of $\Phi (A_1,\dots ,A_{4n})$. My computer experiments indicate that there is some constant $\lambda$ where $\lambda \cdot \Phi (A_1,\dots ,A_{4n})$ has spectrum $\{0,\dots ,n\}$ where the eigenvalue $j$ has multiplicity $3^{n-j}\cdot C(n,j)$.

We can use the $L_2-$spectral radius similarity to measure more complicated similarities between data sets.

Suppose that $A_1,\dots ,A_r$ are $m\times m$-real matrices and $B_1,\dots ,B_r$ are $n\times n$-real matrices. Let $\rho \left(A\right)$ denote the spectral radius of $A$ and let $A\otimes B$ denote the tensor product of $A$ with $B$. Define the $L_2$-spectral radius by setting ${\rho }_2(A_1,\dots ,A_r)=\rho (A_1\otimes A_1+\cdots +A_r\otimes A_r{)}^{1/2}$, Define the $L_2$-spectral radius similarity between $A_1,\dots ,A_r$ and $B_1,\dots ,B_r$ as

$\parallel (A_1,\dots ,A_r)\simeq (B_1,\dots ,B_r){\parallel }_2=\frac{\rho (A_1\otimes B_1+\cdots +A_r\otimes B_r)}{{\rho }_2(A_1,\dots ,A_r){\rho }_2(B_1,\dots ,B_r)}$.

We observe that if $C$ is invertible and $\lambda$ is a constant, then

$\parallel (A_1,\dots ,A_r)\simeq (\lambda C{B}_1{C}^{-1},\dots ,\lambda C{B}_r{C}^{-1}){\parallel }_2=1.$

Therefore, the $L_2$-spectral radius is able to detect and measure symmetry that is normally hidden.

Example: Suppose that $u_1,\dots ,u_r;v_1,\dots ,v_r$ are vectors of possibly different dimensions. Suppose that we would like to determine how close we are to obtaining an affine transformation $T$ with $T\left(u_j\right)=v_j$ for all $j$ (or a slightly different notion of similarity). We first of all should normalize these vectors to obtain vectors $x_1,\dots ,x_r;y_1,\dots ,y_r$ with mean zero and where the covariance matrix is the identity matrix (we may not need to do this depending on our notion of similarity). Then $\parallel (x_1{x}_1^{\ast },\dots ,x_r{x}_r^{\ast })\simeq (y_1{y}_1^{\ast },\dots ,y_r{y}_r^{\ast }){\parallel }_2$ is a measure of low close we are to obtaining such an affine transformation $T$. We may be able to apply this notion to determining the distance between machine learning models. For example, suppose that $M,N$ are both the first few layers in a (typically different) neural network. Suppose that $a_1,\dots ,a_r$ is a set of data points. Then if $u_j=M\left(a_j\right)$ and $v_j=M\left(a_j\right)$, then $\parallel (x_1{x}_1^{\ast },\dots ,x_r{x}_r^{\ast })\simeq (y_1{y}_1^{\ast },\dots ,y_r{y}_r^{\ast }){\parallel }_2$ is a measure of the similarity between $M$ and $N$.

I have actually used this example to see if there is any similarity between two different neural networks trained on the same data set. For my experiment, I chose a random collection of $S\subseteq \{0,1{\}}^{32}\times \{0,1{\}}^{32}$ of ordered pairs and I trained the neural networks $M,N$ to minimize the expected losses $E(\parallel N(a)-b{\parallel }^2:(a,b)\in S),E(\parallel M(a)-b{\parallel }^2:(a,b)\in S)$. In my experiment, each $a_j$ was a random vector of length 32 whose entries were 0's and 1's. In my experiment, the similarity $\parallel (x_1{x}_1^{\ast },\dots ,x_r{x}_r^{\ast })\simeq (y_1{y}_1^{\ast },\dots ,y_r{y}_r^{\ast }){\parallel }_2$ was worse than if $x_1,\dots ,x_r,y_1,\dots ,y_r$ were just random vectors.

This simple experiment suggests that trained neural networks retain too much random or pseudorandom data and are way too messy in order for anyone to develop a good understanding or interpretation of these networks. In my personal opinion, neural networks should be avoided in favor of other AI systems, but we need to develop these alternative AI systems so that they eventually outperform neural networks. I have personally used the $L_2$-spectral radius similarity to develop such non-messy AI systems including LSRDRs, but these non-neural non-messy AI systems currently do not perform as well as neural networks for most tasks. For example, I currently cannot train LSRDR-like structures to do any more NLP than just a word embedding, but I can train LSRDRs to do tasks that I have not seen neural networks perform (such as a tensor dimensionality reduction).

So in my research into machine learning algorithms that I can use to evaluate small block ciphers for cryptocurrency technologies, I have just stumbled upon a dimensionality reduction for tensors in tensor products of inner product spaces that according to my computer experiments exists, is unique, and which reduces a real tensor to another real tensor even when the underlying field is the field of complex numbers. I would not be too surprised if someone else came up with this tensor dimensionality reduction before since it has a rather simple description and it is in a sense a canonical tensor dimensionality reduction when we consider tensors as homogeneous non-commutative polynomials. But even if this tensor dimensionality reduction is not new, this dimensionality reduction algorithm belongs to a broader class of new algorithms that I have been studying recently such as LSRDRs.

Suppose that $K$ is either the field of real numbers or the field of complex numbers. Let $V_1,\dots ,V_n$ be finite dimensional inner product spaces over $K$ with dimensions $d_1,\dots ,d_n$ respectively. Suppose that $V_i$ has basis $e_{i,1},\dots ,e_{i,d_i}$. Given $v\in V_1\otimes \cdots \otimes V_n$, we would sometimes want to approximate the tensor $v$ with a tensor that has less parameters. Suppose that $(m_0,\dots ,m_n)$ is a sequence of natural numbers with $m_0=m_n=1$. Suppose that $X_{i,j}$ is a $m_{i-1}\times m_i$ matrix over the field $K$ for $1\le i\le n$ and $1\le j\le d_i$. From the system of matrices $(X_{i,j}{)}_{i,j}$, we obtain a tensor $T\left(\right(X_{i,j}{)}_{i,j})={\sum }_{i_1,\dots ,i_n}{e}_{i_1}\otimes \cdots \otimes e_{i_n}\cdot X_{1,i_1}\dots X_{n,i_n}$. If the system of matrices $(X_{i,j}{)}_{i,j}$ locally minimizes the distance $\parallel v-T\left(\right(X_{i,j}{)}_{i,j})\parallel$, then the tensor $T\left(\right(X_{i,j}{)}_{i,j})$ is a dimensionality reduction of $v$ which we shall denote by $u$.

Intuition: One can associate the tensor product $V_1\otimes \cdots \otimes V_n$ with the set of all degree $n$ homogeneous non-commutative polynomials that consist of linear combinations of the monomials of the form $x_{1,i_1}\cdots x_{n,i_n}$. Given, our matrices $X_{i,j}$, we can define a linear functional $\varphi :V_1\otimes \cdots \otimes V_n\to K$ by setting $\varphi \left(p\right)=p\left(\right(X_{i,j}{)}_{i,j})$. But by the Reisz representation theorem, the linear functional $\varphi$ is dual to some tensor in $V_1\otimes \cdots \otimes V_n$. More specifically, $\varphi$ is dual to $T\left(\right(X_{i,j}{)}_{i,j})$. The tensors of the form $T\left(\right(X_{i,j}{)}_{i,j})$ are therefore the

Advantages: 

1. In my computer experiments, the reduced dimension tensor $u$ is often (but not always) unique in the sense that if we calculate the tensor $u$ twice, then we will get the same tensor. At least, the distribution of reduced dimension tensors $u$ will have low Renyi entropy. I personally consider the partial uniqueness of the reduced dimension tensor to be advantageous over total uniqueness since this partial uniqueness signals whether one should use this tensor dimensionality reduction in the first place. If the reduced tensor is far from being unique, then one should not use this tensor dimensionality reduction algorithm. If the reduced tensor is unique or at least has low Renyi entropy, then this dimensionality reduction works well for the tensor $v$.
2. This dimensionality reduction does not depend on the choice of orthonormal basis $e_{i,1},\dots ,e_{i,d_i}$. If we chose a different basis for each $V_i$, then the resulting tensor $u$ of reduced dimensionality will remain the same (the proof is given below).
3. If $K$ is the field of complex numbers, but all the entries in the tensor $v$ happen to be real numbers, then all the entries in the tensor $u$ will also be real numbers.
4. This dimensionality reduction algorithm is intuitive when tensors are considered as homogeneous non-commutative polynomials.

Disadvantages: 

1. This dimensionality reduction depends on a canonical cyclic ordering the inner product spaces $V_1,\dots ,V_n$.
2. Other notions of dimensionality reduction for tensors such as the CP tensor dimensionality reduction and the Tucker decompositions are more well-established, and they are obviously attempted generalizations of the singular value decomposition to higher dimensions, so they may be more intuitive to some.
3. The tensors of reduced dimensionality $T\left(\right(X_{i,j}{)}_{i,j})$ have a more complicated description than the tensors in the CP tensor dimensionality reduction.

Proposition: The set of tensors of the form ${\sum }_{i_1,\dots ,i_n}{e}_{1,i_1}\otimes \cdots \otimes e_{n,i_n}{X}_{1,i_1}\dots X_{n,i_n}$ does not depend on the choice of bases $(e_{i,1},\dots ,e_{i,d_i}{)}_i$.

Proof: For each $i$, let $f_{i,1},\dots ,f_{i,d_i}$ be an alternative basis for $V_i$. Then suppose that $e_{i,j}={\sum }_k{u}_{i,j,k}{f}_{i,k}$ for each $i,j$. Then

${\sum }_{i_1,\dots ,i_n}{e}_{1,i_1}\otimes \cdots \otimes e_{n,i_n}{X}_{1,i_1}\dots X_{n,i_n}$

$={\sum }_{i_1,\dots ,i_n}{\sum }_{k_1}{u}_{1,i_1,k_1}{f}_{1,i_1}\otimes \cdots \otimes {\sum }_{k_n}{u}_{n,i_n,k_n}{f}_{n,i_n}{X}_{1,i_1}\dots X_{n,i_n}$

$={\sum }_{k_1,\dots ,k_n}{f}_{1,k_1}\otimes \cdots \otimes f_{n,k_n}{\sum }_{i_1,\dots ,i_n}{u}_{1,i_1,k_1}\dots u_{n,i_n,k_n}{X}_{1,i_1}\dots X_{n,i,n}$

$={\sum }_{k_1,\dots ,k_n}{f}_{1,k_1}\otimes \cdots \otimes f_{n,k_n}\left({\sum }_{i_1}{u}_{1,i_1,k_1}{X}_{1,i_1}\right)\dots \left({\sum }_{i_n}{u}_{n,i_n,k_n}{X}_{i_n}\right)$. Q.E.D.

A failed generalization: An astute reader may have observed that if we drop the requirement that $m_n=1$, then we get a linear functional defined by letting

$\varphi \left(p\right)=\text{Tr}\left(p\right(\left(X_{i,j}{)}_{i,j}\right))$. This is indeed a linear functional, and we can try to approximate $v$ using a the dual to $\varphi$, but this approach does not work as well.

In this note, I will continue to demonstrate not only the ways in which LSRDRs ($L_{2,d}$-spectral radius dimensionality reduction) are mathematical but also how one can get the most out of LSRDRs. LSRDRs are one of the types of machine learning that I have been working on, and LSRDRs have characteristics that tell us that LSRDRs are often inherently interpretable which should be good for AI safety.

Suppose that $N$ is the quantum channel that maps a $n$ qubit state to a $n$ qubit state where we select one of the 6 qubits at random and send it through the completely depolarizing channel (the completely depolarizing channel takes a state as an input and returns the completely mixed state as an output). Suppose that $A_1,\dots ,A_{4n}$ are $2^n$ by $2^n$ matrices where $N$ has the Kraus representation $N\left(X\right)={\sum }_{k=1}^{4n}{A}_{k}X{A}_k^{\ast }$. 

The objective is to locally maximize the fitness level $\rho \left({\sum }_{k=1}^{4n}{z}_k{A}_k\right)/\parallel (z_1,\dots ,z_{4n})\parallel$ where the norm in question is the Euclidean norm and where $\rho$ denotes the spectral radius. This is a 1 dimensional case of an LSRDR of the channel $N$.

Let $A={\sum }_{k=1}^{4n}{z}_k{A}_k$ when $(z_1,\dots ,z_{4n})$ is selected to locally maximize the fitness level. Then my empirical calculations show that there is some $\lambda$ where $\lambda {\sum }_{k=1}^{4n}{z}_k{A}_k$is positive semidefinite with eigenvalues $\{0,\dots ,n\}$ and where the eigenvalue $k$ has multiplicity $(\frac{n}{k})$ which is the binomial coefficient. But these are empirical calculations for select values $\lambda$; I have not been able to mathematically prove that this is always the case for all local maxima for the fitness level (I have not tried to come up with a proof).

Here, we have obtained a complete characterization of $A$ up-to-unitary equivalence due to the spectral theorem, so we are quite close to completely interpreting the local maximum for our fitness function. 

I made a few YouTube videos showcasing the process of maximizing the fitness level here.

Spectra of 1 dimensional LSRDRs of 6 qubit noise channel during training

Spectra of 1 dimensional LSRDRs of 7 qubit noise channel during training

Spectra of 1 dimensional LSRDRs of 8 qubit noise channel during training

I will make another post soon about more LSRDRs of a higher dimension of the same channel $N$.

I personally like my machine learning algorithms to behave mathematically especially when I give them mathematical data. For example, a fitness function with apparently one local maximum value is a mathematical fitness function. It is even more mathematical if one can prove mathematical theorems about such a fitness function or if one can completely describe the local maxima of such a fitness function. It seems like fitness functions that satisfy these mathematical properties are more interpretable than the fitness functions which do not, so people should investigate such functions for AI safety purposes.

My notion of an LSRDR is a notion that satisfies these mathematical properties. To demonstrate the mathematical behavior of LSRDRs, let's see what happens when we take an LSRDR of the octonions.

Let $K$ denote either the field of real numbers or the field of complex numbers ($K$

 could also be the division ring of quaternions, but for simplicity, let's not go there). If $A_1,\dots ,A_r$ are $n\times n$-matrices over $K$, then an LSRDR ($L_{2,d}$-spectral radius dimensionality reduction) of $A_1,\dots ,A_r$ is a collection $X_1,\dots ,X_r$ of $d\times d$-matrices that locally maximizes the fitness level

$\frac{\rho (A_1\otimes \overline{X_1}+\cdots +A_r\otimes \overline{X_r})}{\rho (X_1\otimes \overline{X_1}+\cdots +X_r\otimes \overline{X_r}{)}^{1/2}}$. $\rho$ denotes the spectral radius function while $\otimes$ denotes the tensor product and $\overline{Z}$ denotes the matrix obtained from $Z$ by replacing each entry with its complex conjugate. We shall call the maximum fitness level the $L_{2,d}$-spectral radius of $A_1,\dots ,A_r$ over the field $K$, and we shall wrote ${\rho }_{2,d}^K(A_1,\dots ,A_r)$ for this spectral radius.

Define the linear superoperator $\Gamma (A_1,\dots ,A_r;X_1,\dots ,X_r)$ by setting 

$\Gamma (A_1,\dots ,A_r;X_1,\dots ,X_r)\left(X\right)=A_{1}X{X}_1^{\ast }+\cdots +A_{r}X{X}_r^{\ast }$ and set $\Phi (X_1,\dots ,X_r)=\Gamma (X_1,\dots ,X_r;X_1,\dots ,X_r)$. Then the fitness level of $X_1,\dots ,X_r$ is $\frac{\rho \left(\Gamma \right(A_1,\dots ,A_r;X_1,\dots ,X_r\left)\right)}{\Phi (X_1,\dots ,X_r{)}^{1/2}}$.

Suppose that $V$ is an $8$-dimensional real inner product space. Then the octonionic multiplication operation is the unique up-to-isomorphism bilinear binary operation $\ast$ on $V$ together with a unit $1$ such that$\parallel x\ast y\parallel =\parallel x\parallel \cdot \parallel y\parallel$ and $1\ast x=x\ast 1=1$ for all x$,y\in V$. If we drop the condition that the octonions have a unit, then we do not quite have this uniqueness result. 

We say that an octonion-like algbera is a $8$-dimensional real inner product space $V$ together with a unique up-to-isomorphism bilinear operation $\ast$ such that $\parallel x\ast y\parallel =\parallel x\parallel \cdot \parallel y\parallel$ for all $x,y$.

Let $V$ be a specific octonion-like algebra.

Suppose now that $e_1,\dots ,e_8$ is an orthonormal basis for $V$ (this does not need to be the standard basis). Then for each $j\in \{1,\dots ,8\}$, let $A_j$ be the linear operator from $V$ to $V$ defined by setting $A_{j}v=e_j\ast v$ for all vectors $v$. All non-zero linear combinations of $A_1,\dots ,A_8$ are conformal mappings (this means that they preserve angles). Now that we have turned the octonion-like algebra into matrices, we can take an LSRDR of the octonion-like algebras, but when taking the LSRDR of octonion-like algebras, we should not worry about the choice of orthonormal basis $e_1,\dots ,e_8$ since I could formulate everything in a coordinate-free manner.

Empirical Observation from computer calculations: Suppose that $1\le d\le 8$ and $K$ is the field of real numbers. Then the following are equivalent.

1. The $d\times d$ matrices $X_1,\dots ,X_8$ are a LSRDR of $A_1,\dots ,A_8$ over $K$ where $A_1\otimes X_1+\cdots +A_8\otimes X_8$ has a unique real dominant eigenvalue.
2. There exists matrices $R,S$ where $X_j=R{A}_{j}S$ for all $j$ and where $SR$ is an orthonormal projection matrix.

In this case, ${\rho }_{2,d}^K(A_1,\dots ,A_8)=\sqrt{d}$ and this fitness level is reached by the matrices $X_1,\dots ,X_8$ in the above equivalent statements. Observe that the superoperator $\Gamma (A_1,\dots ,A_8;P{A}_{1}P,\dots ,P{A}_{8}P)$ is similar to a direct sum of  $\Gamma (A_1,\dots ,A_r;X_1,\dots ,X_r))$ and a zero matrix. But the projection matrix $P$ is a dominant eigenvector of $\Gamma (A_1,\dots ,A_8;P{A}_{1}P,\dots ,P{A}_{8}P)$ and of$\Phi (P{A}_{1}P,\dots ,P{A}_{8}P)$ as well.