TL;DR:Platonic Representation Hypothesis holds on out-of-distribution data
The Platonic Representation Hypothesis (PRH) says that models trained with different objectives and across different modalities can converge towards a shared statistical representation of reality. The experiments conducted in the paper focus image models trained on ImageNet (largely on the same underlying data distribution).
It is unclear whether this convergence is only happening because of the high overlap in the training data used across the models, I replicate the analysis on ImageNet-O, a dataset specifically designed to be out-of-distribution relative to ImageNet, and re-evaluate the correlations between alignment scores across a range of image classification models.
The strong correlations persist even in the OOD setting, I.e providing further evidence that PRH is not merely a consequence of models being trained on the same distribution.
The plots below compare the original in-distribution results with the corresponding OOD analysis.
But, what happens if we measure these correlations on purely random data? Do the models still exhibit a high correlation? That would mean that there is no such thing as a shared statistical model of reality, but its just a case of spurious correlation as a result of the models having extremely high-capacity.
This below plot shows the correlation of the alignment scores for the models on purely randomly generated images ,and it clearly shows a much lower correlation on random-data than on natural real world distributions like images from Imagenet-O/Imagenet
These results provide further evidence that it is highly likely that there could be a shared universal-structure being captured by the models as identified by PRH, And not merely agreement on correct predictions due to training-data overlap or other trivial null-hypothesis.
The notebook used for running the experiments is linked here.
This seems very interesting, but I think your post could do with a lot more detail. How were the correlations computed? How strongly do they support PRH? How was the OOD data generated? I'm sure the answers could be pieced together from the notebook, but most people won't click through and read the code.
TL;DR: Platonic Representation Hypothesis holds on out-of-distribution data
The Platonic Representation Hypothesis (PRH) says that models trained with different objectives and across different modalities can converge towards a shared statistical representation of reality. The experiments conducted in the paper focus image models trained on ImageNet (largely on the same underlying data distribution).
It is unclear whether this convergence is only happening because of the high overlap in the training data used across the models, I replicate the analysis on ImageNet-O, a dataset specifically designed to be out-of-distribution relative to ImageNet, and re-evaluate the correlations between alignment scores across a range of image classification models.
The strong correlations persist even in the OOD setting, I.e providing further evidence that PRH is not merely a consequence of models being trained on the same distribution.
The plots below compare the original in-distribution results with the corresponding OOD analysis.
But, what happens if we measure these correlations on purely random data? Do the models still exhibit a high correlation? That would mean that there is no such thing as a shared statistical model of reality, but its just a case of spurious correlation as a result of the models having extremely high-capacity.
This below plot shows the correlation of the alignment scores for the models on purely randomly generated images ,and it clearly shows a much lower correlation on random-data than on natural real world distributions like images from Imagenet-O/Imagenet
These results provide further evidence that it is highly likely that there could be a shared universal-structure being captured by the models as identified by PRH, And not merely agreement on correct predictions due to training-data overlap or other trivial null-hypothesis.
The notebook used for running the experiments is linked here.