All of adamant's Comments + Replies
30
Gotcha, and thank you so much for writing this post!
33
Ah! Were you the one who decorated the Rose Garden Inn? I'm really curious how you made the lighting that looks like the sun coming through a cracked door / coming through the cracks between bricks.
Picture below -- I took a ton of pictures when I was there to steal your interior decoration ideas.
6
I have not worked on the Inn, but the search term for that kind of light is 'recessed channel lighting' :)
(I guess I should mention in the post that my recommendations were rental-space-oriented; Lightcone was able to install this recessed lighting because they own the space and can do whatever they want to it, but it's too invasive for a rental.)
10
Aside which the original author may be interested in -- there has been some work done to reduce the scaling of the context window below O(n^2) -- e.g. https://arxiv.org/pdf/1904.10509v1.pdf. I also think of OpenAI's jukebox which uses a hierarchical strategy in addition to factorized self-attention for generating tokens to effectively increase the context window (https://openai.com/blog/jukebox/)
Very informative toy examples. Regarding this point:
> Some kind of failure of spatial reasoning (wandering items, whatever was going on with some of the sliding square chain-of-thoughts where pieces vanished)
I would strongly agree with this. I actually think the sliding block puzzle is a task which might just be easy for humans on account of our strong spatial priors. In the physical world, things move with spatial locality and two objects cannot be in the same place. For the LLM, it is trained on orders of magnitude less data to learn to represent spat... (read more)