I'm figure out if what I think of "thinking" is fundamentally different from what you describe here - you make me think that these LLMs have a more impressionistic view of the world, where "5 + 3" gets the right answer if you've related that expression to "8" tightly enough, or you have a tight enough definition of "fiveness" "threeness" and "plus-age" you'll arrive at "8". When I do 5 + 3 or a geometric proof or analyze an if-then statement, I feel like I'm doing more than just this impressionistic closeness ranking. Do you think I'm fooling myself, or is what I'm doing fundamentally different from what an LLM is doing?
I also feel like given this definition of thinking, an LLM is limited to creating linear combinations of existing knowledge, some combinations of which we haven't thought of yet, so it would be extremely valuable. But it still seems like it would lack the ability to forge new ground. But again, maybe I'm fooling myself as to what "creativity" truly is.
Next time someone clearly needs a nap I'm going to ask them if they've tried turning themselves off and then on again