Nice! Last weekend I expanded https://www.gptrim.com/ to allow the user to a) see savings in both characters and tokens; b) determine their own combination of word processing functions. Then I saw, like you said, that to save tokens you only want to remove stopwords. I will next add the option to remove punctuation. I also want to give users two general recipes: optimize for saving tokens vs. optimize for saving characters. Always happy to take more ideas.
I will probably write again on this, on my new personal Substack or other websites, reporting what I've learned. Would you like me to cite you and link to your profile? My DMs are open!
P.S: Due to my speedrunning the coding, the website now has an issue where it sometimes adds spaces to the text. I am aware of this and will fix it latest next weekend. The Python package that runs all this is accepting PRs: https://github.com/vlad-ds/gptrim.
I see your point. I think the existing tokenizer is designed to keep all parts of text, while the idea here is to sacrifice some information in favor of compression. But writing this, I also realized that this approach is more effective at saving characters than tokens.
This is what I was hoping for when I wrote this post. Thank you for your insight.
New position: sometimes when using ChatGPT, you only care about the number of characters, because of the character limit in the chat message. In that case, you want to get rid of spaces. But if you want to save on tokens, you probably should keep spaces. I think the solution is: a) allow the user to choose the mix of transformations for their use case; b) show them how much they are saving in characters and tokens so they can optimize for their use case.
Free, universal financial tracker.
I wrote an article on this subject (i.e. why do we play zero-sum games while praising positive-sum games?)
https://native-wonder.blogspot.com/2020/12/things-people-want.html
Thank you, this is very useful. Lately I've been interested in programs that are fully online and could be completed in a year. Would you have any recommendations for that?
Wonderful, thank you!
Strongly upvoted. As a Kindle-dependent newcomer who's delving into the classics, this is precious.
I have read RAZ. Does this file include it? I would actually need only the posts that are not there.
Do you plan to do this for other authors?
I had trouble understanding how the different facts and judgments in your post are connected between each other and with the concept of upside decay.
But I want to say that I really appreciate the concept, because something very similar occurred to me once, though at the time I didn't give it a name. I was studying the careers of creative artists, and there is a lot of discrimination in these fields. Against women, against people who start out in less prestigious institutions, and so on.
My idea was that because many people were excluded and diversity was stifled, this reduced the probability of "hitting the jackpot" with an extremely brilliant artist that would be the far right of the "artistic potential" curve and end up being the next Picasso. I wanted to model this intuition and verify it in the data, but eventually my project changed and I moved on. The idea, anyway, is that you reduce the chance of getting outliers (or even black swans) in the tails, but you only care about positive outliers.
Here's a thought experiment.
In version A, I have a button that non invasively scans my brain and creates 10 perfect copies of my brain state in a computer. I press the button. For an instant, 11 identical mind states exist in the universe. Then each mind starts diverging along different causal chains.
Intuitively, I expect the following:
In this case, I identify myself with the embodied mind.
In version B, the setup is identical except the scan is destructive. The second I press it, my physical body is destroyed.
Now, what happens to me? There's no specific reason for me to end up in one of the minds and not the others. But I cannot go to all 10 minds at the same time — I am a single mind with its own casual chain, not a collection of minds.
For instance, imagine each of the 10 minds is caused to feel a different sensation at the same time. There's nobody to feel all 10 sensations at the same time because the minds are causally isolated. Yet I cannot say that I am feeling a particular sensation and not the others.
So in version B, I still identify myself with the embodied mind, which is destroyed — hence oblivion. Conversely, what happens to the 10 minds if I delete them from the computer? Oblivion.
(This is just my attempt to map my naive intuitions. I have a sense some version of no-self could be the solution, but I'm not there yet. I also feel that naive intuitions fail for Everett branches which is another reason to be suspicious.)