I think there's actually a pretty good solution to this that I've been looking forward to for years, and that I think the technology is now ready to support. Unfortunately, I'm too busy worrying about, I mean, working on AGI alignment to spend time on this currently. The idea is simple: an open source decentralized recommender/search engine that you keep a local copy of and shared your recommendations of search results over a decentralized social network like Mastodon. Like a combination of Google Search, StumbleUpon, and Reddit. Search results get upvoted or downvoted, and ones that were actually helpful to friends of your friends will be higher ranked for you, and even more so for friends of friends who fit a similar pattern of web usage. There's been a variety of specific implementations for this suggested. Here's an example: https://www.researchgate.net/publication/260652116_A_Collaborative_Decentralized_Approach_to_Web_Search
This is along the general theme of 'we ought to take back ownership of our data and our internet rather than leaving it in the hands of corporate giants to mediate for us'.
next levels of spicy above kagi, besides millionshort, are https://teclis.com/ and https://search.marginalia.nu/. there are some other interesting related engines on https://news.ycombinator.com/item?id=30783037 and even more on https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/, notable indieweb mentions include https://wiby.me/ and https://searchmysite.net/ but I tend to prefer teclis and marginalia. kagi is apparently google+teclis, which I didn't realize until just now, and means I probably should have just recommended teclis to begin with.
of course, it's not perfect. there are still SEO groups trying to target these engines. but they're more explicitly focused on anti-seo design.
The issue is not SEO, it is google deciding that it knows better than I do what I want to search for and pushing useless irrelevant results to the top 10 pages.
huh update: another interesting semantic search engine, this one works like a language model but with a live index of urls and embeddings of them. seems to be a more direct style of prediction than some of the research retrieval engines I've tried: https://metaphor.systems/
oh I just use ddg to evade that problem. that and more playing with Google. I still like Google best for question resolution searches, but it's true that you have to be more specific with Google's current shitty English encoder.
Ok I hate DDG and every other search engine out there is done zip for me with this except fairly often a place called Yousearch. I found mentioned in an online article. While far from perfect and sometimes giving sadly similar to google results I have had much luck with it around 67%of the time I think to check it. I wish I wrote code and could work on a search replacement but I love the idea of the open source one here.
As is customary, I whined (on IRC) about Google Search being unhelpful or actively counterproductive with the results these days and how it is not what it used to be. Specifically, I wanted to know something very niche, why many LTE modems sometimes take too long to disconnect. The hits were actively unhelpful, like "why does my internet keep disconnecting?" and there was NOT A SINGLE RELEVANT ENTRY in the first 10 pages. (And yes, I know the information is out there, you just can't form a good query to find it.) Here is the best response by far (emphasis mine):
<quanticle> It reminds me of my friend's dog. This dog was a candidate to be a guide/helper dog for disabled people but, for whatever reason, just couldn't pass the final test of guide dog academy. So, it had like... 90% of the training and instincts of a guide dog, but didn't have anyone to actually guide. So it'd do some cute-but-in-retrospect-neurotic things like, if it noticed my friend cooking, it'd bring her a screwdriver? Why? Because, look, you're doing a thing, and here's a thing! I HALP
<quanticle> Google is like that dog