There’s been a lot of complaining about Google’s search results lately, especially when it comes to high-stakes subjective queries. But no viable competitor has emerged despite many new search engines launching lately.
At least nothing good enough that people are excitedly sharing “Hey, have you tried X lately, it’s way better than Google.” What follows is an effort to explain what’s going on and a pitch for an open-source project I’ve been working on that tries to solve the problem in a new way.
What makes Google so hard to beat?
Google’s moat is simple but powerful — the more people use the search engine, the better it understand what the best answer to any question is.
They use signals like time spent on page and bounce rates to determine what pages best answer a question. Their ownership of Chrome makes this even more powerful. On pages where the search result page (SERP) has an inline answer, the user’s next query is a powerful signal of success.
With a 92% market share, Google has an order of magnitude more of this kind of data than any of it’s competitors.
If they have all this data, why are the results subjectively bad?
There’s a few reasons I can think of that hurt Google search quality, primarily in complex subjective queries.
- Time-on-page and bounce rates are actually not good signals to determine success for complex topics. I may spend 15 minutes reading a page about a complex topic and come away with a weak understanding of the topic, and a good page on the topic would also take 15 minutes to digest. Often, power users will search, command/ctrl + click the top 5 links and read through all of them for queries like this.
- Google is too large to take a stand on which publishers are “credible” and which are not. Google can’t just come out and say “hey, WebMD is fear mongering and UpToDate is a credible source for medical queries”, even though that’s what all the insiders in the field already believe to be true. The best is can do is try to organically rank one above the other. But this is difficult when the better source publishes more complex information that is harder to comprehend for most users (and thus results in higher bounce rates).
- Google makes most of it’s revenue from people making complex life decisions. Things like healthcare and financial decisions. A cynical suggestion may be that if the organic results were much better than the ads above, then significant revenue will be lost.
- It does not matter that results are bad in the “tail” (complex but rare queries) because it makes for a small percent of total queries and most users form search habits based on head queries, which Google is exceedingly good at.
For example, Google invests heavily in short answer snippets and takes full advantage of their data superiority to stay ahead of the competition.
So are we stuck with Google despite it clearly not working well in some areas?
No, users are moving off Google for more complex queries, for example see this trend.
So what are people resorting to instead?
People are increasingly asking the questions that really matter in their Facebook and WhatsApp groups, Twitter, Discord and Slack communities, etc.
But I’m not super well networked, what should I do?
The well-networked amongst us can just ask their following and get great, authentic answers to just about anything.
But for the rest of us or questions we are just too shy to ask, we need a better public search engine.
Some of us are resorting to ever more specific queries to try and go directly to sources we’d trust more. This is evident in Google’s own autocomplete suggestions
So why doesn’t Google rank these sources higher?
There’s a few remaining sources of authentic information on the internet. On the English speaking internet, Reddit is chief amongst them. Google wouldn’t want to drive so much traffic to them that they can eventually cut Google out.
This has actually happened in South Korea with Naver.
Ok, but a good Google competitor can uprank reddit, no?
Yes, absolutely, but as big as Reddit is, the internet is way bigger, and simple heuristics like “uprank reddit” break down at a large enough scale.
Search is fundamentally about people querying things they’re not experts in, so each user’s search queries span a very diverse range of topics that reddit doesn’t serve uniformly well.
For example, it’s tough to trust an anonymous redditor with advice about your child’s medication.
Surely there must be credible sources for each topic?
Yes there are, but there can be too many to remember. For example, a good place to start medical queries is UpToDate, but it’s hard to know that unless you’re in the medical field, and the information is hard to parse.
Now consider that, but across thousands of topics and you see how there’s too many search engines to keep track of.
Why not have a search engine that keeps a list of credible sites and filters just that?
Now we’re beginning to circle around a solution. And full disclosure, this is the space in which I’m personally investing my time.
But first, let’s talk about why Google can’t do this. I mentioned earlier that Google is too big of a company to simply say XYZ is credible. They can sort of indirectly hint at this but can go only so far before antitrust pegs them back.
You can see that they sort of try to do this with publisher biographies pulled from Wikipedia but that’s about as far as they go.
Can a non-Google competitor do better?
Potentially, and this is the core promise of a lot of new startups.
But where they lose out is that habit formation happens in the head, and almost all these competitors rely on Bing for head results, which is simply not as good at Google because of the data advantage we talked about above.
But what if you could find a way to improve Google in the tail only as needed?
That’s our bet. Hypersearch is a desktop browser extension that uses your pre-existing search engine (Google, DuckDuckGo, etc.), but filters it in a sidebar. Hyperweb, our mobile app, does something similar.
We acknowledge that it’s truly difficult to beat Google at everything, so we simply augment it when it makes sense.
But we also recognize that for some verticals, like shopping, many people have already left Google for say Amazon, which has its own similar problems with search quality now that they dominate the vertical.
But search doesn’t end at the SERP, it ends when you make a decision. So both our apps enhance not only Google, but any web page in the search journey, because search is about making decisions, and those decisions don’t happen only on the search results page only.
Finally, we recognize that many people make it work without Google, or would rather stay off it for principled reasons. So we support a variety of other web search engines, such as DuckDuckGo, StartPage, etc.
What’s the catch?
Ultimately, we think Google’s occasional poor results quality in the monetizable tail comes from its incentives being misaligned with that of the users. So we’ve tried to make our incentives as simple as possible
- The mobile app is free to use for this feature and has paid upgrade for other enhancements.
- The desktop extension is 100% free and open source.
Ultimately, we think you can sustain a successful product here with less revenue than what Google brings in.
Wouldn’t Google just remove this from the Chrome store?
We comply fully with the terms of the Chrome store, but of course they could change the terms. We think this solution also complements Google as much as any solution can.
But more generally, there are many ways to achieve what we’re building, so someone else can build their own version. In fact, we GPL licensed the extension so anyone can learn from it and build on top of it.
What does the future of search look like?
We don’t know exactly, but we think it includes rather than replaces Google, which remains the best objective search engine there is.
That’s our bet though, and people will try other ways, and the market will decide. A market that does not have to be winner-take-all forever.