Why is Search Still so Hard?

After so many years, why can’t we get it right?

Photo by Kelly Sikkema on Unsplash

SERP is an acronym for Search Engine Results Page. When you search for something, the resulting list of links shown is known as a SERP, and there is a lot of jockeying done to get placed high on those pages.

Search Today

As all of us here are, more than likely, long time denizens of the Internet. We have seen search evolve from extremely primitive to the complex, interactive, data-driven, service that it is today. Always there on some of our phones, and rarely more than a click, swipe, or gesture away. For those of us who remember the term, it is the killer app of the digital world.

It is November 27, 2020, at 6:25 PT as I write this, and Internet Live Stats is telling me that today Google has made 5.8 Billion searches. According to Statista, they have an 88% market share. Just using these two numbers as our data, today there will be at least 6.6 Billion searches. Perhaps other than breathing and certain other bodily functions, it might be one of the most frequent human activities.

And, there is reason to suspect that 6.6B searches might be low as much as 100%. That number doesn’t include voice search, nor does it include searches not done on Big Search, on Lexis-Nexis or Travelocity for example.

On the web search is as vital as air. Without search, the Internet is a brick.

Think for a moment how many times you searched yesterday. What would you have done if you couldn’t search? How would you have found all of the stuff that search brought you if there wasn’t any way to find it?

We all are highly reliant on clear, unbiased, comprehensive results. The question is, does Big Search give it to us? As the Internet grows, it’ll become even more of a problem. And who knows what new privacy issues find out next year? Next decade?

Futher, what about the growth of the Internet? The Internet grows by thousands of websites per day. And, it’s spawning a whole new technological arm in Blockchain. Blockchain URLS (BCURLS) are being built right now. Big Search doesn’t even have a way to search those.

At the moment, there are really only two choices; Google and Bing. Almost every search engine today relies on one of the two for the index they use to answer our questions. Google has the best results in terms of showing you what’s out there, but they are surveillance capitalists and will suck up all of our data for the rest of our lives as the price to use them for search.

Privacy search engines like DuckDuckGo rely on the Bing index and Bing is the ugly stepchild of Microsoft; it just never gets the funding it would need to provide results from very far around the Internet.

Our choice seems to come down to being surveilled or not getting good results.

What if there’s a better way? Even as I write this, it seems almost anathema to even think about search being different than it is today.

How it could better: Independent Search

Stepping back from that precipice just a bit, let’s imagine for a moment what Independent Search might look like.

I suggest it would have the following attributes:

Always available. It would never be more than one action away — a click or a tap or a swipe.

Unbiased. Independent. Results on Big Search are clearly biased by business interests — why can’t I just get a query answered? If there are really only 4 results that closely match my query, why show me 70,000 or even 5?

Boolean. It should always be possible to search for p and q, p or q, or p and q not z.

Deep. Results on the 100th SERP page do no one any good at all — what if that’s the answer I need?

Transparent. Why is XYZ company at the top of my organic results? Today, no one can answer that question for any given search.

Wildcard enabled. The term *.jpeg should always give me all of the results that have .jpeg as their extension — especially true in search on my local system and the clouds I regularly access for storage.

And, more than almost anything else, it should be checkable. What if Big Search presents the wrong answer? How would you know? How would you check and see that their answer is right? There are simply not a lot of ways to do that today. I’ve had the experience of getting incorrect results on all of the current search engines, as I am sure we all have. It leaves me asking, “Why did they show me that?” There is just no good way to get a view of the results of my query that’s different than Big Search, but just as good.

This goes to transparency and unbiased results. For instance, on the SERP below, there are no ads, so the only results are the results of Google’s ranking, but there is not one single person in the world who can tell me why Whatismybrowser.com is at the top of that list. And there is no way to check. Even going to the site, I still don’t have any idea if they are at all reputable or have any financial agreement with Google. Why can’t we know why that site is there? Is something being hidden?

Source: Author/What browser am I using — Searched August 20, 2020

As with many things on the Internet, as search grew organically, it took on a lot of attributes that no one would find acceptable until, today, those features begin to dominate the whole experience. If we were starting to build search today, would we build a massive, complex system that reaches into almost every part of our lives?

Big Search has become overly intrusive.

Are they going beyond what is needed for good results?

Take a look at this book. It’s How Search Works from the highly respected Search Engine Journal. Look at page 59 and see just one sequence in the middle of a much longer sequence of events in providing your SERP.

From How Search Works Source: SEJ

Admittedly, some of this is needed to provide the results we want and, these days, it’s time to wonder just how much of this really is needed for us to get those results.

In the time it takes to provide a SERP, they have looked deeply into things like how you’ve interacted with websites before, and they use that, in some mysterious way, to determine what results are on your SERP. All in the name of personalized search, which is supposed to make your time searching more productive.

Does it? Or is the whole idea nonsensical? I can see why it’s better for Big Search, but just because I was looking for Macintosh computers yesterday doesn’t mean that I’m not searching for Macintosh apples today. The reasoning behind personalized search just happens to handily align with Big Search’s business interests — not our interests necessarily, and certainly not all the time.

Yes, the Internet is big and complex. But, why does it take How Search Works 148 pages to explain search? Shouldn’t search be a bit more straightforward? OK, sure, there are pictures and diagrams in the book, but still — let’s think for a minute about what it might take to provide high-quality SERPs. For this essay, let’s say that a high-quality SERP is a list of links that allow me to answer my question quickly and accurately.

What’s the minimum amount a search engine could do to provide that for me? Isn’t that, in these days of anti-trust suits and global warming, what most of us are looking for?

There should be a Hippocratic Oath for search — cause no harm. Be minimally invasive. Strive to consume less energy.

The real costs of doing all of that

And when we start looking at the real cost of search, the numbers are scary. From a recent Forbes article,

Last year, Google used about 12.4 terawatt-hours of electricity, which means it uses more electric power than entire countries, including ones like Sri Lanka and Zambia. If Google were an independent nation, its electricity use would rank among the top 90 countries in the world. Furthermore, Google’s electricity use is doubling every three years or so.

Big Search does so much that they consume huge amounts of power, which has to lead one to the question, “Is it really all needed?” And, what is the global carbon footprint of search? How much greenhouse gas is emitted from every search we do?

Search in Action

Coming down just a bit from our 30,000-foot view, let’s see how some of the larger Search providers stack up against our six attributes of what’s needed for search — available, unbiased, boolean, deep, transparent, and wildcards.


Now, in the name of full transparency, I am an admitted Apple guy — very happy in my walled garden, thank you.

But search has been an issue for Apple for a long time, and it’s not a lot better today. Spotlight on the Mac is a huge fail. And it gets worse if you try to include anything in the cloud.

On iOS general search is better but not in the native apps. To search in Message, or a number of other native apps, I have to scroll all the way to the top of the list. So if I am 300 or 3,000 messages down, that’s a lot of scrolling. It fails the always available test.


The 6,000,000 pound gorilla. Consuming 98% of mobile search and something around 67% of fixed search, depending on who’s numbers you believe.

But is it good? What do you think?

It’s actually a difficult question to answer, isn’t it? It’s kind of like, for those of us old enough to remember this, asking if you liked TV when there was only The Big Three (ABC/CBS/NBC), consuming 98% of the airwaves. Until cable TV came along, there just wasn’t a lot of comparisons you could make.

The question isn’t really, is Google a good search engine? The question is, what is Google? Is it a search engine? Eric Schmidt said no several years ago — I’m still searching for the reference, but I read the original article. According to him, it’s an advertising platform that happens to serve search as its attraction. Same as The Big Three TV channels, they were advertising engines that served entertainment as their attraction.

Back to the question — is Google a good search engine? Well, a tech company, or any company, can only optimize for one thing at a time. If you’re optimized for serving ads, you’re not optimizing for search. It serves up searches well, but it’s a great advertising platform. But a search engine? We’ll see.

Even Brin and Page, in their senior thesis, said that advertising would pollute search.

Let’s put a pin in that one for the moment.


Here is an even more dramatic version of you can only optimize for one thing at a time. Amazon should be a search engine, they would likely make even more money, but they keep showing me rows of women’s pajama bottoms when I search for men’s. They’re wasting my time and their resources.


Bing is the index that almost all the non-Google search engines use. Duck-Duck-Go (DDG), for instance, uses Bing, as do the nascent search engines, such as Neeva and Qwant.

The problem is, well, just search the same thing on Google and compare the results to any of the Bing-run search engines. The results from the Bing index are just unsatisfying.

From a developer standpoint they are a lot easier to work with than Google, but in every instance, DDG or any of the Bing-driven search engines, the results are a subset of what Google can show you.

The problem is that the Bing index is limited. None of the search engines publish real numbers about how many pages they have indexed, so it’s a bit of a mystery, but the available data point to Google’s index being the mother of them all.

These all seem to be part of the answer to why search isn’t better. Some of the implementations are funky and break the all-important always available attribute. But the unbiased question still looms large. There is no doubt that Big Search’s SERPs are influenced by their business interests. We just have no idea how.

Photo by Maxim Ilyahov on Unsplash

Other attributes

Deep is an interesting topic. I believe that Big Search is trying hard to provide good results in the depths of its SERPs, such as any page higher than 4. Certainly, the revenue incentive is reduced that far down the SERP for the businesses — they’re making less money down there, so there is less reason to skew the results. Big Search is good for broad, shallow searches, but they don’t do depth too well. Searching deeply into a specific topic frequently requires a different approach, as researchers will tell you.

How could we fix that? What about a set of easily accessible and understandable controls that could turn the butter knife into a laser scalpel? Big Search has little to no incentive to offer this. But it sure would be valuable to those of us who are more than just shallow searchers.

Ultimately it comes down to a question of how to surface content that appears low-value to most people but is really important to me? In order to provide that, a search engine would have to have a different business model, one not driven by advertising — back to Paige and Brin’s early work.

Wildcard enabled. The major search engines do a good job of this. Where this is lacking is when you off the engines and go to sites that don’t search well, that things break down. Most of us have had the frustration of being on a company’s site, forced to search down their funnel rather than just getting what we want. A trick I have learned is to go back to Big Search and use the site: operator, for example, search for site: nyt.com Swisher and Biden, and you get results just from the NYT on anything that has both names.

So I hope that someday soon, every CEO of every company has to find something on his own company’s support page as a consumer. Maybe it should be a national holiday.

Here, for instance, is an example of the support page of one of my favorite apps — references deleted because it is one of my favorites:

Image from the author

What’s missing? A way to find what I want without paging through every article until I find what I need. Search.

Let’s address the unbiased attribute for a moment because it’s really important. It’s at the core of the distrust around search. Here’s what Sergey Brin and Larry Page had to say in 1998:

The goals of the advertising business model do not always correspond to providing quality search to users.

That may just be truer today than ever before and harder to determine.

How much are Google’s financial incentives biasing the results they provide us? We may never know. How the algorithms work is secret. What biases the results is hidden and completely out of our control. And now Google couldn’t change its business model even if it wanted to. But that’s what it would take to provide search that causes no harm, is minimally invasive, and doesn’t suck up as much energy as Zambia.

Perhaps, now that this is becoming so apparent, we can start asking for something better.

If Yahoo was Search 1.0, Google is Search 2.0.

Perhaps it’s time for Search 3.0?

Watch this space.

Storyteller, seeker, always curious, work-in-progress

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store