Tim Bray, former Google employee, currently working at Amazon, writes:
I think Google has stopped indexing the older parts of the Web. I think I can prove it. Google’s competition is doing better.
It’s an interesting theory for sure, but it seems hard to back this up with any tangible evidence. How would you even test this? You can pick specific web sites to test this with, but that will always be an incredibly small – infinitesimally, unbelievably small – subset of web sites, and there’s no way to extrapolate any of that to the web as a whole. To make matters worse, Google tailors search results to the information they have on you, making this even harder.
…since they got rid of easy boolean searches, and began putting their moronic “suggestions” at the top of the page. All I want are two search engines: one for exact text searches; one for giving me actual website addresses for companies, and not bestlistofcompanies2018lololol.menu and their delightful, useless top ten of anything.
I completely agree. I cannot quite put my finger on what got worse, though it feels as if Google had given up proper (text-based) search and just used clickthrough data. Which might be a useful resource, but it is not a panacea. In my case, the results have gotten much worse, compared to a few (?) years ago.
Thom Holwerda,
Shouldn’t be too difficult to find an SEO guys who knows, they’re testing these things all the time.
I don’t know if the author has evidence that those pages were crawled originally?
Through my clients though, I’ve experienced how difficult it can be to get pages to show up in google. It’s frustrating that google not show some of our pages with literal text matches in favor of more popular sites with fuzzy matches. This is true even of pages that have been crawled. They only seem to show up after you type the domain name in addition to the query.
Alas, it’s the epitome of the network effect, sites get good google rankings because they’re popular, but conversely they’re popular because google ranks them highly. Sites that don’t get listed have to find a way to become popular beyond just putting up content. Google won’t even crawl all the pages if you aren’t popular enough.
While it’s officially looked down upon (aka the notorious “penguin” updates), and despite google’s stated goal of fighting web spam, increasing inbound links is still the main strategy that I’ve seen work. If you get a popular site to link to you, suddenly all of your links start getting listed (which is exactly what happened here). Google would have to concede that nothing whatsoever changed with regards to the quality of his content quality, the only thing that changed was that popular sites began linking to him.
Heck, osnews.com, with a google page rank of 6, could save many sites from the slums of the internet just by writing an article for them. Not saying you should, but that’s the way it works.
Edit:
As far as literal text matches go, one reason google may ignore it is that if google always showed literal text matches, this would be easy to exploit. But it just sucks for people who are looking for a specific unpopular site with a literal text match and not being able to find it. One of my huge gripes with google is when I type something in quotes, they still return popular sites/pages that don’t match the criteria?! This is NEVER useful to me, I used the quotes for a reason, so google gets a failing grade with quotes!
Edited 2018-01-16 23:09 UTC
I am too cool for Google, therefore I use duckduckgo.com. Sometimes I am too cool even for that and will therefore use http://3g2upl4pq6kufc4m.onion.
…I don’t even want to click that link
I guess you are not a curious kind of person. But I will spoil it for you anyway: It is DuckDuckGo on tor.
It has nothing to do with curiosity, but much more with common sense. Clicking on obfuscated links in the days of Meltdown and Spectre feels as safe as surfing the .ru domain…
Actually the link is not obfuscated at all. This is what normal onion URL’s look like.
In addition to that you could always use virtualization.
.ru domains are most likely fairly safe, it’s not like Russian cybercriminals can’t get .com, .net and so on domains.
Plus supposedly they have an unwritten agreement with authorities not to target local population…
It’s just terribly awful. I find most of the time it ends up searching for stuff completely irrelevant to the search terms i put into it. It’s so determined to search for what it thinks i want, instead of what i ask it to, i end up switching to competitors to actually get the results i want. It drives me fucking ballistic!
for my work on our #t2sde open source, build from source system development environment (https://t2sde.org) I constantly see obscure compile errors, due new or old gcc, glibc, musl, cross compilation, mips, powerpc, arm, assembly you name it. A decade ago Google always found other projects and bug reports for such errors. Nowadays I often get zero or not many hits. For months I already had the feeling Google’s search results for hard core tech became worse, …
Edited 2018-01-17 16:01 UTC
here is what I was working on over christmas and new year and needed google for – Sgi Octane MIPS64/Linux: https://www.youtube.com/watch?v=AU_RV8uoTIo
A little anecdote:
My tiny almost irrelevant blog (on web dev stuff) back in the day, I experimented with ads. Not because I was trying to make money, but because I wanted to see what was involved with adding them on a WordPress site.
What I found was after adding Google’s AdSense to my site, the reported visits to my site jumped up a bit. When I removed them, they went right back down.
Now, I don’t know how much of this is coincidence, but Google’s revenue model is dependant on selling ads. They don’t make any money if they send you to a relevant site, that they don’t have any ads on. I don’t take the requirements of a business’s revenue model lightly – I’m certain that it has a very large impact on the decisions, both macro and micro made through a company (including Google and Facebook – both ad driven).
I am not sure what is his claim:
A. His old articles were never indexed by Google (so they didn’t stop ‘now’)
B. The old articles used to be in Google’s index database but got removed at some point (again, something else than stopping now)
In contrast to few posters above, I think Google is great.
Almost always it gives me what I want in first search, sometimes after one refinement of terms …heck, it usually gives too many relevant links / I kinda waste time reading interesting info.
Maybe I ~subconciously learned to ask it the “proper” way, but I like to think I represent the silent majority that’s satisfied with the results.