Table of contents10
Two years ago, a friend called me in a panic. Her food blog had been climbing steadily for months — and then, over three weeks, her Google traffic fell by almost half. Nothing had changed on the site. Same recipes, same posting schedule. She was convinced Google was punishing her for something.
I am Sarah Mitchell, and I have spent years cleaning up WordPress sites that stopped growing for no visible reason. When I opened her Google Search Console, the answer was staring at us: 127 crawl errors. Googlebot kept hitting dead ends, broken redirects, and pages her server refused to serve. No penalty. No mystery. Just a site Google could not walk through anymore.
That is the fear every site owner should take seriously. Crawl errors are silent. Your site looks perfect to you, while Google quietly gives up on whole sections of it.
In this guide, I will show you the exact process I use on every site I touch: how to find every crawl error in five minutes, how to fix each type correctly — 404s, soft 404s, server errors, redirect loops, and robots.txt blocks — and the one validation ritual that makes sure each fix actually sticks. If you run a WordPress site and you have never checked for crawl errors, this guide is for you. Ready? Let us get started.
You do not need a developer, a paid tool, or any coding to do this. Google Search Console is free, and every fix in this guide happens in your WordPress dashboard or with copy-paste code I give you.
Here is what this guide covers:
- What crawl errors actually are (and why they steal traffic quietly)
- How to read the Pages report in Search Console like a pro
- Fixing “Not found (404)” errors the right way
- The tricky ones: soft 404s and redirect errors
- Server errors (5xx): when your host is the problem
- “Blocked by robots.txt”: stop accidentally hiding your own site
- How to validate the fix so Google trusts you again
- How to keep crawl errors from ever coming back
Ready? Let us get started.
What Crawl Errors Actually Are (and Why They Steal Traffic Quietly)
A crawl error is simply Googlebot trying to open one of your pages and failing. Think of Googlebot as a librarian walking through your website. Every time she opens a door and finds an empty room, a locked door, or a hallway that loops back on itself, that is a crawl error. She does not complain to you. She just stops exploring as deeply.
Here is the deal: Google has a limited appetite for your site. Every crawl budget article teaches the same truth — when Googlebot wastes time on broken pages, it spends less time on your good pages. New posts take longer to get indexed. Updates take longer to appear in search results. Your whole site slows down in Google’s eyes.
And here is what most site owners get wrong: crawl errors are not a Google penalty. Your site is not being punished. It is being misunderstood. That is actually good news, because a penalty is hard to lift, while a crawl error is just a broken door. Fix the door, and Google walks right back in.
There are five types you will meet in Search Console. A 404 (Not found) means the page does not exist. A soft 404 means the page exists but Google thinks it is useless — often a thin page that says almost nothing. A server error (5xx) means your server failed to respond. A redirect error means a chain of redirects broke or looped. And blocked by robots.txt means you (or something on your site) told Google not to enter.
I fix them in a specific order every time: find, diagnose, fix, validate. Never fix before diagnosing, and never skip validating. That order is the whole secret.
Step 0: Open the Pages Report and Learn to Read It
Before touching anything, you need a map. Google Search Console gives you one for free, and reading it correctly is half the battle.
Log into Search Console, pick your site, and go to Indexing, then Pages. This is your Pages report. It sorts every URL Google knows about into buckets. The buckets that matter to you are the “Not indexed” ones — that is where crawl errors hide.
Click into “Not indexed” and you will see reasons listed with counts, like “Not found (404)”, “Server error (5xx)”, or “Page with redirect”. Click any reason and Google shows you the affected URLs. This is your hit list. Here is my personal routine the first time I audit a site: I click each error reason, sort by the number of impressions (the traffic column), and fix the high-traffic ones first. A 404 on a page that earned 500 clicks last month matters far more than a 404 on a page nobody ever visited.
Now, the single most important skill in this guide: click one of the affected URLs and open the URL Inspection tool. Paste the URL into the inspection bar at the top of Search Console and press enter. Google will tell you the last time it crawled the page, what it found, and whether the page is currently in the index. This is your diagnosis machine. Every fix in this guide starts with “inspect the URL first.”
One more thing people miss: look at the Crawl stats report too (Settings, then Crawl stats). If your average response time is climbing or Googlebot’s crawl requests are dropping, you have a server-level problem brewing — and no amount of fixing individual 404s will help until the server is healthy.
Fixing “Not Found (404)” Errors the Right Way
The 404 is the most common crawl error on WordPress, and most of them come from the same three causes: a page you deleted, a URL you changed, or a link you typed wrong. Here is how I handle each one.
First, diagnose. Open the broken URL in a new browser tab. Does it show your 404 page? Good — that confirms it is really gone. Now ask: did I delete this page on purpose, or did it break by accident? Check where the 404 is being linked from — the Pages report in Search Console sometimes shows the referring page, and a quick site search for the URL usually finds the broken link.
If you deleted the page on purpose, do not just leave the 404 hanging there. That is like removing a shop from a mall and leaving its sign up. Set up a 301 redirect — a permanent redirect — pointing the old URL to the most relevant live page. A 301 tells Google “this page moved here permanently,” and it passes most of the old page’s link value to the new one. Never redirect everything to your homepage. Google treats mass homepage redirects as soft 404s, which defeats the purpose. Send each dead URL to the closest living relative.
How do you create the redirect? The technique matters more than the tool. If your server runs Apache, you can add one line to your .htaccess file: Redirect 301 /old-page/ /new-page/. That is fast, reliable, and needs no plugin. If you are not comfortable editing server files, use any reputable redirect plugin — the concept is identical everywhere: source URL in, destination URL out, 301 status. One honest disclosure: test every redirect after creating it by visiting the old URL in an incognito window. I have seen hundreds of redirects that pointed to pages which were themselves deleted. Always verify.
If you changed a URL by accident — maybe you edited a slug and did not notice — the fix is even simpler. Change the slug back. WordPress keeps the old slug out of the way, so restoring it is a ten-second job in the post editor.
If you have dozens or hundreds of 404s, do not fix them one by one. Export the list. Search Console lets you export the affected URLs, and then you can work through them in a spreadsheet, grouping similar URLs and redirecting whole patterns. A site I audited last year had 400 404s from a deleted product category. One redirect rule for the whole category path fixed all 400 at once. Bulk thinking beats bulk clicking.
Bonus Tip: After fixing 404s, check your internal links. A redirect fixes the symptom for Google, but your own pages should link to the new URL directly. Update the links in your content so visitors never ride the redirect at all.
The Tricky Ones: Soft 404s and Redirect Errors
Soft 404s are sneaky because your site looks fine. The page loads. It returns a 200 status code. But Google looks at it and says, “There is nothing useful here.” Then it labels it a soft 404 and refuses to index it. This usually happens with thin pages: tag archives with one post, empty search result pages, “coming soon” pages, or product pages where the product was removed but the page shell remains.
The fix depends on the page. If the page has no value, delete it and redirect it to something useful — do not leave thin pages lying around. If the page should have value, give it value. I once found a client’s entire set of city landing pages flagged as soft 404s because each one had only a headline and a contact form. We added real content — 300 words about the service in that city, a photo, one testimonial — and every single page got indexed within two weeks. Google was not being unfair. The pages really were empty.
Redirect errors are a different beast. They happen when redirects chain too long (A redirects to B redirects to C redirects to D), when they loop (A redirects to B redirects to A), or when a redirect points to a URL that itself errors. Googlebot follows a few hops and then gives up. The technique here is simple but non-negotiable: every redirect must go directly to the final URL in one hop. Audit your redirect chains by pasting the original URL into a redirect-checker mindset — or just open it and watch the address bar jump. If it jumps more than once, flatten the chain. I have seen sites where a page moved three times over five years, creating a four-hop chain. Pointing the original straight at the final page fixed it in minutes.
Also watch for redirects to blocked or deleted pages. A redirect to a page that returns a 404 or is blocked by robots.txt is a redirect error waiting to happen. Every redirect destination must be a live, crawlable page. This is why the diagnose-first habit matters so much.
Server Errors (5xx): When Your Host Is the Problem
A 5xx error means Googlebot knocked on your server’s door and the server said “I can’t deal with this right now.” Sometimes it is a one-time hiccup. Sometimes your site is genuinely falling over. Telling the difference saves you from both panic and complacency.
First, the URL Inspection test. Paste the affected URL into Search Console’s URL Inspection tool and click Test live URL. If the live test loads fine, the error was probably a temporary timeout — Googlebot arrived during a busy moment. Note it, and move on. If the live test also fails, you have a real problem, and it is on your server, not in your content.
What causes real 5xx errors on WordPress? The usual suspects, in order of how often I see them: PHP memory exhaustion (a plugin or theme eating more memory than your plan allows), too many database queries (bloated plugins hammering the database on every page load), a crashing plugin (one bad update can take down whole sections), and a genuinely overloaded server (shared hosting buckling under traffic spikes).
Here is my diagnostic order. First, check whether your site is actually down for visitors right now — if it is, this is an emergency, not an SEO task. Second, look at your hosting control panel’s error logs; PHP fatal errors leave fingerprints there. Third, deactivate your most recently updated or installed plugins one at a time and re-test. Fourth, if nothing in WordPress explains it, contact your host with the exact timestamps from Search Console. Hosts can check server-side logs you cannot see.
One pattern I see constantly: 5xx errors that only happen during crawl spikes. Everything works for visitors, but when Googlebot arrives with dozens of requests per minute, a weak shared server buckles. If your Crawl stats report shows errors clustering at certain hours, ask your host about crawl rate and server resources. Sometimes the fix is a hosting upgrade, not a WordPress tweak. Be honest with yourself about that.
“Blocked by robots.txt”: Stop Accidentally Hiding Your Own Site
This one hurts because it is self-inflicted. Your robots.txt file tells search engines which parts of your site they may crawl. A wrong line in that file can hide your entire site from Google — and I have seen it happen to experienced site owners.
The classic accident: WordPress has a setting under Settings, then Reading called “Discourage search engines from indexing this site.” It is meant for sites under construction. But I have lost count of the live sites I have audited where this box was still checked months after launch. When it is checked, WordPress serves a robots.txt that blocks everything. If your whole site suddenly drops out of Google, check this box first. It takes ten seconds and fixes the scariest-looking problem in this guide.
The subtle version is blocking specific paths you actually want indexed. The Pages report will show “Blocked by robots.txt” with the affected URLs. Common culprits on WordPress: blocking /wp-content/ too aggressively (which can block your images and CSS, hurting how Google renders pages), or leftover rules from a staging site that someone copied to production.
Here is a technique most guides skip: stop Google from crawling your internal search URLs. WordPress search pages (yoursite.com/?s=keyword) generate an infinite number of thin, low-value URLs, and Googlebot loves to waste crawl budget on them. Add these two lines to your robots.txt:
Disallow: /?s=
Disallow: /search/This does not hurt your real content at all — it just stops Google from wandering through your search results. On content-heavy sites, this one change can noticeably improve how quickly new posts get crawled, because Googlebot stops wasting visits on search pages.
Important nuance: blocking a URL in robots.txt stops Google from *crawling* it, but it does not guarantee Google will not *index* it. If other sites link to a blocked URL, Google may index the URL without its content. So use robots.txt for crawl control, and use noindex for pages you truly want out of the index. They are different tools for different jobs.
After any robots.txt change, use Search Console’s robots.txt tester (under Settings) to verify the new rules behave the way you expect, then re-check the Pages report after a few days.
Validate the Fix (Don’t Skip This Ritual)
Here is where most people fail, and it is the technique I promised in the intro. Fixing an error does not tell Google about it. You have to ask Google to come back and check — and you have to do it in the right order.
The confirm-before-validate ritual: Before you click anything in Search Console, open the fixed URL in a new private browser tab. Does the page load correctly? Does the redirect go to the right place? Is the robots.txt rule really gone? Confirm it with your own eyes first. I learned this the hard way years ago when I clicked “Validate fix” on a batch of 404s I had redirected — except one redirect rule had a typo, and Google re-crawled a still-broken page. The validation failed, the error clock restarted, and I waited another two weeks. Never click Validate on a page you have not personally confirmed.
Once confirmed, go back to the Pages report, click the error reason, select the fixed URLs, and click Validate fix. Google then re-crawls those URLs over the coming days and updates the report. Validation is not instant — give it several days, sometimes a couple of weeks on large sites. If validation fails, Google tells you which URLs still have problems, which is genuinely useful feedback, not a punishment.
Note: You do not need to validate every single URL individually. Validating the error reason covers the URLs in that group. Work in batches: fix all the 404s, confirm a sample, validate the group, then move to the next error type.
And here is the mindset shift that separates site owners who stay healthy from those who keep firefighting: a clean Pages report is a snapshot, not a trophy. New errors appear whenever you delete pages, change URLs, or update plugins. The goal is not zero errors forever — it is a routine that catches them early.
How to Keep Crawl Errors From Coming Back
Fixing errors once feels great. Keeping them fixed is what actually protects your traffic. Here is the maintenance routine I give every client, and it takes about twenty minutes a month.
Monthly: Open the Pages report and scan for new errors. Fix anything with traffic first. Check that your sitemap is still submitted and readable — sitemap errors and crawl errors often arrive together, and a broken sitemap makes everything harder for Google. If you want the full sitemap walkthrough, read our guide on WordPress sitemap errors and how to find and fix them.
Every time you delete or move content: Create the redirect *before* you delete the page, not after. This single habit prevents most 404s from ever existing. I also keep a simple spreadsheet of every redirect I create — source URL, destination URL, date, reason. Six months later, when something looks odd, that log is worth its weight in gold.
Every time you change SEO settings: A misconfigured SEO plugin is one of the fastest ways to accidentally noindex half your site or block crawling. After any settings change, inspect three URLs in Search Console: your homepage, one post, one page. Thirty seconds of checking prevents weeks of invisible damage.
Quarterly: Run a full SEO audit without paying for tools and review your crawl budget health. Crawl errors and crawl budget are two sides of the same coin — errors waste Google’s attention, and a bloated crawl wastes it further. Sites that stay on top of both get indexed faster and rank more steadily. That is not a theory. It is what I see on every site I maintain.
Frequently Asked Questions
How long does it take Google to recognize my crawl error fixes? After you click Validate fix, Google re-crawls the affected URLs over several days. Small sites often clear within a week. Large sites can take two to three weeks. If validation fails on some URLs, fix those specifically and validate again — the process is iterative, not one-shot.
Will crawl errors hurt my rankings directly? Not as a penalty, but they hurt indirectly in ways that matter. When Google cannot crawl a page, it cannot index updates to it. When crawl budget is wasted on errors, your important pages get crawled less often. Over time, that means slower indexing, stale search snippets, and weaker performance. Fix errors because they slow you down, not because Google is punishing you.
Should I fix 404 errors on pages that never got any traffic? Fix them if they are linked from anywhere — your own site or external sites — because those links carry value and visitors. For orphan 404s with no links and no traffic, a simple redirect to the closest relevant page is still good hygiene, but do not lose sleep over them. Prioritize by traffic and links, always.
What is the difference between a 404 and a soft 404? A 404 means the page does not exist — the server says so clearly. A soft 404 means the page loads fine but Google judges it to have no useful content, so it treats it like a 404. Fix a 404 with a redirect or restoration. Fix a soft 404 by either deleting the thin page (with a redirect) or adding real, useful content to it.
Can too many redirects cause crawl errors? Yes. Long redirect chains and redirect loops are their own crawl error category. Keep every redirect to a single hop from the old URL to the final destination, and never redirect to a page that is itself deleted, blocked, or redirecting elsewhere. Audit chains whenever you move content that was moved before.
Final Thoughts
My friend’s food blog recovered. It took about a month — two weeks to find and fix everything properly, and two more weeks for Google to re-crawl and trust the site again. Her traffic did not just return to where it was. It grew past it, because Google could finally reach pages that had been invisible for months.
That is the real lesson of crawl errors. They are not dramatic. Nobody hacks your site, nobody sends you a warning. Your pages just quietly stop being reachable, and your traffic quietly stops growing. The site owners who win are not the ones with the cleverest SEO tricks. They are the ones who check the Pages report once a month and fix what they find.
You now know the whole process: find errors in the Pages report, diagnose each URL before touching it, fix by type, confirm with your own eyes, and validate the fix. Do that loop, keep the monthly habit, and Google will never have a reason to stop walking through your site.
Found a weird crawl error this guide did not cover? Drop it in the comments — I read every one, and the strange cases are how we all learn.
2 comments