How Often Does Google Crawl Your Website, and What Slows It Down

Google decides how often to visit your site, and your web host gets a vote.

Most firms never see either number. The question turns up when a new service page has been live a fortnight and still cannot be found.

Knowing how often Google crawls my website is a settings question before it is an SEO question, because Search Console already holds ninety days of the answer.

Open the screen before you buy anything to fix it.

There is no crawl schedule, and Google explains why

Nothing about it is fixed. Google's crawl budget guidance describes two forces meeting in the middle, and the first one is about your server rather than your content.

Google says its crawlers work out a crawl capacity limit, in its own words because Google "wants to crawl your site without overwhelming your servers".

The second force is demand, and Google names what feeds it.

• Size and update frequency: A site that changes weekly earns more attention than one frozen since 2023.
• Page quality: Thin pages pull the whole site's demand down over time.
• Relevance against other sites: Your demand is measured next to everybody else's, so it is never yours alone.
• Popularity: Google says popular pages get crawled more often than quiet ones.
• Staleness: Systems recrawl to pick up changes, so a page nobody edits gets visited less.

None of that produces a number you can promise a partner. It gives you a range you can shift, and one lever in it belongs to your hosting company.

Where your real crawl numbers live

Search Console holds them, in a screen almost nobody opens. Google's crawl stats report shows "statistics about Google's crawling history on your website", covering how many requests were made and when, what your server said back, and any availability problems along the way.

Getting to it takes four clicks.

1. Open Search Console and choose the property for your site.
2. Click Settings in the left menu, near the bottom.
3. Find Crawl stats and open the report.
4. Read the total requests chart before anything else.

One trap stops most firms here. The report opens only for a root-level property, so a property covering a single folder shows nothing at all.

Which property do you actually own? Agencies often verify a folder or a single subdomain, which is enough for the performance numbers and useless for this screen. Verify the domain properly, then wait a few days for data to build.

Ninety days is the window.

The report holds no more than that, so a problem from last spring has already gone.

What the host status light is telling you

It grades your server. Google runs three checks behind one traffic light, and the wording of each colour is precise about timing.

The three checks underneath are worth knowing by name, because each one points at a different supplier.

• Fetching the robots file: A blocked or missing robots file stops Google before it starts.
• Is your DNS resolving? A registrar or DNS provider fault looks the same as a site being down.
• Connecting to the server: Timeouts and refused connections land here, and this is the hosting one.

A red light is the cheapest evidence you will ever get in a conversation with a hosting company.

Which server responses make Google crawl less?

Errors and rate limits do it, and Google spells out the result. If a site responds with server errors or rate-limiting signals, Google says, "the limit goes down and Google crawls less".

The 503 response is the one worth knowing, because it is the polite version of a failure. Mozilla's reference for 503 Service Unavailable says the code "indicates that the server is not ready to handle the request" and names two ordinary causes, maintenance and overload.

The standard agrees and adds the timing. RFC 9110 defines 503 as a state that "will likely be alleviated after some delay", which is exactly what a crawler needs to hear during a planned window.

So a maintenance window served as 503 with a retry header costs you very little. The same window served as a broken page costs you crawl rate for weeks.

Discovery against refresh, and why a new page waits

Google splits every request into two jobs, and the split explains the wait on a brand new page.

Discovery covers a page Google has never crawled, in Google's words one where "the URL requested was never crawled by Google before", and refresh covers every page it already knows.

Most of your requests will be refresh. A site with sixty pages and one new one gets visited constantly and still leaves the new page waiting, because refresh traffic keeps flowing to the pages Google already knows and rates, and the one page nobody links to sits at the back of a queue it never joined properly in the first place.

Three moves shift a new page from waiting to crawled.

1. Link to it from a page Google visits often, which usually means the service page or the blog index.
2. Include it in your XML sitemap and check the sitemap was actually submitted.
3. Use the inspection tool to request indexing once, and never queue the same page repeatedly.

The report also breaks requests down by crawler, so image, video and page-resource fetches all show separately. A firm with a heavy photo library often finds most of its crawl going to pictures.

Does more crawling mean better rankings?

No, and treating the number as a score wastes a month. Crawling is how Google finds and refreshes pages, so it sits upstream of everything else and decides nothing on its own.

• A page crawled daily and thin still ranks nowhere.
• A page crawled monthly and strong holds its position fine.
• A crawl spike often means something broke and Google is retrying.
• A crawl fall usually means the server got slower or the site stopped changing.

Read the crawl chart as a health line and read average position as the performance one. The two tools in your account answer different questions, and crawl stats is the only one that reports on your server.

Watch the shape of the line.

A flat line with the odd dip is a healthy site. A jagged line with error codes climbing is a talk with your host. A line that drops and then stays down is usually the week somebody moved hosting plans or switched a caching layer on without telling anybody, and the giveaway is that the fall starts on a Tuesday and never recovers, because nothing about your content changed at all and nothing about it is going to explain the chart to you no matter how long you stare at it.

How do you tell a hosting fault from an SEO one?

Take the four checks in the sequence below, because the free ones come first and settle most cases.

1. Read the host status light and note the colour and the date.
2. Open the response code breakdown and look for 5xx or 429 climbing.
3. Compare the total request line against the week the trouble started.
4. Only then look at content, links and JavaScript rendering.

• Red light, 5xx rising: Your host is the conversation, and no amount of content work fixes it.
• Green light, requests steady, rankings falling: The problem sits in the content or the competition.
• Requests collapse with no errors: Check whether the robots file changed or a staging block went live.
• Odd crawler patterns and strange pages: Rule out a hacked site before anything else.

If the pages are missing altogether instead of merely slow to update, that is a different diagnosis and the route for not indexed at all starts somewhere else.

The one screen worth ten minutes a month

Crawl stats is the only report in Search Console that grades your supplier instead of your marketing. Ten minutes a month gives you a dated record of your host's behaviour, which is worth having long before you need it.

• Note the host status colour: A dated screenshot beats a memory in any hosting dispute.
• Watch the 5xx share: A rising share is an early warning and never a rounding error.
• Ask who owns uptime: Somebody at the firm should hold that answer before a red light appears.
• Check the crawler split: Images eating your crawl is a fixable problem.
• Look after a release: A release that broke something shows up here within days.

A firm that reads this beside its speed and uptime numbers often finds one supplier behind several symptoms, which is the point where a website audit pays for itself.

Our team reads these numbers for professional firms every month and takes the hosting conversation off the partners' desk. Ask us about website analytics if you would rather have the report read for you.

Frequently Asked Questions

Can we ask Google to visit more often?

Search Console offers no dial for it. Publishing and updating real pages raises demand on its own, and a faster server lifts the ceiling that demand runs into.

Why does the total in the report look too low?

Google says some requests may not be counted, so treat the totals as a trend line and never as an audit figure. The shape over weeks is the useful part.

Our developer wants to block Googlebot during a release. Sensible?

A short maintenance response with a retry header is the standard way to handle it, and Google copes well. Blocking outright or serving broken pages for days is what causes lasting damage to crawl rate.

Is the data live, or does it lag?

Figures arrive a day or two behind, so something published this morning may not register until tomorrow. Treat any gap under 48 hours as normal reporting delay.

Does a content delivery network change these numbers?

Requests still get counted, and the responses Google logs come from whatever answered the call. Ask which layer served the errors before you go blaming your own origin server for them.