Technical SEO
Log-File SEO Analysis Workflow
A practical workflow for using server logs to understand crawler behavior, crawl waste, status-code patterns, rendering risk, and crawl focus.
Log-file SEO analysis uses server request logs to see how crawlers actually request a site. It is useful when crawl behavior, indexation, large templates, redirects, error patterns, or bot access are hard to understand from crawls and Search Console alone.
Most small sites do not need deep log analysis every month. It becomes valuable when the site is large, changes often, has many duplicate URL patterns, depends on JavaScript rendering, has migration risk, or needs evidence about what search crawlers spend time requesting.
What server logs can answer
| Question | Log Evidence |
|---|---|
| Is Googlebot requesting important pages? | Requests by verified crawler and URL group. |
| Is crawl activity wasted on low-value URLs? | Requests to filters, parameters, search pages, old archives, duplicates, or redirected URLs. |
| Are crawlers hitting errors? | 3xx, 4xx, 5xx, timeout, and blocked-request patterns. |
| Are important pages crawled after updates? | Last request date by page group. |
| Are redirects slowing discovery? | Chains, repeated old URL requests, and destination status. |
| Are bots reaching canonical pages? | Crawl share by canonical URL group versus duplicate patterns. |
| Is crawl behavior changing after a migration? | Before/after request mix and response-code mix. |
Logs show requests, not rankings. Treat them as technical evidence that must be connected to indexation, page quality, and business outcomes.
Data to collect
Ask for a log export that includes:
- Timestamp.
- Request URL.
- HTTP method.
- Status code.
- User agent.
- IP address or verified crawler identity.
- Response bytes.
- Response time where available.
- Referrer where available.
- Hostname and protocol.
Collect enough history to compare patterns. For migrations and traffic drops, use at least two windows: before the event and after the event. For ongoing crawl focus, 30 to 90 days is usually more useful than a single day.
Verify crawler identity
User agents can be spoofed. Do not assume every request claiming to be Googlebot is actually Googlebot.
Use one of these methods where available:
- Search Console Crawl Stats for Google-owned crawl data.
- Reverse DNS lookup and forward-confirmation for sampled IPs.
- Published crawler IP ranges when the crawler provides them.
- Your CDN or log tool’s verified bot field if documented and trusted.
Separate verified Googlebot from other bots, SEO crawlers, uptime monitors, scrapers, ad bots, and internal tools.
Segment URLs
Raw logs are noisy. Group URLs before analysis.
Useful segments include:
- Homepage and brand pages.
- Product, service, category, or collection pages.
- Guides, articles, tools, templates, and research assets.
- Local pages.
- Pagination.
- Facets and filters.
- Internal search pages.
- Parameters and tracking URLs.
- Old URLs and redirects.
- Error URLs.
- Static assets.
- API routes.
- Staging or unexpected hostnames.
Segmentation turns a giant log export into an SEO diagnosis.
Core metrics
Track these by crawler and URL segment:
| Metric | Why It Matters |
|---|---|
| Requests | Shows crawler attention by group. |
| Unique URLs requested | Shows breadth and possible duplicate expansion. |
| Crawl share | Shows whether important segments receive enough attention. |
| Status-code mix | Reveals redirects, missing URLs, blocked URLs, and server errors. |
| Last crawled | Helps validate recency after updates or migrations. |
| Bytes downloaded | Helps identify heavy templates or wasteful asset crawling. |
| Response time | Helps find slow templates that may reduce crawl efficiency. |
| Parameter/request patterns | Reveals crawl traps and duplicate URL spaces. |
Do not optimize for “more crawling” alone. The goal is better crawl focus.
Diagnosis patterns
| Pattern | Likely Meaning | First Action |
|---|---|---|
| High crawl share on filters or parameters | Crawl waste or duplicate URL discovery. | Review internal links, canonicals, robots rules, and sitemap policy. |
| Important pages rarely crawled | Weak internal discovery, low authority, stale sitemap, or low perceived value. | Strengthen internal links and ensure indexable canonical URLs. |
Many repeated 301 requests | Old URLs still discovered internally, externally, or in sitemaps. | Update internal links and sitemap URLs to final destinations. |
Many 404 requests from internal links | Broken discovery paths. | Fix internal links and route patterns. |
Many 5xx responses to bots | Server instability or crawl throttling risk. | Investigate infrastructure, cache, and bot handling. |
| Crawl spikes on staging or query URLs | Exposure, malformed links, or parameter traps. | Block or fix exposure carefully; avoid blocking URLs that must be indexed. |
Logs should lead to a specific technical hypothesis, not a generic “crawl budget problem” label.
Pair logs with other evidence
Use logs with:
| Source | Adds |
|---|---|
| Search Console Crawl Stats | Google’s aggregate crawl totals, response codes, file types, and host status. |
| Search Console Performance | Whether crawled page groups earn impressions and clicks. |
| Page indexing report | Whether requested pages become eligible for indexing. |
| Site crawl | Internal links, canonicals, status codes, meta robots, and duplicate templates. |
| XML sitemaps | Whether crawlers are sent to canonical, indexable URLs. |
| Analytics and CRM | Whether crawled/indexed pages create qualified outcomes. |
Logs say what crawlers requested. They do not say whether the page deserved to rank.
Analysis workflow
- Define the question: migration validation, crawl waste, indexation issue, error spike, or large-site crawl focus.
- Export logs for the right date window.
- Verify bot identity.
- Normalize hostnames, protocols, trailing slashes, parameters, and status codes.
- Segment URLs by page type and SEO role.
- Compare request share, unique URL count, status-code mix, response time, and last-crawled date.
- Cross-check with Search Console, crawl data, sitemaps, and analytics.
- Write a diagnosis by URL group.
- Prioritize fixes that improve access to important canonical pages.
- Recheck logs after fixes ship.
The output should be a short set of crawl-focus actions, not a giant spreadsheet nobody reads.
Common mistakes
- Treating all user-agent strings as verified search crawlers.
- Looking at raw URL counts without segmenting page types.
- Calling every large site issue a crawl budget issue.
- Blocking URL patterns before understanding indexation and internal links.
- Ignoring response time and server errors.
- Measuring requests without checking whether crawled pages are useful.
- Forgetting that image, video, ads, and other bots can appear alongside search crawling.
Useful source references
- Google Search Central: What crawl budget means for Googlebot
- Google Search Central: Googlebot and verifying Googlebot
- Google Search Central: Troubleshoot crawling errors
For the next layer, use Crawl Budget Diagnostics to turn log findings into a crawl-focus plan.