Stand Alone Service Type:
Show Your Clients Exactly How Google Crawls Their Site
White-labeled server log analysis revealing how Googlebot actually crawls your client’s site — delivered as a prioritised action plan under your agency brand.
Standard SEO tools — crawlers, Search Console, analytics — show what’s happening on the surface. Server log files show what’s actually happening. Every request Googlebot makes, every URL it visits, how often, in what order, and where it stops. It’s the only way to see whether your client’s most important pages are getting the crawl attention they deserve, and where budget is being wasted on low-value URLs that should never have been crawled in the first place. We analyse your client’s server logs and deliver a clear, prioritised action plan — white-labeled under your agency brand — that you can present directly in your next client review.
Unlike simulated crawls or surface‑level reports, log file analysis allows you to verify actual bot activity and uncover issues such as unexpected 404 errors, server errors, long redirect chains, or orphan pages that might otherwise slip through your site architecture. By identifying these patterns, you gain the insights needed to optimize internal linking, fix crawl traps, and ensure that your most valuable content receives adequate crawl attention.
For large sites or those preparing for major technical changes or migrations, regularly analyzing log files can be transformative. It helps pinpoint inefficiencies that impede indexing, reveals bot behavior trends that inform crawl priority adjustments, and surfaces technical problems before they impact performance.
To deepen your understanding of the technical side and benefits of log file analysis, we recommend referencing the Ahrefs SEO Glossary article on the topic, which provides a practical overview and real‑world context for why this process is vital for comprehensive SEO success: https://ahrefs.com/seo/glossary/log-file-analysis.
What This Service Includes
Server logs reveal the truth about how search engines actually crawl your site. Unlike analytics, log files show every request—including pages Google visits but doesn’t index, crawl errors, and bot behavior patterns.
This service analyzes your server logs to uncover crawl inefficiencies and opportunities invisible in other tools.
Order this for clients who:
– Have large sites where indexation is patchy or slow
– Show high numbers of ‘discovered — not indexed’ pages in Search Console and standard crawls haven’t identified the cause
– Are preparing for a migration and need to understand current crawl behavior before changing anything
– Have recovered from a Google penalty or algorithm update and need to understand what bots are doing now
– Have had technical SEO work done but haven’t seen the expected indexation improvement
– Run e-commerce stores with crawl budget concerns across large product catalogs
What We Deliver For Your Client
Log Collection & Normalization
– Collection and processing of the client’s server logs across all supported formats (Apache, Nginx, IIS)
– Isolation of Googlebot and Bingbot activity from general user and bot traffic
– Data cleaning and normalization to ensure accurate analysis — confirmed with you before we begin
Crawl Pattern Analysis
– Identification of which pages on the client’s site Google crawls most and least frequently
– Discovery of important pages receiving insufficient crawl attention
– Crawl frequency mapped against the client’s priority page hierarchy
– Quantification of crawl waste — low-value pages consuming budget that should be going to priority content
Bot Behavior Insights
– Googlebot vs Bingbot comparison
– Mobile vs desktop bot activity
– Crawl budget allocation across templates
– Response code analysis (200, 404, 301, 500 patterns)
Indexation Correlation
– Cross-reference crawl data with Search Console indexation
– Identify pages crawled but not indexed
– Find indexation opportunities based on crawl patterns
Action Plan Development
– Prioritized list of pages needing more crawl attention — ranked by impact
– Specific robots.txt and internal linking recommendations to redirect crawl budget
– Technical issue identification with implementation guidance your developer can act on
– Full crawl efficiency improvement strategy — documented in a white-labeled report ready for client presentation
Deliverables
What your agency receives — ready to present to your client:
– White-labeled log file analysis report showing actual Googlebot crawl behavior
– Crawl waste quantification — how many URLs are consuming budget unnecessarily
– Bot behavior pattern documentation (Googlebot vs Bingbot, mobile vs desktop)
– Prioritized action plan with specific recommendations your developer can implement
– Cross-reference with Search Console indexation data to explain gaps between crawled and indexed pages
FAQs
Common Questions
Log file access depends on the hosting environment. Dedicated servers, VPS hosting, and most managed WordPress hosts provide log file access either directly or through the hosting control panel. Shared hosting plans sometimes restrict access — in those cases we can often work with partial log data, or recommend the client's host as a first step. Shopify and some other hosted platforms don't provide server log access at all, in which case we use Search Console data and crawl analysis as the primary diagnostic tools. We confirm what's available before scoping the project so there are no surprises.
For a meaningful analysis we typically need at least 30 days of log data — ideally 60 to 90 days for sites with lower traffic volumes where crawl patterns may not be immediately apparent in a shorter window. We'll confirm the right period based on the client's site size and the issues being investigated. You arrange log file export through the client's hosting — all access requests go through your agency.
A standard crawl audit shows what a crawler tool finds when it visits the site — essentially simulating what Google might see. Log file analysis shows what Google actually did. The two are complementary but different. Crawl audits identify technical issues. Log file analysis reveals whether Google is encountering those issues in practice, which pages it's prioritising, and whether recent technical fixes have changed crawl behaviour. For diagnosing persistent indexation problems that crawl audits haven't explained, log file analysis is usually the missing piece.
Yes — the comprehensive log file analysis report, crawl waste documentation, bot behavior summary, and prioritized action plan are all delivered white-labeled under your agency brand. The report is formatted to be presented directly to your client in a review meeting. Harper Media Group is not referenced anywhere in the deliverables.
Once we receive the log files, analysis and report preparation typically takes 5–8 business days depending on the volume of data and the complexity of the site. For very large sites with months of log data across high-traffic pages, we'll confirm the timeline during scoping. We don't start the clock until we confirm log file receipt and quality — incomplete or corrupted log files need to be resolved before analysis can begin.
Log File Analysis Pricing
Requires client to provide log files — note this in your SOW. Pairs with Crawl Budget Optimization.
Basic
7-9 hrs
$575
- Log file parsing & setup
- Crawl frequency by bot report
- Wasted crawl resource identification
- Top crawled vs. strategic pages gap
Standard
13-16 hrs
$1075
- Everything in Basic
- Bot behavior trend analysis (30–90 days)
- Orphaned page crawl mapping
- Cross-reference with GSC coverage
- Actionable crawl optimization plan
Premium
19-24 hrs
$1750
- Everything in Standard
- Technical barrier identification
- Crawl simulation & recommendation
- Robots.txt + sitemap alignment
- Executive summary report
Ready to Add Technical SEO and AI Optimization to Your Service Menu?
Book a free 30-minute partner strategy call. We’ll walk through your client roster, identify which services fit, show you the AI Analytics Platform live, and confirm pricing. No obligation.