Audit Methodology

Audit Setup
and Pre-Crawl Intelligence

What to do before you run a single tool.

Where to find it: Google Search Console > Overview | Client onboarding questionnaire | robots.txt and sitemap.xml manual review

What It Is

The pre-crawl phase is the intelligence-gathering stage — reading existing data before generating new data. This includes reviewing Search Console history (traffic trends, manual actions, coverage issues), checking robots.txt for unexpected disallows, reviewing sitemap.xml for completeness, reading any previous audit reports, and establishing baseline metrics. Agencies that skip pre-crawl phase waste hours diagnosing issues already documented in Search Console, or miss context that changes how findings should be prioritized.

Why It Matters

A site with a Manual Action needs that resolved before any other technical work begins — everything else is wasted effort until that's clear. A site that lost 60% of traffic 8 months ago needs that drop investigated before a crawl is configured. Pre-crawl intelligence sets the audit's priority order and prevents hours of work being directed at secondary issues while the primary cause sits undiscovered in Search Console.

Common Audit Failure Points

What Goes Wrong

Understanding where audits fail — and why — is the first step to executing them correctly.

Skipping Search Console Before Crawling

Starting a crawl before reviewing Search Console misses issues already documented — manual actions, sudden coverage drops, and indexation errors that should dictate the entire audit's priority order.

Not Reviewing robots.txt First

Crawling a site without manually checking robots.txt first risks running a full crawl while Googlebot is blocked from critical sections — producing misleading data about what Google actually sees.

No Baseline Metrics Established

Proceeding without documenting baseline organic traffic, indexed page count, CWV scores, and top rankings makes it impossible to measure the impact of fixes after the audit — destroying the before/after story.

No Historical Context From Client

Missing information about recent site migrations, CMS changes, or developer deployments that correlate with traffic shifts — the most common cause of misdiagnosing a technical problem as organic performance decline.

Interactive Standard Operating Procedure

The Audit Playbook (Interactive SOP)

Check off each step to track your audit progress live!

Audit Progress: 0% Completed (0/7)

1. Pull the 16-Month Search Console Performance Report

Request Search Console access and export the 16-month performance view. Note overall traffic trends, impressions vs. clicks gap, any sharp traffic drops by date, and which pages/queries drove the biggest changes. This is the audit's starting context.

2. Review the Coverage Report and Check for Manual Actions

In Search Console > Indexing > Pages, categorize all error types and note counts for each status. Navigate to Security & Manual Actions and check for any active manual penalties — if one exists, it becomes the audit's first priority before anything else.

3. Manually Review robots.txt

Navigate directly to /robots.txt on the live domain. Confirm Googlebot isn't blocked from critical content sections — product pages, service pages, blog posts, and sitemaps. Look specifically for staging Disallow rules that weren't removed when the site went live.

4. Review sitemap.xml for Completeness and Accuracy

Open the sitemap.xml and verify: it's been submitted to Search Console, it contains only canonical indexable URLs, and it doesn't include redirect destinations, noindexed pages, or error URLs. A sitemap with 400 URLs when the site has 4,000 pages signals systematic gaps.

5. Complete the Client Onboarding Questionnaire

Collect: any recent site migrations, CMS platform changes, developer deployments, URL structure changes, or agency transitions in the past 12 months. Map any dates provided against the traffic trends from step 1 — correlations here often explain drops before any crawl data is needed.

6. Document All Baseline Metrics

Record: current organic traffic (monthly), total indexed pages (from Search Console), current rich result count by type, CWV field data scores (mobile and desktop), and top 10 ranking keywords with their current positions. This baseline becomes the before state for the audit's impact measurement.

7. Set Audit Scope and Priority Order

Based on what pre-crawl intelligence reveals, define the audit's priority order. A site with a Manual Action audits reconsideration eligibility first. A site with a large indexation gap audits coverage first. A site with known CWV failures audits performance first. Never run a generic audit when pre-crawl data reveals a specific priority.

Tools

Google Search Console
Free — required access | Performance report, Coverage report, Manual Actions, Rich Results — the primary pre-crawl intelligence source
Ahrefs or Semrush
Paid | Historical traffic and ranking baselines — supplements Search Console data with competitor context and keyword-level history
robots.txt Tester
Free (in Search Console) | Tests specific URL patterns against the live robots.txt rules — faster than reading the file manually for large rule sets

Time Investment

2–3 hours

Pre-Crawl Intelligence

Never skip this phase

Non-negotiable

Pro Tip

Always check for Manual Actions before any other audit work.

A site under a Manual Action won't recover from any technical improvements until the action is resolved and a reconsideration request is approved. Fixing canonical tags, improving internal links, and optimizing page speed all produce zero ranking benefit while a manual penalty is active. Check Manual Actions first — if one exists, the entire audit becomes secondary to building the reconsideration case.

Ep 2: Crawl Configuration and Execution

Audit Setupand Pre-Crawl Intelligence