1) Control how Google crawls and indexes your site
Debugging SEO is easier when you think in three stages: Crawling → Indexing → Serving. A URL can fail at any stage (blocked resources, duplicates, quality issues, or noindex signals).
- Crawling: Googlebot fetches HTML and critical resources (JS/CSS/images) and renders the page.
- Indexing: Google understands content, chooses a canonical, and stores it.
- Serving: Pages are ranked and shown for relevant queries.
Use URL Inspection in Search Console to render the live page and confirm Google sees what you expect.
2) Duplicate content and canonical URLs
Duplicate content is normal (parameters, alternate routes, print views). The key is telling Google the canonical version you want indexed.
- Ensure each important page has a single canonical URL (consistent protocol, host, and path).
- Avoid multiple URLs with the same content unless you intentionally support alternates (like languages).
3) Make sure resources are crawlable
If Google can’t fetch important resources (CSS/JS/images), it may render an incomplete page and misunderstand content. Confirm that:
- Resources are accessible to an anonymous user (no login wall for public pages).
- They are not blocked by
robots.txt. - They return 200 (avoid 403/404 for critical assets).
4) Use robots.txt for crawling control (not indexing)
Use robots.txt to manage crawling (what to fetch), and use noindex (meta robots / X‑Robots‑Tag) to manage indexing.
- Block crawling for truly unimportant or state-changing URLs (account creation, posting comments, carts).
- Keep it simple; overly complex rules create accidental blocks.
5) Sitemaps: prioritize discovery and recrawl
Sitemaps help Google discover your important URLs and prioritize recrawling—especially when:
- Pages change frequently
- Some pages aren’t well-linked internally
- You publish non-text content (images/video)
Maintain accurate lastmod and avoid including URLs you don’t want indexed.
6) Multi-lingual / international sites (hreflang)
If you have multiple languages or country variants, implement hreflang so Google can show the right version to the right users.
- Provide reciprocal hreflang links between alternates.
- If content changes by locale automatically, test how bots experience those variants (caching, geolocation).
7) Migrating a page or a site
Migrating a single URL
- Use 301 for permanent moves; use 302 only for temporary moves.
- Update internal links to point at the new URL.
Custom 404s (avoid soft-404)
- Removed pages should return a real 404 (not 200 with an error message).
- Custom 404 UI is fine as long as the HTTP status is truly 404.
Migrating an entire site
- Implement full 301 mapping (old → new) before launch.
- Update sitemaps and submit the new sitemap in Search Console.
- Monitor crawl errors, indexing, and traffic fluctuations during the transition.
8) Crawling & indexing best practices
- Make links crawlable: use real
<a href>links for navigation. - Nofollow when appropriate: paid/untrusted/user-generated links should not pass signals.
- JavaScript: keep SEO-critical content available server-side (SSR) where possible.
- Infinite scroll: provide paginated URLs for discovery and crawling.
9) Help Google understand your pages (structured data)
Prefer text over critical information embedded only in images. Add structured data when it’s appropriate (Course, Organization, FAQ, HowTo, etc.) and validate using rich-results tools.
10) Manage the user experience (rankings + conversions)
- HTTPS: always use HTTPS for trust and security.
- Performance: optimize Core Web Vitals (LCP/INP/CLS), reduce JS, and compress images.
- Mobile: make sure the primary experience is mobile-friendly (mobile-first crawling).
11) Control search appearance
- Use clear titles that match on-page headings and intent.
- Write descriptions that summarize the page accurately (no clickbait).
- Use favicon and (where relevant) dates for articles.
12) Use Search Console as your control panel
- Page Indexing: learn why URLs aren’t indexed (blocked, duplicate, canonical, quality).
- URL Inspection: test live rendering and request indexing for important updates.
- Core Web Vitals: track field performance signals.
Tip: keep a lightweight change log when you ship major SEO-impacting changes (titles, URL structure, template/layout changes, redirects). It makes debugging Search Console issues dramatically faster.