Feel frustrated that irrelevant pages dilute your site’s search performance? Or wish you could keep helpful internal pages private while showcasing your best content? Employing the noindex tag gives you granular control to cherry-pick what search engines see.
What Is the NoIndex Tag?
The noindex meta tag gives a directive to search engines not to include the tagged page in their search indexes. When a crawler encounters the noindex tag, it will drop that page from the indexing process without crawling any links or content on the page.
Pages marked with noindex will not appear in search engine results pages (SERPs). The page becomes essentially invisible to search engines.
There are a few key reasons webmasters implement noindex tags:
- To avoid duplicate content issues by noindexing alternate versions of pages;
- To hide low-quality or thin-content pages;
- For pages not intended for search visitors, like login pages;
- Temporarily, while working on a new site section.
Whatever the reason, the mechanics of implementing a noindex tag are straightforward once you understand the methods.
Different Aspects of NoIndex Tags
Before diving into how to noindex a page, you should understand nuances of the noindex tag and how it differs from related indexing directives.
Meta Robots Tags
The most common application of the noindex directive is through HTML meta robots tags. The robots meta tag goes inside the head section of a webpage and looks like:
<meta name="robots" content="noindex">This tag tells all crawlers encountering it not to add the page to their search indexes.
There are also a few variations of the meta tag structure:
<meta name="robots" content="noindex,follow">
<meta name="googlebot" content="noindex">The first example adds a follow value, telling search engines they may follow links on the page. The second targets Googlebot specifically.
Response Header Option
In addition to meta tags, web servers can also communicate noindexing directives via response headers, like:
X-Robots-Tag: noindexWhen a crawler requests a page, the server can include X-Robots response headers detailing indexing (as well as following) instructions.
Comparison to Other Meta Tags
The noindex robots meta tag is often confused with similar directives like nofollow, noarchive, and nosnippet. Here is a comparison:
- nofollow – This value makes search engines stop following any links found on a particular page. Link equity will not pass.
- noarchive – Prevents caching of web pages by sites like Google and the Internet Archive.
- nosnippet – Blocks displaying descriptive snippets for the page in SERP listings.
So, noindex blocks the indexing of page content and serves it in the results. The other tags serve different purposes related to links, caching, and snippet creation.
How to Implement NoIndex Tags
Noindexing a page seems simple in theory – just stick a meta tag in the header or a response header from the server!
But in practice, you need to apply noindex instructions in specific ways to avoid issues.
HTML Implementation
First, the standard noindex HTML meta tag structure:
<meta name="robots" content="noindex">Key rules for implementation:
- Place the HTML noindex tag in the head section within the first 1024 bytes of the page code;
- Use the “robots” meta name – not all crawlers recognize other names;
- Watch character case – “noindex” not “Noindex”;
- Use the content attribute, not the http-equiv attribute.
So, for example:
<!DOCTYPE html>
<html>
<head>
<title>Page Title</title>
<meta name="robots" content="noindex">
</head>
<body>
(page content)
</body>
</html>Following those best practices ensures you the broadest crawler compatibility.
As we noted previously, you can chain directives like:
<meta name="robots" content="noindex,follow">Which noindexes but allows following page links.
HTTP Response Header Option
For server-side solutions, configure noindexing via HTTP response headers.
Here are examples for major server platforms:
Apache
Within a .htaccess file or Apache config:
Header set X-Robots-Tag "noindex"Nginx
In the site or page config file:
add_header X-Robots-Tag "noindex";IIS (Internet Information Services)
In web.config:
<system.webServer>
<httpProtocol>
<customHeaders>
<add name="X-Robots-Tag" value="noindex" />
</customHeaders>
</httpProtocol>
</system.webServer>Express.js (Node.js framework)
res.set('X-Robots-Tag', 'noindex');Those cover the major web server platforms. Defining an X-Robots header works across all search engines.
Note some recommend removing them after launch for performance. But for a permanent noindex approach, response headers are more foolproof than meta tags alone.
Noindex Best Practices
Adding noindex tags seems simple, but you need to incorporate them judiciously following SEO best practices.
SEO Implications
Noindexing pages significantly alters their treatment by search engines with downstream impacts:
- Pages will not appear in SERPs or drive any traffic;
- Internal rankings may shift as relative page strength changes;
- Indexing useless pages wastes crawler resources later deployed to useful pages;
- Avoid losing reputation from indexing problematic pages.
So, noindexing improves the efficiency of crawling resources. But it also modifies the search footprint of a website.
You should monitor index coverage reports in search consoles to get visibility into changes after noindexing content.
When to Employ NoIndex Tags
Don’t blindly noindex pages without reason! Here are smart use cases for applying noindex directives:
Duplicate Content Pages
Most commonly, noindex gets used to remove duplicate content from SERPs. For example:
- Printer-friendly versions of articles;
- Different language editions with the same content;
- Domains serving identical content.
Choose a canonical version and noindex the other duplicative pages.
Thin-Content Pages
Maybe you have location pages with just basic business info and no unique content. Or sparse category pages stretched out by extra navigation.
Consider noindexing:
- Pages with less than ~250 words of substantive, unique text
- Category archives with automatically generated text like “Category: Shoes” repetitively.
Temporary Development Pages
Launching a site redesign or beta section? During early construction, adding a temporary noindex avoids search engines crawling missing pages and dead links. Remove once development completes.
Irrelevant Pages
Every site contains pages with no value to search visitors. Common examples:
- User login and account management pages
- Shopping cart and checkout process pages
- Auto-generated calendars or schedules
These solely serve site users but not prospective customers from SEO traffic. So, noindexing prevents waste crawl prioritization.
Problematic Pages
Does a page display improperly? Does the content seem spammy or violate guidelines? Rather than editing or removing, implement a noindex.
That preserves access for existing site users without exposing the page to search spiders.
Get in the habit of routinely evaluating low-value pages that offer no unique informational or commercial value to be considered for noindexing.
Common NoIndex Mistakes
Sometimes someone gets overzealous noindexing everything in sight without realizing they nuked a chunk of key pages sending traffic into freefall. As Google warns here, you have to be thoughtful about what and how much you noindex.
Accidental Indexing Removal
Review access logs after applying noindex tags at scale to confirm public traffic levels remain consistent with session and visitor counts as expected.
If unexplained drops occur, you may have unintentionally noindexed high-traffic pages. Double-check for noindex and remove it for key content.
Forgetting Site Structure Changes
Did you recently reorganize your category structure or move popular content to new URLs?
It’s easy to overlook outdated noindex directives lingering on redirected pages. Audit previously indexed sites after migrations to prevent the removal of currently relevant pages.
Improperly Implemented Tags
We covered the specifics earlier about careful meta tag structure and placement – small syntax issues prevent proper parsing.
Always test sampled pages after adding noindex tags to confirm they register as intended when crawled. Fix any implementation bugs revealed.
Advanced NoIndex Techniques
So far, this is typical noindex usage. But how about a few more complex tactics for specific deindexing challenges?
Mixing NoIndex and Disallow
The robots.txt Disallow directive prevents the crawling of pages but does not provide a noindex instruction.
Some cases call for layering both for optimal protection, like:
Robots.txt
User-Agents: *
Disallow: /private-page/
Disallow: /temporary-content/Robots Meta Tag
<meta name= "robots" content= "noindex"> (on all /private-page/ and /temporary-content/ URLs)So, robots.txt blocks search spiders from crawling those sections, while noindex provides additional assurance they will not get indexed.
Strategic De-Indexing with Robots.txt
You can leverage Disallow to deindex parts of sites strategically. For example:
Disallow: /content/articles/Disallows crawling of all article content while permitting crawling of other site sections.
So, search engines still discover pages, links, and site features used for ranking without diluting those signals across low-value article pages.
It presents almost “phantom” content to bots, concentrating signals on key pages for more focused crawling and indexing.
Similar selective deindexing works well for blogs, news sites, merchandising catalogs, and other content-heavy sites.
Wrap-up
While noindexing is not a cure-all, it can profoundly shape a site’s search visibility. Rather than just blocking entire sections, you can perform search engine appearance surgery with precision now – highlighting your best assets while downplaying less relevant pages.
Related Posts
Written by
