Prevent Search Engines from Caching Content

GeekThis remains completely ad-free with zero trackers for your convenience and privacy. If you would like to support the site, please consider giving a small contribution at Buy Me a Coffee.

At one point or another you will need to force a website or a webpage to not be in a search engines cache or want to make sure it doesn’t get cached. I’m going to show you how to prevent Google and a few other search engines and caching services from caching your website. Please note that having a cached version of your site is often a good thing since if your website is under heavy load or no longer loads, users can still see your content.

The first thing we have to do is modify the header of our website. I always have a separate file for the header contents so I only need to change it once. If you have the <head> tag in multiple locations you will need to modify them all or only for the pages you wish not to be cached by Google.

    <!-- All other head tag contents -->

This will prevent most robots from archiving your webpage. Of course crawlers and spiders don’t have to follow this setting, but most search engines will. To my knowledge, Google, MSN, Bing, Yahoo all abide by the NOARCHIVE tag.

Internet Archive -

To prevent the Internet Archive from having a history of your website available, you will need to modify the “robots.txt” file for your website. Since the Internet Archive isn’t a search engine, prevent them from crawling all pages is perfectly fine and won’t affect your search rankings. But if you want to, you can make it so some pages are available for them to archive.

The below code will block the Internet Archive on all pages of your website.

User-agent: ia_archiver
Disallow: /

To only block certain pages for the Internet Archive you will use something similar to the following code. The below robots file will allow Internet Archive to cache all pages except “page1”, “contact” and all the files inside of “folder”.

User-agent: ia_archiver
Disallow: /page1
Disallow: /contact
Disallow: /folder/

This will hopefully prevent some sites from now archiving your content. I personally have most of my sites opted out of being archived by the Internet Archive but I keep search engines on the list of allowed.

Related Posts

Scraping Title and Meta Data with Scrapebox

Scrape a list of URLs to get site title, description, and keywords to research competition or find missing meta tags using Scrapebox.