This will come handy when you are trying to recover an accidentally deleted website or you need to retrieve a web page that no longer exists at the original location.

You opened a web page on the Internet but the server hosting the site returns a 404 error meaning that either the web page has been removed or moved to a different location.

To recover the lost page, the best option is that you search the page across all three major search engines (Google, Yahoo, Windows Live Search) and hope that a copy of the web page exists in the cache somewhere.

All major search engines store cached copies of web pages

If the original page is not available in any of the search engine’s cache, you can repeat the search process at Internet Archive’s Wayback Machine - it is the largest web repository holding a snapshot (or backup) of over 10 billion web pages.

The Internet Archive doesn’t store web pages created or modified in the past 6-12 months while search engines may have the most recent version of the web pages in their cache.

Recover Deleted Websites Automatically

While it is often possible to recover lost websites using a combination of search engine caches and web archives, the process can be very time-consuming especially if you are trying to recover a large site that had more than a few dozen web pages.

To ease the site recovery process, Frank McCown at Harding University created a tool called Warrick that lets you reconstruct any lost website (or single web page) automatically. Simply type the URL of the web site and Warrick will let you know via email once the recover process is over.

The tool is essentially a web crawler that scans and collects missing web pages from all the four web repositories - Internet Archive, Google, Live Search, and Yahoo. If a web page is found in more than one web repository, Warrick saves the page with the most recent date.

The recovery process may take some time for large websites. For instance, I tried Warrick for reconstructing Digital Inspiration and it took about a week to complete the job. The recovered web pages were provided as a zipped archive (~50 MB).

Warrick is available both as an online service or you can download the Perl source files and run them locally on your own computer.

If you have accidentally deleted or overwritten your web pages, make sure your run Warrick before Google and other search bots attempt to re-crawl the site and replace their cached copies with something else.