Google Search Scraper written in Google Scripts
Google search will block your IP address temporarily if you use a download utility like wget to download search pages for scraping Google. However, if you run the scraper on Google servers itself through Apps Script, the request won’t be blocked as the IP address of the scraper would be one of Google’s own IP address.
The other alternative is Import HTML, a Google Docs function that you can used to scrape Google search results inside a Google Sheet.
function scrapeGoogle() {
var response = UrlFetchApp.fetch('http://www.google.com/search?q=labnol');
var myRegexp = /<h3 class=\"r\">([\s\S]*?)<\/h3>/gi;
var elems = response.getContentText().match(myRegexp);
for (var i in elems) {
var title = elems[i].replace(/(^\s+)|(\s+$)/g, '').replace(/<\/?[^>]+>/gi, '');
Logger.log(title);
}
}
Amit Agarwal
Google Developer Expert, Google Cloud Champion
Amit Agarwal is a Google Developer Expert in Google Workspace and Google Apps Script. He holds an engineering degree in Computer Science (I.I.T.) and is the first professional blogger in India.
Amit has developed several popular Google add-ons including Mail Merge for Gmail and Document Studio. Read more on Lifehacker and YourStory