Build your own Reddit Scraper with Google Apps Script

Reddit Scraper is a Google Script that pulls all posts from any Reddit (subreddit) and saves the information in a Google sheet. The script extracts the post’s title, description, permalink and the posting date but can be easily extended to including user comments and thumbnail images as well.

The script runs through a background trigger every 5 minutes (configurable) and the trigger is automatically deleted once all the posts have been processed.

/* Reddit Scraper written by Amit Agarwal */
/* January 9, 2013 */

/* Replace LifeProTips with the Subreddit Name */
var REDDIT = 'LifeProTips';

function run() {
  deleteTriggers_();

  /* Fetch Reddit posts every 5 minutes to avoid hitting
     the reddit and Google Script quotas */
  ScriptApp.newTrigger('scrapReddit').timeBased().everyMinutes(5).create();
}

function scrapReddit() {
  // Process 20 Reddit posts in a batch
  var url = 'http://www.reddit.com/r/' + REDDIT + '/new.xml?limit=20' + getLastID_();

  // Reddit API returns the results in XML format
  var response = UrlFetchApp.fetch(url);
  var doc = XmlService.parse(response.getContentText());
  var entries = doc.getRootElement().getChildren('channel')[0].getChildren('item');

  var data = new Array();

  for (var i = 0; i < entries.length; i++) {
    /* Extract post date, title, description and link from Reddit */

    var date = entries[i].getChild('pubDate').getText();
    var title = entries[i].getChild('title').getText();
    var desc = entries[i].getChild('description').getText();
    var link = entries[i].getChild('link').getText();

    data[i] = new Array(date, title, desc, link);
  }

  if (data.length == 0) {
    /* There's no data so stop the background trigger */
    deleteTriggers_();
  } else {
    writeData_(data);
  }
}

/* Write the scrapped data in a batch to the
   Google Spreadsheet since this is more efficient */
function writeData_(data) {
  if (data.length === 0) {
    return;
  }

  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var sheet = ss.getSheets()[0];
  var row = sheet.getLastRow();
  var col = sheet.getLastColumn();

  var range = sheet.getRange(row + 1, 1, data.length, 4);
  try {
    range.setValues(data);
  } catch (e) {
    Logger.log(e.toString());
  }
}

/* Use the ID of the last processed post from Reddit as token */
function getLastID_() {
  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var sheet = ss.getSheets()[0];
  var row = sheet.getLastRow();
  var col = sheet.getLastColumn();

  var url = sheet.getRange(row, col).getValue().toString();
  var pattern = /.*comments\/([^\/]*).*/;
  var id = url.match(pattern);

  return id ? '&after=t3_' + id[1] : '';
}

/* Posts Extracted, Delete the Triggers */
function deleteTriggers_() {
  var triggers = ScriptApp.getProjectTriggers();
  for (var i = 0; i < triggers.length; i++) {
    ScriptApp.deleteTrigger(triggers[i]);
  }
}
Amit Agarwal

Amit Agarwal

Google Developer Expert, Google Cloud Champion

Amit Agarwal is a Google Developer Expert in Google Workspace and Google Apps Script. He holds an engineering degree in Computer Science (I.I.T.) and is the first professional blogger in India.

Amit has developed several popular Google add-ons including Mail Merge for Gmail and Document Studio. Read more on Lifehacker and YourStory

Awards & Titles

Digital Inspiration has won several awards since it's launch in 2004.

Google Developer Expert

Google Developer Expert

Google awarded us the Google Developer Expert award recogizing our work in Google Workspace.

ProductHunt Golden Kitty

ProductHunt Golden Kitty

Our Gmail tool won the Lifehack of the Year award at ProductHunt Golden Kitty Awards in 2017.

Microsoft MVP Alumni

Microsoft MVP Alumni

Microsoft awarded us the Most Valuable Professional (MVP) title for 5 years in a row.

Google Cloud Champion

Google Cloud Champion

Google awarded us the Champion Innovator title recognizing our technical skill and expertise.

Email Newsletter

Sign up for our email newsletter to stay up to date.

We will never send any spam emails. Promise.