Scheduling an R scraper to Google Cloud: a walkthrough

R lang + GCP = . In this article we will see how to use… | by Viacheslav  Kovalevskyi | Deep Learning as I See It

1. Preparing the R script

library(googledrive)# download service account token
temp<-tempfile(fileext = '.json')
download.file('<DROPBOX LINK TO SERVICE ACCOUNT TOKEN>',temp)
drive_auth(path=temp)
# download CSV files from drive and read into R
d<-list()
n<-drive_ls()$id
for (i in n){
temp <- tempfile(fileext = '.csv')
d[which(n==i)]<-list(read.csv(drive_download(
as_id(i), path = temp, type='csv',overwrite = TRUE)$local_path))
}

2. Containerising the scraper

3. Deploying the image to GCP

Workflow courtesy of Google.

4. Checking that your scraper works

library(pushoverr)pushover(message='drive updated', 
user="<ADD USER KEY HERE>",
app="<ADD APP KEY HERE>")

Concluding thoughts

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store