HOW-TO get parallel HTTP connections in R

parallel_http.R

library(parallel)
library(httr)
library(RCurl)
detectCores()
f <- function(x) {
  #h <- httr::handle('http://slowwly.robertomurray.co.uk/')
  #httr::GET('http://slowwly.robertomurray.co.uk/delay/2000/url/http://www.google.co.uk', handle = h)
  RCurl::httpGET('http://slowwly.robertomurray.co.uk/delay/2000/url/http://www.google.co.uk')
  #Sys.sleep(2)
  return(T)
}
vf <- Vectorize(f)
x <- 1:6
system.time(pvec(x, vf, mc.cores = 1))
system.time(pvec(x, vf, mc.cores = 2))
system.time(pvec(x, vf, mc.cores = 4))
system.time(pvec(x, vf, mc.cores = 6))

Using httr I couldn't manage to make it launch parallel HTTP connections; mc.cores = 6 execution is almost as slow as mc.cores = 1. With RCurl parallel results are faster.

httr 6 cores

> system.time(pvec(x, vf, mc.cores = 6))
   user  system elapsed 
  0.024   0.080  12.201 

RCurl 6 cores

> system.time(pvec(x, vf, mc.cores = 6))
   user  system elapsed 
  0.048   0.080   2.260 

More info:

• r-lib/httr#99
• r-lib/httr#271