Packages & Data

Welcome to the Big Data in R with Arrow workshop. On this page you will find information about the software, packages and data we will be using during the 1-day workshop.

We will be using Posit Workbench to learn together. Workbench will be setup with all the below software and data—all participants need to bring is a laptop that can connect to wifi! If you would prefer to run code locally on your own laptop on the day, or if you want to know how to get set-up locally, you can use the below instructions to install the software & packages and download the data.

Software

If you choose to instead work directly on your laptop, you’ll need R and the RStudio Desktop IDE installed and with sufficient disk storage space for the workshop datasets and exercises—we recommend a minimum of about ~80GB to work with the “larger-than-memory” data option or ~2-3GB to work with the smaller example—or “tiny”—data option.

Packages

To install the required core packages for the day, run the following:

install.packages(c(
  "arrow", "dplyr", "duckdb", "dbplyr", "stringr", "lubridate", "tictoc"
))

And to load them:

library(arrow)
library(dplyr)
library(duckdb)
library(dbplyr)
library(stringr)
library(lubridate)
library(tictoc)

Please install the latest CRAN versions of the core packages. Note that some CRAN builds for arrow are suboptimal at the time of writing, e.g. the macOS builds lack Parquet support—it is best to install arrow from R-universe on these platforms.

While the core workshop doesn’t focus on spatial data, the “wrapping up” section near the end of day results in a ggplot map. If you want to run the “wrapping up” workflow code you will need to install the following spatial and plotting packages:

install.packages(c("ggplot2", "ggrepel", "sf", "scales", "janitor"))

And to load them:

library(ggplot2)
library(ggrepel)
library(sf)
library(scales)
library(janitor)

Data

During the 1-day workshop, we will use the following datasets:

NYC Yellow Taxi Trip Record Data: Partitioned parquet files released as open data from the NYC Taxi & Limousine Commission (TLC) with a pre-tidied subset (~40GB) downloaded with either arrow or via https from an AWS S3 bucket
Seattle Public Library Checkouts by Title: A single CSV file (9GB) from the Seattle Open Data portal downloaded via https from an AWS S3 bucket
Taxi Zone Lookup CSV Table & Taxi Zone Shapefile: two NYC Taxi trip ancillary data files from the TLC open platform downloaded via https directly from this repository

Larger-Than-Memory Data Option

1. NYC Taxi Data

This is the main dataset we will use on the day. It’s pretty hefty—40 GB in total—and there are a couple of options for how to acquire it, depending on your internet connection speed.

Option 1—the simplest option—for those with a good internet connection and happy to let things run

If you have a solid internet connection, and especially if you’re in the US/Canada, this option is the simplest. You can use arrow itself to download the data. Note that there are no progress bars displayed during download, and so your session will appear to hang, but you can check progress by inspecting the contents of the download directory. When we tested this with Steph’s laptop and a fast internet connection, it took 67 minutes, though results will likely vary widely.

This method requires the arrow R package to have been built with S3 support enabled. This is on by default for most MacOS and Windows users, but if you’re on Linux, take a look at the instructions here.

After installing arrow, run the following code:

library(arrow)
library(dplyr)

data_path <- "data/nyc-taxi" # Or set your own preferred path

open_dataset("s3://voltrondata-labs-datasets/nyc-taxi") |>
    filter(year %in% 2012:2021) |> 
    write_dataset(data_path, partitioning = c("year", "month"))

Once this has completed, you can check everything has downloaded correctly by calling:

open_dataset(data_path) |>
    nrow()

It might take a moment to run (the data has over a billion rows!), but you should expect to see:

[1] 1150352666

If you get an error message, your download may have been interrupted at some point. The error message will name the file which could not be read. Manually delete this file and run the nrow() code snippet again until you successfully load the remaining data. You can then download any missing files individually using option 2.

Option 2—one file at a time via https

If you have a slower internet connection or are further away from the data S3 bucket location, it’s probably going to be simpler to download the data file-by-file. Or, if you had any interruptions to your download process in the previous step, you can either try instead with this method, or delete the files which weren’t downloaded properly, and use this method to just download the files you need.

We’ve created a script for you which downloads the data one file at a time via https. The script also checks for previously downloaded data, so if you encounter problems downloading any files, just delete the partially downloaded file and run again—the script will only download files which are missing.

download_via_https <- function(data_dir, years = 2012:2021){

    # Set this option as we'll be downloading large files and R has a default
    # timeout of 60 seconds, so we've updated this to 30 mins
    options(timeout = 1800)
    
    # The S3 bucket where the data is stored
    bucket <- "https://voltrondata-labs-datasets.s3.us-east-2.amazonaws.com"
    
    # Collect any errors raised during the download process
    problems <- c()
    
    # Download the data from S3 - loops through the data files, downloading 1 file at a time
    for (year in years) {
      
      # We only have 2 months for 2022 data
      if(year ==2022){
        months = 1:2
      } else {
        months = 1:12
      }
      
      for (month in months) {
        
        # Work out where we're going to be saving the data
        partition_dir <- paste0("year=", year, "/month=", month)
        dest_dir <- file.path(data_dir, partition_dir)
        dest_file_path <- file.path(dest_dir, "part-0.parquet")
        
        # If the file doesn't exist
        if (!file.exists(dest_file_path)) {
          
          # Create the partition to store the data in
          if(!dir.exists(dest_dir)){
            dir.create(dest_dir, recursive = TRUE)
          }
           
          # Work out where we are going to be retrieving the data from
          source_path <- file.path(bucket, "nyc-taxi", partition_dir, "part-0.parquet")
          
          # Download the data - save any error messages that occur
          tryCatch(
            download.file(source_path, dest_file_path, mode = "wb"),
            error = function(e){
              problems <- c(problems, e$message)
            }
          )
        }
      }
    }
    
    print("Downloads complete")
    
    if(length(problems) > 0){
      warning(call. = FALSE, "The following errors occurred during download:\n", paste(problems, collapse =  "\n"))
    }
}


data_path <- "data/nyc-taxi" # Or set your own preferred path

download_via_https(data_path)

Once this has completed, you can check everything has downloaded correctly by calling:

open_dataset(data_path) |>
    nrow()

It might take a moment to run (the data has over a billion rows), but you should expect to see:

[1] 1150352666

If you get an error message, your download may have been interrupted at some point. The error message will name the file which could not be read. Manually delete this file and run the nrow() code snippet again until you successfully load the data. You can then download any missing files by re-running download_via_https(data_path).

2. Seattle Checkouts by Title Data

This is the data we use to explore some data storage and engineering options. It’s a good sized, single CSV file—9GB on-disk in total, which can be downloaded from the an AWS S3 bucket via https:

options(timeout = 1800)
download.file(
  url = "https://r4ds.s3.us-west-2.amazonaws.com/seattle-library-checkouts.csv",
  destfile = "data/seattle-library-checkouts.csv"
)

Tiny Data Option

If you don’t have time or disk space to download the larger-than-memory datasets (and still have disk space do the exercises), you can run the code and exercises in the course with “tiny” versions of these data. Although the focus in this course is working with larger-than-memory data, you can still learn about the concepts and workflows with smaller data—although note you may not see the same performance improvements that you would get when working with larger data.

1. Tiny NYC Taxi Data

We’ve created a “tiny” NYC Taxi dataset which contains only 1 in 1000 rows from the original dataset. So instead of working with 1.15 billion rows of data and about 40GB of files, the tiny taxi dataset is 1.15 million rows and about 50MB of files. You can download the tiny NYC Taxi data directly from this repo via https:

options(timeout = 1800)
download.file(
  url = "https://github.com/posit-conf-2023/arrow/releases/download/v0.1.0/nyc-taxi-tiny.zip",
  destfile = "data/nyc-taxi-tiny.zip"
)

# Extract the partitioned parquet files from the zip folder:
unzip(
  zipfile = "data/nyc-taxi-tiny.zip", 
  exdir = "data/"
)

2. Tiny Seattle Checkouts by Title Data

We’ve created a “tiny” Seattle Checkouts by Title dataset which contains only 1 in 100 rows from the original dataset. So instead of working with ~41 million rows of data in a 9GB file, the tiny Seattle checkouts dataset is ~410 thousand rows and in an 90MB file. You can download the tiny Seattle Checkouts by Title data directly from this repo via https:

options(timeout = 1800)
download.file(
  url = "https://github.com/posit-conf-2023/arrow/releases/download/v0.1.0/seattle-library-checkouts-tiny.csv",
  destfile = "data/seattle-library-checkouts-tiny.csv"
)

Both Data Options / Everyone

3. Taxi Zone Lookup CSV Table & Taxi Zone Shapefile

You can download the two NYC Taxi trip ancillary data files directly from this repo via https:

options(timeout = 1800)
download.file(
  url = "https://github.com/posit-conf-2023/arrow/releases/download/v0.1.0/taxi_zone_lookup.csv",
  destfile = "data/taxi_zone_lookup.csv"
)

download.file(
  url = "https://github.com/posit-conf-2023/arrow/releases/download/v0.1.0/taxi_zones.zip",
  destfile = "data/taxi_zones.zip"
)

# Extract the spatial files from the zip folder:
unzip(
  zipfile = "data/taxi_zones.zip", 
  exdir = "data/taxi_zones"
)