Hello Arrow

Poll: Arrow


Have you used or experimented with Arrow before today?

Vote using emojis on the #workshop-arrow discord channel!

1️⃣ Not yet

2️⃣ Not yet, but I have read about it!

3️⃣ A little

4️⃣ A lot

Hello Arrow
Demo


Some “Big” Data

https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page

NYC Taxi Data

  • big NYC Taxi data set (~40GBs on disk)
open_dataset("s3://voltrondata-labs-datasets/nyc-taxi") |>
  filter(year %in% 2012:2021) |>
  write_dataset("data/nyc-taxi", partitioning = c("year", "month"))
  • tiny NYC Taxi data set (<1GB on disk)
download.file(url = "https://github.com/posit-conf-2023/arrow/releases/download/v0.1.0/nyc-taxi-tiny.zip",
              destfile = "data/nyc-taxi-tiny.zip")

unzip(
  zipfile = "data/nyc-taxi-tiny.zip",
  exdir = "data/"
)

Larger-Than-Memory Data


arrow::open_dataset()


Arrow Datasets allow you to query against data that has been split across multiple files. This division of data into multiple files may indicate partitioning, which can accelerate queries that only touch some partitions (files). Call open_dataset() to point to a directory of data files and return a Dataset, then use dplyr methods to query it.

NYC Taxi Dataset

library(arrow)

nyc_taxi <- open_dataset("data/nyc-taxi")

NYC Taxi Dataset

nyc_taxi |> 
  nrow()
[1] 1150352666


1.15 billion rows 🤯

NYC Taxi Dataset: A question


What percentage of taxi rides each year had more than 1 passenger?

NYC Taxi Dataset: A dplyr pipeline

library(dplyr)

nyc_taxi |>
  group_by(year) |>
  summarise(
    all_trips = n(),
    shared_trips = sum(passenger_count > 1, na.rm = TRUE)
  ) |>
  mutate(pct_shared = shared_trips / all_trips * 100) |>
  collect()
# A tibble: 10 × 4
    year all_trips shared_trips pct_shared
   <int>     <int>        <int>      <dbl>
 1  2012 178544324     53313752       29.9
 2  2013 173179759     51215013       29.6
 3  2014 165114361     48816505       29.6
 4  2015 146112989     43081091       29.5
 5  2016 131165043     38163870       29.1
 6  2017 113495512     32296166       28.5
 7  2018 102797401     28796633       28.0
 8  2019  84393604     23515989       27.9
 9  2020  24647055      5837960       23.7
10  2021  30902618      7221844       23.4

NYC Taxi Dataset: A dplyr pipeline

library(tictoc)

tic()
nyc_taxi |>
  group_by(year) |>
  summarise(
    all_trips = n(),
    shared_trips = sum(passenger_count > 1, na.rm = TRUE)
  ) |>
  mutate(pct_shared = shared_trips / all_trips * 100) |>
  collect()
toc()

6.077 sec elapsed

Your Turn

  1. Calculate the longest trip distance for every month in 2019

  2. How long did this query take to run?

➡️ Hello Arrow Exercises Page

What is Apache Arrow?

A multi-language toolbox for accelerated data interchange and in-memory processing

Arrow is designed to both improve the performance of analytical algorithms and the efficiency of moving data from one system or programming language to another

https://arrow.apache.org/overview/

Apache Arrow Specification

In-memory columnar format: a standardized, language-agnostic specification for representing structured, table-like data sets in-memory.


A Multi-Language Toolbox

Accelerated Data Interchange

Accelerated In-Memory Processing

Arrow’s Columnar Format is Fast

The contiguous columnar layout enables vectorization using the latest SIMD (Single Instruction, Multiple Data) operations included in modern processors.

arrow 📦


arrow 📦

Today

  • Module 1: Larger-than-memory data manipulation with Arrow—Part I
  • Module 2: Data engineering with Arrow
  • Module 3: In-memory workflows in R with Arrow
  • Module 4: Larger-than-memory data manipulation with Arrow—Part II

🔗 pos.it/arrow-conf24

Hello Arrow

  1. Slides

  2. Tools

  3. Close
  • Hello Arrow
  • Poll: Arrow
  • Hello ArrowDemo
  • Some “Big” Data
  • NYC Taxi Data
  • Larger-Than-Memory Data
  • NYC Taxi Dataset
  • NYC Taxi Dataset
  • NYC Taxi Dataset: A question
  • NYC Taxi Dataset: A dplyr pipeline
  • NYC Taxi Dataset: A dplyr pipeline
  • Your Turn
  • What is Apache Arrow?
  • Apache Arrow Specification
  • A Multi-Language Toolbox
  • Accelerated Data Interchange
  • Accelerated In-Memory Processing
  • arrow 📦
  • arrow 📦
  • Today
  • f Fullscreen
  • s Speaker View
  • o Slide Overview
  • e PDF Export Mode
  • ? Keyboard Help