Cloudflare R2 Data Catalog Skill Reference

Expert guidance for Cloudflare R2 Data Catalog - Apache Iceberg catalog built into R2 buckets.

Reading Order

New to R2 Data Catalog? Start here:

1. Read "What is R2 Data Catalog?" and "When to Use" below
2. configuration.md - Enable catalog, create tokens
3. patterns.md - PyIceberg setup and common patterns
4. api.md - REST API reference as needed
5. gotchas.md - Troubleshooting when issues arise

Quick reference? Jump to:

• Enable catalog on bucket
• PyIceberg connection pattern
• Permission errors

What is R2 Data Catalog?

R2 Data Catalog is a managed Apache Iceberg REST catalog built directly into R2 buckets. It provides:

• Apache Iceberg tables - ACID transactions, schema evolution, time-travel queries
• Zero-egress costs - Query from any cloud/region without data transfer fees
• Standard REST API - Works with Spark, PyIceberg, Snowflake, Trino, DuckDB
• No infrastructure - Fully managed, no catalog servers to run
• Public beta - Available to all R2 subscribers, no extra cost beyond R2 storage

What is Apache Iceberg?

Open table format for analytics datasets in object storage. Features:

• ACID transactions - Safe concurrent reads/writes
• Metadata optimization - Fast queries without full scans
• Schema evolution - Add/rename/delete columns without rewrites
• Time-travel - Query historical snapshots
• Partitioning - Organize data for efficient queries

When to Use

Use R2 Data Catalog for:

• Log analytics - Store and query application/system logs
• Data lakes/warehouses - Analytical datasets queried by multiple engines
• BI pipelines - Aggregate data for dashboards and reports
• Multi-cloud analytics - Share data across clouds without egress fees
• Time-series data - Event streams, metrics, sensor data

Don't use for:

• Transactional workloads - Use D1 or external database instead
• Sub-second latency - Iceberg optimized for batch/analytical queries
• Small datasets (<1GB) - Setup overhead not worth it
• Unstructured data - Store files directly in R2, not as Iceberg tables

Architecture

┌─────────────────────────────────────────────────┐
│  Query Engines                                  │
│  (PyIceberg, Spark, Trino, Snowflake, DuckDB)  │
└────────────────┬────────────────────────────────┘
                 │
                 │ REST API (OAuth2 token)
                 ▼
┌─────────────────────────────────────────────────┐
│  R2 Data Catalog (Managed Iceberg REST Catalog)│
│  • Namespace/table metadata                     │
│  • Transaction coordination                     │
│  • Snapshot management                          │
└────────────────┬────────────────────────────────┘
                 │
                 │ Vended credentials
                 ▼
┌─────────────────────────────────────────────────┐
│  R2 Bucket Storage                              │
│  • Parquet data files                           │
│  • Metadata files                               │
│  • Manifest files                               │
└─────────────────────────────────────────────────┘

Key concepts: