Design a Paste Service

Understanding the Problem

Paste services like Pastebin, GitHub Gist, and Hastebin let users store text snippets and share them via a short URL. Let's design one from scratch.

First, let's clarify the requirements:

Functional Requirements:

• Users can create a paste by submitting text content and receive a unique URL.
• Users can retrieve paste content by visiting the unique URL.
• Pastes can optionally have an expiration time (e.g., 10 minutes, 1 hour, never).
• Optional: syntax highlighting for popular programming languages.
• Optional: custom short URLs (aliases).

Non-Functional Requirements:

• Read-heavy: More reads than writes. Estimated ratio ~5:1.
• Low latency: Paste retrieval should be fast (< 50ms for cached content).
• High availability: The service should be highly available — shared links must always work.
• Durability: Once created, paste content must not be lost until it expires.
• Size limits: Individual pastes capped at ~10 MB to prevent abuse.

Estimation

Let's size this system:

• 5M new pastes per month (writes): ~2 pastes/second
• Read:Write ratio 5:1: ~10 reads/second, peak ~30/s
• Average paste size: ~10 KB (most pastes are small code snippets or config files)
• Storage per month: 5M × 10 KB = 50 GB/month
• Storage for 5 years: 50 GB × 60 months = 3 TB
• Metadata per paste: ~500 bytes (key, title, language, timestamps, user ID)
• Metadata storage (5 years): 300M pastes × 500B = 150 GB
• Cache: Cache the top 20% hot pastes. 20% × 3 TB = ~600 GB (or cache metadata only: ~30 GB)

The paste content is the big storage cost. Storing it in an object store (like S3) is far cheaper than a relational database. Metadata stays in a traditional database.

API Design

Simple REST API:

Create Paste

Get Paste

Content vs Metadata separation: The API returns both metadata and content together, but internally they are stored separately. Metadata (title, language, timestamps) lives in a database for fast queries. The actual paste content lives in an object store (S3) for cost-effective, durable storage.

Storage Design

The key insight: separate metadata from content.

Metadata (SQL or NoSQL database):

• paste_key (VARCHAR 8, PRIMARY KEY) — Unique short key
• title (VARCHAR 255) — Optional title
• language (VARCHAR 32) — Syntax highlighting language
• s3_path (VARCHAR 255) — Path to content in object store
• content_size (INT) — Size in bytes
• user_id (BIGINT) — Creator (nullable for anonymous)
• created_at (TIMESTAMP)
• expires_at (TIMESTAMP, nullable) — Null means never expires
• view_count (BIGINT) — Number of reads

Content (Object Store — S3, GCS, or MinIO):

• Stored as plain text files keyed by paste_key
• Object stores are optimized for large blob storage — much cheaper per GB than databases
• Built-in durability (11 nines for S3), replication, and CDN integration

Why not just store content in the database? At 10 KB average per paste, 300M pastes would mean 3 TB of BLOB data in your database. This bloats the DB, makes backups slow, and is 10-50x more expensive than object storage. The database should only handle the small, structured metadata.

Expiration & Cleanup

Expiration: When a paste is created with expires_in, compute expires_at and store it in metadata. On reads, check expires_at — if past, return 404. This is a lazy deletion approach.

Active cleanup: Run an async cleanup worker on a schedule (e.g., every hour). It queries for expired pastes, deletes the S3 content, and removes the metadata row. The paste key can optionally be returned to the KGS pool for reuse.

Rate Limiting: Prevent abuse with per-IP and per-user rate limits. Use a token bucket or sliding window. Typical limits: 10 pastes/minute for anonymous, 60/minute for authenticated users. Also enforce the 10 MB content size limit at the API gateway level.