System Design

Audit Log

Design an audit log service for a multi-tenant SaaS product like Salesforce or GitHub.

Functional requirements

• Any internal service can record an event describing who did what to which resource and when (e.g., user 42 changed user 77's role to admin in org acme at 14:03 UTC), plus a small JSON details payload.
• Tenant admins can query their own organization's events filtered by actor, action type, target resource, and time range, paged newest-first.
• Events are append-only: no one, including the tenant, can edit or delete them, and an admin can verify that a range of events has not been altered since it was written.
• Events stay in a hot tier for 90 days and an archive tier for 7 years; older events remain retrievable with slower response times.
• Tenant admins can request an export of a filtered time range as a file (CSV or JSON lines) and download it when it is ready.

Non-functional requirements

• Recording an event is the critical path for callers: it must add under 5ms p99 to the calling service, and once acknowledged an event must never be lost.
• Hot-tier queries return a page of 50 filtered events with p99 under 500ms; archive-tier queries may take up to a minute.
• Availability over freshness for reads: an event may take up to 30 seconds to become queryable, but a returned event is always exactly what was written, never partial or altered.
• Durability is regulatory: events are retained for 7 years and survive the loss of any single zone; losing a tenant's audit history is a compliance failure.
• Strict tenant isolation: a query can never return another tenant's events, and tampering must be detectable even by a tenant who does not trust the operator's database administrators.

Scaling & constraints

• 50,000 tenant organizations; 10,000 events per second on average across all services, with peaks of 50,000/sec during US business hours.
• Read:write ratio is roughly 1:1,000; a typical query pages through a few hundred events from the last 7 days, and an export can span 90 days (~10M events for a large tenant).
• An average event is ~1 KB (actor, action, target, timestamp, IP, plus a JSON details blob that can reach 8 KB).
• The hot tier holds 90 days and the archive holds 7 years; event volume grows ~30% per year.
• Tenant skew: the largest 1% of tenants produce ~60% of all events, and a single tenant can burst to 20,000 events/sec during bulk imports or automation runs.
• Events come from ~300 internal services whose clocks can be skewed by up to 5 seconds; the delivery path is at-least-once, so ~0.1% of events arrive twice.

Out of scope

• User authentication and deciding which product actions are auditable.
• Real-time alerting and anomaly detection on audit events.
• SIEM integrations and streaming delivery to third-party log systems.
• Billing, tenant provisioning, and the admin user interface.