Skip to content
ZJ

Case study /10 · In production

AEMS

Automation · Data Engineering · Event Intelligence

A scalable event acquisition and enrichment system for collecting, validating and publishing structured event data.

AEMS (Automated Event Management System) is the data engine behind event discovery: it finds events across many sources, cleans them, resolves duplicates, pins them to real locations and publishes them as structured records.

  • Go
  • Python
  • PostgreSQL
  • Redis
  • Crawlers
  • Scheduling
  • Data validation
  • Geocoding
AEMS end-to-end event pipeline flow: acquire, extract, validate, normalize, deduplicate, enrich and publish, with typed rejections feeding a per-source quality score that adjusts crawl cadence.

Context

Event data powers products like CLZTR, but no single source is complete or reliable. Venues post to their own sites, aggregators overlap, formats drift weekly. The only way to a trustworthy catalog is a pipeline that assumes every input is partially wrong.

Problem

Four failure modes had to be engineered away: missed events (coverage), broken records (validation), the same event appearing five times (deduplication), and events floating without usable coordinates (enrichment).

Constraints

  • Sources change markup and structure without notice; the pipeline had to degrade per-source, not globally.
  • Politeness and correctness toward sources: scheduling, rate limits and caching were mandatory.
  • Deduplication had to work across sources that disagree on titles, times and venue naming.
  • Downstream consumers needed a stable schema regardless of upstream chaos.

Responsibilities

  • Crawler framework in Go with per-source adapters; Python extractors for irregular sources.
  • Scheduling system controlling crawl cadence per source reliability and change rate.
  • Validation and normalization stages enforcing the canonical event schema.
  • Deduplication engine matching events across sources.
  • Geographic enrichment resolving venues and addresses to coordinates.
  • Publishing layer emitting structured events to PostgreSQL and Redis consumers.

Architecture

  • A staged pipeline — acquire → extract → validate → normalize → deduplicate → enrich → publish — with every stage independently observable and retryable.
  • Go handles concurrent crawling and orchestration; Python handles messy extraction where its parsing ecosystem wins.
  • PostgreSQL stores canonical events and full stage lineage; Redis carries work queues and hot lookups.
  • Failed records park in inspectable dead-letter queues per stage instead of vanishing.

Key decisions

  • Schema-first: a strict canonical event model with explicit optionality, so downstream code never guesses.
  • Deduplication by composite similarity — time proximity, venue resolution, fuzzy title matching — tuned to prefer false splits over false merges.
  • Store source lineage on every record: when an event is wrong, you can see exactly which source and stage produced it.
  • Per-source circuit breakers so one broken adapter can't poison a crawl cycle.

Implementation

  • Adapters declare their capabilities and cadence; the scheduler composes crawl plans from those declarations.
  • Validation produces typed rejection reasons, which feed a per-source quality score that in turn adjusts scheduling.
  • Geographic enrichment resolves venue strings through a cached geocoding layer with manual-override support.
  • Publishing is idempotent — replaying any stage is always safe.

Performance

  • Concurrent crawling is bounded per source and globally; backpressure propagates through the queue layer.
  • Deduplication candidates are pre-filtered by time and geographic buckets, keeping comparisons near-linear.
  • Redis-backed caches keep repeat geocoding and repeat-fetch costs near zero.

Outcomes

  • A continuously updated catalog of structured, deduplicated, geolocated events in production.
  • Source breakage became a routine, per-adapter maintenance task instead of a system outage.
  • The same pipeline architecture absorbed new sources without structural change.
  • [PLACEHOLDER — add source counts and event volumes here when publishable.]

Lessons

  • Pipelines are products: their operators are users, and observability is their interface.
  • Designing for partial failure per source is what makes 'scalable' actually true.
  • Data quality is an accumulation of small, boring, testable rules — not one clever algorithm.

Software Architecture

Seven-stage pipeline from the messy public web to clean, deduplicated, geolocated events. 1. Acquire: scheduled, polite crawling through a Go crawler framework, with cadence per source plus rate limits and caching. 2. Extract: pull structure out of inconsistent markup using Python extractors and per-source adapters, with circuit breakers when a source changes. 3. Validate: a record either matches the canonical schema or produces a typed rejection, parked in an inspectable dead-letter queue per stage. 4. Normalize: one shape for titles, times and venues, with explicit optionality, timezone and unit rules, and source lineage attached. 5. Deduplicate: five listings become one event using time and geographic pre-buckets and fuzzy title matching, tuned to prefer false splits over false merges. 6. Enrich: every event gets real coordinates through venue resolution and cached geocoding, with manual override support. 7. Publish: a stable schema downstream, idempotently, into the PostgreSQL canonical store and Redis consumers, so replaying any stage is safe. A per-source quality loop turns typed rejections into a quality score that adjusts crawl cadence, and an observability band covers stage-level metrics, that quality score and inspectable dead letters.
Architecture diagram in four columns. Sources: venue sites each with their own markup, overlapping aggregators with partial coverage, feeds and APIs in irregular formats, and change drift that breaks structure weekly. Crawl layer: a Go crawler framework with bounded concurrency, a scheduler setting cadence by source reliability, source adapters that declare their capabilities, and circuit breakers so one broken adapter cannot poison a crawl cycle. Processing: Python extractors for cases where messy parsing wins, a validation stage enforcing the canonical event schema, a dedup engine matching on composite similarity, and geographic enrichment through cached venue resolution. Storage and publish: PostgreSQL holding canonical events plus full stage lineage, Redis queues for work and hot lookups, per-stage dead letters that stay inspectable, and an idempotent publisher emitting downstream. A closing band describes the quality loop: typed rejections produce a per-source quality score, which adjusts crawl cadence, which improves coverage unattended.