# Fresh context: change data capture, not batch ETL

**Tagline:** News & Media | **Authors:** John Noonan | **Categories:** Tech DE | **Published:** 2026-08-12 | **Updated:** 2026-08-12

In many systems, the reason an agent quotes yesterday's data isn't the model. It's the pipeline behind it: a nightly ETL job that refreshed the agent's context hours ago. Change data capture (CDC) can shrink that staleness window [from hours to seconds](https://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-system).

In February 2024, a British Columbia tribunal ruled against Air Canada. It ordered the airline to pay [812.02 Canadian dollars (CAD)](https://welpartners.com/blog/2024/03/moffatt-v-air-canada-bereavement-fares-do-your-research) after its support chatbot told a grieving customer he could apply for a bereavement fare refund after booking. An agent that quotes yesterday's policy or yesterday's pricing can create liability or other commercial consequences. The airline's actual policy didn't allow retroactive applications. Air Canada argued the chatbot was "a [separate legal entity](https://www.cbc.ca/news/canada/british-columbia/air-canada-chatbot-lawsuit-1.7116416) that is responsible for its own actions." The tribunal held the airline liable for negligent misrepresentation instead. "It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot."

This guide covers what CDC is and how it works, why batch ETL leaves agents acting on stale context, and how event-driven sync keeps agent context current.

## What change data capture is & how it works

Change data capture (CDC) is a pattern for keeping systems in sync as data changes. Instead of reloading everything on a schedule, CDC watches a source database for individual changes — inserts, updates, and deletes — and forwards each one to the systems that need it. The result: downstream systems, like the context store an agent reads from, stay aligned with the source as it changes.

Traditional pipelines ask, "what does this table look like right now?" on a fixed schedule. CDC flips the question to "what just changed?" and answers it continuously. Implementations may [capture change events](https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc) from a database transaction log, use database triggers, or poll tables for new rows. With log-based or correctly configured trigger-based CDC, inserts, updates, and deletes all become events other systems can react to. Polling approaches, on the other hand, may miss deletes entirely.

There are a few ways to detect those changes, and they aren't equivalent:

- **Log-based CDC** reads the database's transaction log directly. A transaction log is the running record a database keeps of every change it commits, so reading it is the closest thing to a live feed of what the database is doing. PostgreSQL streams committed changes in commit order through its [logical decoding feature](https://www.postgresql.org/docs/current/logicaldecoding-explanation.html), and MySQL's row-based binlog records row events with the relevant before or after image for each operation, subject to its row-image configuration. Because the log is parsed asynchronously (separately from the transactions writing to the database), this approach keeps change capture off the write path, captures deletes, and needs no schema changes like adding a "Last Updated" column. In supported Debezium configurations, MySQL and PostgreSQL connectors can propagate changes with [millisecond-range capture delay](https://debezium.io/documentation/reference/stable/features.html), though end-to-end latency depends on the workload and pipeline.

- **Trigger-based CDC** uses database triggers — small pieces of logic that fire automatically on a change — to write each change into a shadow table for downstream consumers to read. Correctly configured triggers can capture inserts, updates, and deletes, but they execute as part of the source database's own write transactions, [adding latency](https://debezium.io/blog/2019/10/01/audit-logs-with-change-data-capture-and-stream-processing) to every write.

- **Query-based (polling) CDC** repeatedly queries the source for rows with new timestamps or version numbers. It's simple and can be fine for hourly propagation, but it typically misses deletes (a deleted row leaves nothing behind to query), and after downtime it captures [only the latest state](https://debezium.io/blog/2018/07/19/advantages-of-log-based-change-data-capture) of a record, losing any intermediate changes.

Of the three, log-based CDC is the most widely supported by production-oriented tools such as Debezium, an open-source CDC platform built on Kafka Connect. Roughly [90% of Debezium users](https://www.infoq.com/presentations/cdc-microservices) deployed it on top of Apache Kafka, with one topic per captured table by default.

<!-- CTA block omitted -->

CDC isn't free, though. Log-based connectors need source-specific setup like replication slots, binlog configuration, permissions, and retention. And Debezium's baseline guarantee is based on [at-least-once delivery semantics](https://debezium.io/blog/2023/06/22/towards-exactly-once-delivery), meaning a change may occasionally be delivered more than once, so downstream consumers have to handle duplicates. These are engineering costs you take on knowingly. They're also a different class of problem than the one batch pipelines have, which is structural.

## Batch ETL is stale by design

Batch pipelines don't just happen to serve old data — the staleness is built into how they're scheduled. In a batch model, jobs run on a clock (say, every hour or once a night) and process whatever accumulated since the last run. Apache Airflow, a common workflow scheduler, only kicks off a run after its [data interval ends](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html): a job covering January 1's data generally doesn't start until January 2. Even an hourly batch adds up to an hour of delay before processing even begins.

The delays compound in real systems. Batch ingestion lag at one production data lake lasted for hours, and sometimes days. Re-platforming ingestion on Flink, a stream-processing engine, cut freshness [from hours to minutes](https://www.uber.com/us/en/blog/from-batch-to-streaming-accelerating-data-freshness-in-ubers-data-lake). One of Uber's batch pipelines took [around 14 hours](https://www.uber.com/us/en/blog/ubers-lakehouse-architecture) to complete, so a single failure pushed downstream freshness back by many more hours.

Netflix went through the same shift, moving away from [24-hour batch collection](https://www.infoq.com/presentations/netflix-event-stream-flink). Its real-time pipeline later reported [sub-minute latency](https://netflixtechblog.com/evolution-of-the-netflix-data-pipeline-da246ca36905). And in one micro-batch case study, [scheduling and orchestration overhead](https://www.infoq.com/articles/micro-batch-streaming-lessons-learned) — the time spent coordinating jobs rather than processing data — were among the main contributors to freshness lag. Faster hardware doesn't help a pipeline that isn't scheduled to run yet.

If you've ever watched a nightly job slip an hour and then spent the morning explaining why a table looks wrong, you know the shape of this. For dashboards and daily reports, it's often fine; a report that's an hour behind is still a useful report. The math changes when the consumer of the data isn't a person reading a chart but an agent taking an action.

## Agents act on what they retrieve

That change in consumer is the whole problem. An agent is typically just a large language model (LLM) that combines [tool use and feedback](https://www.anthropic.com/engineering/building-effective-agents) in a loop, and that loop is what separates it from a question-and-answer (Q&A) system: agents can produce side effects through tool calls. They issue refunds, quote prices, update records, and call downstream APIs. When the context an agent retrieves is stale, the error doesn't stay on the screen — it propagates into actions. And teams may not discover the stale context until a customer reports the result.

The consequences of agents speaking without authoritative validation are already public. In April 2025, Cursor's support bot invented a one-device plan restriction that didn't exist; users canceled their plans, and the company later called it [an incorrect response](https://fortune.com/article/customer-support-ai-cursor-went-rogue) from a front-line AI support bot and offered refunds. In late 2023, a [manipulated dealership chatbot](https://incidentdatabase.ai/cite/622) made headlines. The bot was discussing a 2024 Chevy Tahoe. It agreed to sell it for $1, replying "That's a deal, and that's a legally binding offer – no takesies backsies." The dealership didn't honor the deal and pulled the chatbot after screenshots went viral. Neither incident demonstrates pipeline staleness on its own; together, they show that fresh data alone does not prevent hallucinations or adversarial prompting.

Stale context also degrades output quality in measurable ways. Once false or outdated information makes it into the context window (the text an LLM sees when generating a response), the model may treat it as authoritative in subsequent steps — a failure mode known as [context poisoning](https://www.langchain.com/blog/context-engineering-for-agents). It's one of several failure modes that compound under [context rot](https://redis.io/blog/quality-context-ai-agents), the gradual degradation of an agent's working context over time. Staleness also shows up in evaluations of [retrieval-augmented generation (RAG)](https://redis.io/redis-for-ai/), the common pattern where an app retrieves relevant context from a data store and passes it to the model. One benchmark on version-sensitive questions measured standard RAG at [58% accuracy](https://arxiv.org/html/2604.20598v1). A separate evaluation measured a [version-aware pipeline](https://arxiv.org/html/2506.07270) at 90%. The results aren't directly comparable because they come from different papers, but the version-aware evaluation identified stale or contradictory retrieval context as a leading cause of fluent but false generation.

## How CDC & event-driven sync close the gap

That concern is also reflected in the projected shift toward event-driven infrastructure: adoption of data streaming for agentic AI is projected to grow [beyond 60% by 2028](https://www.gartner.com/en/newsroom/press-releases/2026-06-16-gartner-identifies-the-top-trends-for-data-and-analytics). It was [under 15% in 2025](https://www.gartner.com/en/newsroom/press-releases/2026-06-16-gartner-identifies-the-top-trends-for-data-and-analytics).

The alternative to reloading context on a schedule is reacting to changes as they commit. The core pattern runs: source database → CDC stream → processing or vector embedding step → serving store the agent reads from. Debezium documents this pattern directly: it captures real-time changes from a database and feeds those changes into [vector database systems](https://debezium.io/blog/2025/05/19/debezium-as-part-of-your-ai-solution) to keep AI systems current with fresh, domain-specific data.

This pattern also avoids dual writes, where the app writes to the database and then separately updates the search index, cache, and vector store. With CDC, that follow-on work happens when the data is committed in the original database.

The latency figures come from production systems, not just lab benchmarks. LinkedIn's Databus, an early production CDC system, reported [end-to-end latencies in milliseconds](https://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-system). It handled thousands of change events per second per server. Against the minutes-to-hours floor that batch scheduling imposes, supported pipelines can collapse the staleness window from "since last night's run" to seconds or, in some systems, milliseconds. End-to-end freshness still depends on the connector, backlog, data processing, vector embedding generation, target writes, and serving behavior.

<!-- CTA block omitted -->

## Where Redis fits: CDC into the store your agents read

A CDC stream still needs somewhere fast to land. Redis is a real-time data platform that provides [sub-millisecond latency](https://redis.io/blog/redis-enterprise-extends-linear-scalability-200m-ops-sec/) for many core operations in AI workloads, with [vector search](https://redis.io/redis-for-ai/), data storage for [semantic caching](https://redis.io/langcache/) and [agent memory](https://redis.io/blog/ai-agent-orchestration-platforms/), and separate Redis services and tools supporting those patterns. A Redis benchmark on Amazon Web Services (AWS) used a [20-node AWS cluster](https://redis.io/blog/redis-enterprise-extends-linear-scalability-200m-ops-sec/). It reported more than 100 million operations per second. Command latency remained sub-millisecond in that benchmark. If your app team already runs Redis for caching or session storage, a compatible Redis deployment can hold vectors, agent-memory data, and operational data synchronized by an adjacent Redis Data Integration (RDI) pipeline. Teams including Wix, Swiggy, Comcast, DoorDash, and Uber use Redis as the foundation for an [online feature store](https://redis.io/blog/building-feature-stores-with-redis-introduction-to-feast-with-redis).

[Redis Data Integration (RDI)](https://redis.io/docs/latest/integrate/redis-data-integration) is Redis' CDC system: it tracks changes in a non-Redis source database, such as Oracle, PostgreSQL, MySQL, SQL Server, or MariaDB, and applies the corresponding changes to a Redis target. Under the hood it's a [three-part pipeline](https://redis.io/docs/latest/integrate/redis-data-integration/architecture): a Debezium-based collector captures source changes, [Redis Streams](https://redis.io/resources/architecture-diagrams/redis-streams/) buffer them, and a stream processor applies data mappings defined in YAML Ain't Markup Language (YAML) configuration, no coding required, before writing the results to the target as [JSON documents](https://redis.io/docs/latest/integrate/redis-data-integration/architecture), hashes, sets, or streams. If the target falls behind, backpressure slows pipeline consumption while unprocessed changes remain available in the source log or intermediate stream, subject to retention limits.

An RDI pipeline starts with a full snapshot of the source, then switches to streaming, applying changes to Redis [within seconds](https://redis.io/docs/latest/develop/ai/context-engine/data-integration). Like Debezium itself, RDI is designed for at-least-once delivery of changes in the defined dataset. The payoff for agent builders: RDI can keep configured operational data current within its propagation window. The same Redis target can also support vector search capabilities and agent memory, but those components retain their own update and consistency requirements.

## Fresh context is an architecture decision

Batch ETL is commonly used for reporting workloads that can handle bounded staleness. Agents changed the contract: they quote prices, state policies, and trigger downstream actions, and tribunals and customers hold companies to what their agents say. For supported configurations, log-based CDC replaces the scheduled reload with a continuous feed of change events read straight from the transaction log, turning staleness from a scheduling artifact measured in hours into propagation delay that may be measured in seconds or, in some systems, milliseconds. CDC reduces source-to-context drift, though the rest of the retrieval pipeline still determines overall freshness.

Redis gives that fresh data a fast, in-memory serving layer for real-time retrieval. RDI keeps configured operational data synchronized with the databases you already run, helping your agent's retrieval layer reduce drift from its source of truth. Redis can serve as the vector store and cache in one architecture, with RDI as an adjacent sync pipeline, and your agents are less likely to answer from last night's snapshot. CDC reduces one source of stale context, but it does not prevent model hallucinations or adversarial prompting.

You don't have to rebuild the pipeline to test whether this helps. If you're building agents that act on live business data, [try Redis free](https://redis.io/try-free/) and point RDI at your existing database, or [talk to our team](https://redis.io/meeting/) about designing a real-time context pipeline for your agents.