Observe the Semantic Cache with OpenTelemetry

RedisVL can report what SemanticCache does through OpenTelemetry: a trace for every lookup that separates embedding time from search time, a hit rate, and the distance of the nearest match on every lookup, hits and misses alike. That last signal is what you need to tune distance_threshold.

RedisVL depends only on the OpenTelemetry API. Your application owns the SDK and the exporters, and RedisVL emits nothing until you configure them. With no SDK configured, the instrumentation adds about 11 µs to a lookup that embeds, measured on an Apple M4 Pro.

See Cache LLM Responses for how to use the cache itself.

Install

Install RedisVL with the otel extra, plus the SDK and an exporter of your choice:

pip install "redisvl[otel]" "opentelemetry-sdk>=1.32" opentelemetry-exporter-otlp-proto-http

Use opentelemetry-sdk 1.32.0 or later. RedisVL’s histograms carry bucket advice, and earlier SDK releases drop it on OTLP export (fixed in 1.31.0) or apply only the first histogram’s advice (fixed in 1.32.0).

Configure an SDK

Set the global tracer and meter providers once, at application start-up. RedisVL picks them up automatically, whether you create the cache before or after; there is nothing RedisVL-specific to call.

from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from redisvl.extensions.cache.llm import SemanticCache

tracer_provider = TracerProvider()
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(tracer_provider)

metrics.set_meter_provider(
    MeterProvider(metric_readers=[PeriodicExportingMetricReader(OTLPMetricExporter())])
)

cache = SemanticCache(name="llmcache", redis_url="redis://localhost:6379")
cache.store(prompt="What is the capital of France?", response="Paris")
cache.check(prompt="What's the capital of France?")

The OTLP exporters send to http://localhost:4318 unless you set OTEL_EXPORTER_OTLP_ENDPOINT, so run an OpenTelemetry Collector there, or point them at a backend that accepts OTLP. With nothing listening, the exporters log connection errors and retry, which can delay shutdown, but your application keeps working.

All RedisVL telemetry uses the instrumentation scope redisvl, with RedisVL’s version and the semantic-conventions schema URL it targets.

What RedisVL emits

Spans

Each check, store, update and drop call produces a span tree. A lookup looks like this:

semantic_cache.check     INTERNAL
├── embeddings {model}   CLIENT
└── retrieval {index}    CLIENT

store, update and drop produce semantic_cache.store, semantic_cache.update and semantic_cache.drop, and async methods produce the same trees as their sync forms. There is an embeddings span only when the operation embeds a prompt: a check or store given a precomputed vector has none, and update and drop never embed. The embeddings span comes from the vectorizer, so SemanticRouter, SemanticMessageHistory and direct calls to embed and embed_many produce it too.

Span Attribute Value
semantic_cache.* redisvl.cache.name The cache name
redisvl.cache.result hit or miss, on check only
embeddings {model} gen_ai.operation.name embeddings
gen_ai.provider.name For example openai, azure.ai.openai, cohere, mistral_ai, gcp.vertex_ai, gcp.gemini or aws.bedrock; other vectorizers report their type, such as hf
gen_ai.request.model The vectorizer’s model
gen_ai.embeddings.dimension.count The vectorizer’s dimensions, when known
redisvl.embeddings_cache.result hit, miss, or partial for a batch; only when the vectorizer has an EmbeddingsCache and the call doesn’t pass skip_cache=True
retrieval {index} gen_ai.operation.name retrieval
gen_ai.data_source.id, db.collection.name The index name
gen_ai.retrieval.top_k num_results
db.system.name redis
db.operation.name FT.SEARCH
redisvl.cache.distance_threshold The threshold the lookup used
redisvl.cache.distance The nearest candidate’s distance, when there was one

A failed operation sets an error status and error.type on every RedisVL span it passes through. error.type is the exception’s class, prefixed by its module unless it is a builtin, such as redis.exceptions.ConnectionError.

Metrics

Metric Unit Attributes Recorded on
redisvl.semantic_cache.operation.duration s redisvl.cache.name, redisvl.cache.operation, redisvl.cache.result (on check), error.type (on failure) Every check, store, update and drop, sync or async
redisvl.semantic_cache.distance 1 redisvl.cache.name, redisvl.cache.result Every completed lookup that found at least one candidate
gen_ai.client.operation.duration s gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, error.type (on failure) Every embedding call that reaches the provider, but not an embeddings-cache hit. One embed_many call records one point, covering all its batches

There is no counter: the duration histogram’s count is the number of operations. Cache-operation buckets start at 100 µs, because a lookup against local Redis takes about 250 µs before embedding, while embedding-duration buckets follow the GenAI conventions’ advice, from 10 ms. Distance buckets are 0.01 wide up to 0.1, then coarser up to 2, the maximum cosine distance. To change any of them, use an SDK View.

Names in the gen_ai.* namespace follow the OpenTelemetry GenAI conventions, which are still in Development status and may change in a future release.

Tune the distance threshold

The distance histogram, split by result, shows how close your lookups come to the threshold, which defaults to 0.1:

Pattern What it suggests What to do
Misses bunched just above the threshold Some may be near-duplicates the cache is refusing Consider loosening the threshold to turn the genuine ones into hits
Hits bunched just below the threshold These are the matches most likely to be wrong Consider tightening the threshold, trading hit rate for accuracy
Misses spread far above the threshold Genuinely new prompts Nothing: no threshold would serve them

The distance of misses, as a heatmap:

sum by (le) (rate(redisvl_semantic_cache_distance_bucket{redisvl_cache_result="miss"}[1h]))

The share of misses within 0.05 of a 0.1 threshold:

sum(rate(redisvl_semantic_cache_distance_bucket{redisvl_cache_result="miss", le="0.15"}[1h]))
/
sum(rate(redisvl_semantic_cache_distance_count{redisvl_cache_result="miss"}[1h]))

The share counts only misses that found a candidate: a lookup whose filter excluded every entry, or that ran against an empty cache, records no distance. The le value must be one of the bucket boundaries: a value between them, such as 0.23, returns nothing. The metric does not record which threshold a lookup used, so after you change it, compare windows from before and after rather than mixing them.

A distance cannot tell you whether a match was right. Before you move the threshold, look at the prompts behind those distances with content capture on, as described under Privacy.

Starter queries

These PromQL queries use the names the OpenTelemetry Prometheus exporter produces. Other pipelines can translate names differently, so check your backend’s metric names first.

Hit rate, per cache:

(
  sum by (redisvl_cache_name) (rate(redisvl_semantic_cache_operation_duration_seconds_count{redisvl_cache_operation="check", redisvl_cache_result="hit"}[5m]))
  or
  0 * sum by (redisvl_cache_name) (rate(redisvl_semantic_cache_operation_duration_seconds_count{redisvl_cache_operation="check", redisvl_cache_result="miss"}[5m]))
)
/
sum by (redisvl_cache_name) (rate(redisvl_semantic_cache_operation_duration_seconds_count{redisvl_cache_operation="check", redisvl_cache_result=~"hit|miss"}[5m]))

The or term reports 0 for a cache that has never hit, which would otherwise drop out of the result because it has no hit series. Failed checks carry error.type and no result, so they count towards neither side.

95th-percentile duration, per operation:

histogram_quantile(0.95, sum by (le, redisvl_cache_operation) (rate(redisvl_semantic_cache_operation_duration_seconds_bucket[5m])))

95th-percentile embedding latency, per provider and model:

histogram_quantile(0.95, sum by (le, gen_ai_provider_name, gen_ai_request_model) (rate(gen_ai_client_operation_duration_seconds_bucket{gen_ai_operation_name="embeddings"}[5m])))

Add redis-py’s own metrics

redis-py 7.4 and later ship their own OpenTelemetry metrics for connections and commands. RedisVL never initialises them, so turn them on yourself after setting the meter provider. Command latency is not in the default groups:

from redis.observability import MetricGroup, OTelConfig, get_observability_instance

get_observability_instance().init(
    OTelConfig(
        metric_groups=[
            MetricGroup.CONNECTION_BASIC,
            MetricGroup.RESILIENCY,
            MetricGroup.COMMAND,
        ]
    )
)

redis-py emits metrics only, not spans, which is why RedisVL traces the search itself as the retrieval span.

If you also run a provider’s own GenAI instrumentation, such as one for the OpenAI SDK, both can report the same embedding call: expect a nested client span, and a second set of gen_ai.client.operation.duration points under that library’s instrumentation scope.

Privacy and content capture

By default RedisVL records no prompt, response, metadata or filter values, in any span, event or metric. It never records exception messages on its spans either, because they can contain prompt text: a failure records only the exception’s class. Cache names and model identifiers are exported, so keep tenant or user ids out of cache names.

To see what the cache matched, set:

export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY

With SPAN_ONLY or SPAN_AND_EVENT, the retrieval span also records:

  • gen_ai.retrieval.query.text: the prompt, whenever the lookup was passed one, even alongside a vector.
  • gen_ai.retrieval.documents: a JSON list of the candidates’ entry ids and scores. Each score is the cosine similarity, 1 - distance, so higher is closer, as the GenAI conventions define a relevance score; the nearest raw distance stays on redisvl.cache.distance. Entry ids are unsalted SHA-256 hashes of the stored prompt and filters, so treat them as content too.

RedisVL never records response text, but the variable is process-wide: other GenAI instrumentations that read it may also start recording prompts and responses. Unlike some of them, RedisVL does not treat true as a span mode, and needs no stability opt-in. It reads the variable once, on the first cache lookup.

Prompts can contain personal data. Capture content in place rather than copying production traffic elsewhere, send it only to a trace backend approved to hold prompt data, sample to limit the volume, and turn capture off again when you have what you need, which takes a restart.

Turn it off

To silence RedisVL’s telemetry while keeping the rest of your application’s, set:

export REDISVL_OTEL_DISABLED=true

Only true (case-insensitive) disables it. RedisVL reads the variable once, the first time it records telemetry, so set it before your application starts.

RATE THIS PAGE
Back to top ↑