{
  "schema_version": 2,
  "id": "develop/ai/context-engine/agent-memory/memory-etl",
  "title": "Extract memories from documents with MemoryETL",
  "url": "https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/memory-etl/",
  "summary": "Extract long-term memories from documents, track processing, and inspect source evidence.",
  "tags": [
    "docs",
    "develop",
    "ai"
  ],
  "last_updated": "2026-11-03T10:02:56-08:00",
  "page_type": "content",
  "content_hash": "bf3b4ab2aa225be5317c5a303bc2e0bf42a74c6fd582bb5b77955359bb84d566",
  "sections": [
    {
      "id": "code-examples",
      "title": "Code examples",
      "role": "example",
      "text": "These examples connect to Iris services using SDKs or HTTP requests with curl. Follow one language throughout each walkthrough.\n\n\nMemoryETL uses extract, transform, load (ETL) processing to turn text or Markdown documents into long-term memories. For example, a travel assistant can learn a user's plans from trip notes they upload before the conversation starts.\n\nAn **extractor** tells MemoryETL what to extract: a custom memory type and instructions for the model. Creating an extractor is an administrative task. Applications can then submit documents to it.\n\nUse [session extraction](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/sessions) for conversation events. Use [direct memory creation](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/long-term-memory#create-a-custom-memory-directly) when your application already has the records it wants to save."
    },
    {
      "id": "prepare-the-service",
      "title": "Prepare the service",
      "role": "content",
      "text": "MemoryETL is available in Redis Cloud and enabled in self-managed deployments. For a self-managed service's model, processing limits, and worker settings, see [Configure document ingestion](https://redis.io/docs/latest/operate/iris/agent-memory/self-managed/configuration#enable-document-ingestion).\n\nThese examples continue from the [quickstart](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/quickstart). Load the same `.env` connection values and reuse its user ID. Expand an example to copy a complete program that connects for you.\n\nEach run repeats every write in the program. Session-event writes and memory creates with a new idempotency key add new records, so run cleanup or change the identifiers before rerunning.\n\n\nThe examples save memories for `quickstart-user`. In your application, use the authenticated user ID that owns the document. For a self-managed service with ownership enforcement, use the authenticated client setup in [delegation](https://redis.io/docs/latest/operate/iris/agent-memory/self-managed/authentication#delegate-a-user-with-an-agent-key)."
    },
    {
      "id": "create-an-extractor",
      "title": "Create an extractor",
      "role": "content",
      "text": "Ask your operator to [register a memory type and create a document extractor](https://redis.io/docs/latest/operate/iris/agent-memory/self-managed/api-examples#create-an-extractor). Add the returned extractor ID to `.env` as `EXTRACTOR_ID`. Applications use their Data Plane credential to submit documents.\n\n[code example]"
    },
    {
      "id": "submit-and-track-a-document",
      "title": "Submit and track a document",
      "role": "content",
      "text": "Create `trip-notes.md` containing the following line. The examples create it if it's missing. The extractor publishes only information that its memory type captures, so use a document that matches your extractor.\n\n[code example]\n\nOnly `text/plain` and `text/markdown` are accepted. Encode the UTF-8 bytes as base64 **once** before passing `content` to REST or an SDK. The SDK expects the encoded string, not a file object or raw text.\n\nEach submission requires these fields:\n\n| Field | Purpose | Example |\n|:------|:--------|:--------|\n| `documentId` | Your ID for the logical document. | `example-trip-notes` |\n| `revision` | An immutable, case-sensitive version. Use a new value when the content changes. | `1` |\n| `content` | The file's UTF-8 bytes, base64-encoded once. | The examples encode `trip-notes.md`. |\n| `mediaType` | The document's content type. | `text/plain` or `text/markdown` |\n| `ownerId` | The authenticated user the memories belong to. | `quickstart-user` |\n| `extractorId` | The extractor's returned `id`. | `$EXTRACTOR_ID` |\n| `idempotencyKey` | A key to reuse when retrying the same submission. | `example-trip-notes-revision-1` |\n\nSubmit the file. The server returns a separate ingestion ID to track the run. Save it; you need it even if polling times out.\n\n**Available in:** Python (Iris), cURL (Iris)\n\n##### Python (Iris)\n\n[code example]\n\n##### cURL (Iris)\n\n[code example]\n\n\n\nREST returns `202 Accepted`; this confirms submission, not successful extraction. In Python, the submission content is under `accepted.result`.\n\nPoll the ingestion until it reaches a terminal processing state. The examples check every 10 seconds for up to 15 minutes:\n\n**Available in:** Python (Iris), cURL (Iris)\n\n##### Python (Iris)\n\n[code example]\n\n##### cURL (Iris)\n\n[code example]\n\n\n\nList the IDs of the memories the run published. Pass `nextCursor` as `cursor` to read the next page:\n\n**Available in:** Python (Iris), cURL (Iris)\n\n##### Python (Iris)\n\n[code example]\n\n##### cURL (Iris)\n\n[code example]"
    },
    {
      "id": "interpret-results-and-recover",
      "title": "Interpret results and recover",
      "role": "content",
      "text": "MemoryETL processes a document in two stages. First, each chunk is extracted into candidate memories. These candidates are staged, not published. Then a final reconciliation step merges duplicates and continued observations across chunks and publishes the result. No memories are searchable until reconciliation finishes. Publication is incremental, so a run that stops during publication can leave some memories.\n\n| Processing state | Meaning and action |\n|:-----------------|:-------------------|\n| `QUEUED` or `RUNNING` | Keep polling the saved ingestion ID. Do not create another submission to check progress. `completedChunks` can equal `totalChunks` while reconciliation is still running. |\n| `SUCCEEDED` | Processing finished. Inspect `createdCount` and `filteredCount`; success can produce no memories. |\n| `PARTIAL` | Processing stopped after some memories were published. Inspect `error`, counts, and available output IDs. Published memories remain. |\n| `FAILED` | Inspect `error` and available outputs. Do not assume that every prior write was rolled back. |\n\n`completedChunks`, `totalChunks`, `extractorId`, and `extractorRevision` help identify what ran. The chunk counts cover source chunks only; reconciliation isn't counted. `cleanup` is independent: `PENDING`, `DELETED`, or `FAILED` describes temporary source cleanup. A successful extraction with failed cleanup needs operator attention; it is not a reason to submit the document again. Extracted memories and provenance can remain after source bytes are deleted.\n\nReuse the same idempotency key and original request after a timeout or lost submission response. Changing the input under the same key conflicts. Reusing an immutable document revision with different content also conflicts. Use a new revision for changed content; do not change the bytes merely to force a retry of an existing run.\n\nThe service bounds decoded bytes, estimated input tokens, chunks, output records, queued bytes, retries, execution time, and source/checkpoint retention. The reference self-managed configuration allows 8,000 input tokens per extraction request; reconciliation has a separate budget that defaults to 32,000 tokens per request. Inspect the deployment settings and `retentionDeadline`. Idempotency and status retention are bounded; retain your logical document IDs, revisions, ingestion IDs, and recovery decisions in your application. A new revision does not promise automatic removal of every memory from an older revision.\n\nExtracted records return `documentManaged: true`. They appear in ordinary long-term-memory searches unless you set `excludeDocumentManaged: true`. Their text and attributes are managed by document ingestion: generic overwrite and PATCH requests cannot replace them. The exception is a PATCH that changes only the namespace: it moves the memory, keeps its content, and keeps its current expiration. Document chunk spans use UTF-8 byte offsets; do not apply those offsets to a character-indexed string.\n\nWith [ingestion webhooks](https://redis.io/docs/latest/develop/ai/context-engine/agent-memory/ingestion-webhooks), the submission and the reconciled candidates can also be reviewed before they change data. Check hook decisions separately from ingestion processing. Receiver failure and explicit filtering have different outcomes."
    },
    {
      "id": "trace-memories-to-their-sources",
      "title": "Trace memories to their sources",
      "role": "content",
      "text": "Provenance connects a memory to document or session inputs used during extraction. A `sourceId` identifies one revisioned origin in the store; an `ingestionId` identifies a particular document run. Session evidence lists input event IDs. Document evidence identifies ingestions, chunks, and optional source spans. These describe the input supplied to extraction, not a guarantee that each sentence in the resulting memory is a quoted fact.\n\n| Request | Python method | REST endpoint |\n|:--------|:--------------|:--------------|\n| Memory → sources | `list_memory_source_links` | `GET .../memories/{memoryId}/sources` |\n| Memory/source → more evidence | `list_memory_source_evidence` | `GET .../memories/{memoryId}/sources/{sourceId}/evidence` |\n| Source → memories for an owner | `list_source_memories` | `GET .../sources/{sourceId}/memories` |\n\nThe examples take the first output memory ID. They follow its source pages and each source's independent evidence pages, then list memories that the same document source produced for `quickstart-user`:\n\n**Available in:** Python (Iris), cURL (Iris)\n\n##### Python (Iris)\n\n[code example]\n\n##### cURL (Iris)\n\n[code example]\n\n\n\nThe evidence response wraps `source`; it does not return a top-level `items` list. The source object contains `evidence` for documents or `sessionEvidence` for sessions. Newer evidence can also include `evidenceId`, `schemaVersion`, `capturedAt`, and `producer`. Handle their absence on legacy data. `capturedAt` is when extraction input was prepared, not when the original event occurred. A retried delivery retains its evidence identity.\n\nFor reverse lookup, use `GET /iris/data/v1/agent-memory/stores/{storeId}/sources/{sourceId}/memories` with `ownerId`, or `list_source_memories`. Reverse lookup covers document sources; a session source returns `404`. Follow its `nextCursor` independently. Keep every token with the exact query and endpoint that issued it. Tokens expire; restart a query when one is invalid, and URL-encode tokens in REST calls.\n\nA source link does not grant access to the original document, session, or another owner's memories. The service checks source and record access separately; sharing an extracted memory does not share the private document it came from."
    }
  ],
  "examples": [
    {
      "id": "create-an-extractor-ex0",
      "language": "dotenv",
      "code": "EXTRACTOR_ID='<extractor-id>'",
      "section_id": "create-an-extractor"
    },
    {
      "id": "submit-and-track-a-document-ex0",
      "language": "text",
      "code": "Trip to Kyoto, April 3 to April 10. I am vegetarian and want to try local tofu dishes.",
      "section_id": "submit-and-track-a-document"
    },
    {
      "id": "submit-and-track-a-document-ex1",
      "language": "python",
      "code": "accepted = memory.client.create_document_ingestion(\n    document_id=\"example-trip-notes\",\n    revision=\"1\",\n    content=base64.b64encode(Path(\"trip-notes.md\").read_bytes()).decode(\"ascii\"),\n    media_type=\"text/markdown\",\n    owner_id=OWNER_ID,\n    extractor_id=EXTRACTOR_ID,\n    idempotency_key=\"example-trip-notes-revision-1\",\n)\ningestion_id = accepted.result.ingestion_id\nprint(\"Ingestion ID:\", ingestion_id)",
      "section_id": "submit-and-track-a-document"
    },
    {
      "id": "submit-and-track-a-document-ex2",
      "language": "bash",
      "code": "jq -n --rawfile text trip-notes.md --arg owner \"$OWNER_ID\" --arg extractor \"$EXTRACTOR_ID\" '{\n  documentId: \"example-trip-notes\",\n  revision: \"1\",\n  content: ($text | @base64),\n  mediaType: \"text/markdown\",\n  ownerId: $owner,\n  extractorId: $extractor,\n  idempotencyKey: \"example-trip-notes-revision-1\"\n}' > ingestion.json\nACCEPTED=$(curl --fail-with-body --silent --show-error \\\n  -X POST \"$MEMORY_API/document-ingestions\" \\\n  -H \"$AUTH\" -H 'Content-Type: application/json' --data-binary @ingestion.json)\nINGESTION_ID=$(jq -er .ingestionId <<< \"$ACCEPTED\")\necho \"Ingestion ID: $INGESTION_ID\"",
      "section_id": "submit-and-track-a-document"
    },
    {
      "id": "submit-and-track-a-document-ex3",
      "language": "python",
      "code": "for _ in range(90):\n    status = memory.client.get_document_ingestion(ingestion_id=ingestion_id).ingestion.status\n    if status.processing.value in {\"SUCCEEDED\", \"PARTIAL\", \"FAILED\"}:\n        print(status.model_dump_json(by_alias=True, exclude_none=True))\n        break\n    time.sleep(10)\nelse:\n    raise TimeoutError(f\"Continue polling ingestion {ingestion_id}\")",
      "section_id": "submit-and-track-a-document"
    },
    {
      "id": "submit-and-track-a-document-ex4",
      "language": "bash",
      "code": "STATE=''\nfor _ in $(seq 90); do\n  RECORD=$(curl --fail-with-body --silent --show-error \\\n    \"$MEMORY_API/document-ingestions/$INGESTION_ID\" -H \"$AUTH\")\n  STATE=$(jq -r .ingestion.status.processing <<< \"$RECORD\")\n  case \"$STATE\" in\n    SUCCEEDED|PARTIAL|FAILED) jq .ingestion.status <<< \"$RECORD\"; break ;;\n  esac\n  sleep 10\ndone\ncase \"$STATE\" in\n  SUCCEEDED|PARTIAL|FAILED) ;;\n  *) echo \"Continue polling ingestion $INGESTION_ID\" >&2; exit 1 ;;\nesac",
      "section_id": "submit-and-track-a-document"
    },
    {
      "id": "submit-and-track-a-document-ex5",
      "language": "python",
      "code": "output_ids = []\ncursor = None\nwhile True:\n    page = memory.client.list_document_ingestion_outputs(\n        ingestion_id=ingestion_id, limit=100, cursor=cursor\n    )\n    output_ids.extend(page.items)\n    cursor = page.next_cursor\n    if not cursor:\n        break\nprint(\"Output memory IDs:\", output_ids)\nif status.processing.value != \"SUCCEEDED\":\n    raise RuntimeError(f\"Inspect ingestion {ingestion_id}: {status.processing.value}\")\nif not output_ids:\n    raise SystemExit(\"The extractor published no memories from this document.\")",
      "section_id": "submit-and-track-a-document"
    },
    {
      "id": "submit-and-track-a-document-ex6",
      "language": "bash",
      "code": "OUTPUT_IDS=()\nCURSOR=''\nwhile :; do\n  PAGE=$(curl --fail-with-body --silent --show-error --get \\\n    \"$MEMORY_API/document-ingestions/$INGESTION_ID/outputs\" -H \"$AUTH\" \\\n    --data-urlencode 'limit=100' ${CURSOR:+--data-urlencode \"cursor=$CURSOR\"})\n  while IFS= read -r id; do OUTPUT_IDS+=(\"$id\"); done < <(jq -r '.items[]' <<< \"$PAGE\")\n  CURSOR=$(jq -r '.nextCursor // empty' <<< \"$PAGE\")\n  [ -n \"$CURSOR\" ] || break\ndone\necho \"Output memory IDs: ${OUTPUT_IDS[*]:-}\"\n[ \"$STATE\" = SUCCEEDED ] || { echo \"Inspect ingestion $INGESTION_ID: $STATE\" >&2; exit 1; }\n[ ${#OUTPUT_IDS[@]} -gt 0 ] || { echo \"The extractor published no memories from this document.\" >&2; exit 1; }",
      "section_id": "submit-and-track-a-document"
    },
    {
      "id": "trace-memories-to-their-sources-ex0",
      "language": "python",
      "code": "memory_id = output_ids[0]\nsource_cursor = None\nwhile True:\n    links = memory.client.list_memory_source_links(\n        memory_id=memory_id, limit=100, cursor=source_cursor\n    )\n    for source in links.items:\n        print(source.model_dump_json(by_alias=True, exclude_none=True))\n        evidence_cursor = source.evidence_next_cursor\n        while evidence_cursor:\n            more = memory.client.list_memory_source_evidence(\n                memory_id=memory_id, source_id=source.source_id,\n                limit=100, cursor=evidence_cursor,\n            )\n            print(more.source.model_dump_json(by_alias=True, exclude_none=True))\n            evidence_cursor = more.source.evidence_next_cursor\n        if source.kind.value != \"DOCUMENT\":\n            continue  # reverse lookup covers document sources\n        related_cursor = None\n        while True:\n            related = memory.client.list_source_memories(\n                source_id=source.source_id, owner_id=OWNER_ID,\n                limit=100, cursor=related_cursor,\n            )\n            for record in related.items:\n                print(\"From the same source:\", record.memory_id, record.text)\n            related_cursor = related.next_cursor\n            if not related_cursor:\n                break\n    source_cursor = links.next_cursor\n    if not source_cursor:\n        break",
      "section_id": "trace-memories-to-their-sources"
    },
    {
      "id": "trace-memories-to-their-sources-ex1",
      "language": "bash",
      "code": "MEMORY_ID=${OUTPUT_IDS[0]}\nSOURCE_CURSOR=''\nwhile :; do\n  LINKS=$(curl --fail-with-body --silent --show-error --get \\\n    \"$MEMORY_API/memories/$MEMORY_ID/sources\" -H \"$AUTH\" \\\n    --data-urlencode 'limit=100' ${SOURCE_CURSOR:+--data-urlencode \"cursor=$SOURCE_CURSOR\"})\n  jq '.items' <<< \"$LINKS\"\n  while IFS= read -r SOURCE; do\n    SOURCE_ID=$(jq -r .sourceId <<< \"$SOURCE\")\n    EVIDENCE_CURSOR=$(jq -r '.evidenceNextCursor // empty' <<< \"$SOURCE\")\n    while [ -n \"$EVIDENCE_CURSOR\" ]; do\n      MORE=$(curl --fail-with-body --silent --show-error --get \\\n        \"$MEMORY_API/memories/$MEMORY_ID/sources/$SOURCE_ID/evidence\" -H \"$AUTH\" \\\n        --data-urlencode 'limit=100' --data-urlencode \"cursor=$EVIDENCE_CURSOR\")\n      jq .source <<< \"$MORE\"\n      EVIDENCE_CURSOR=$(jq -r '.source.evidenceNextCursor // empty' <<< \"$MORE\")\n    done\n    [ \"$(jq -r .kind <<< \"$SOURCE\")\" = DOCUMENT ] || continue  # reverse lookup covers document sources\n    RELATED_CURSOR=''\n    while :; do\n      RELATED=$(curl --fail-with-body --silent --show-error --get \\\n        \"$MEMORY_API/sources/$SOURCE_ID/memories\" -H \"$AUTH\" \\\n        --data-urlencode \"ownerId=$OWNER_ID\" --data-urlencode 'limit=100' \\\n        ${RELATED_CURSOR:+--data-urlencode \"cursor=$RELATED_CURSOR\"})\n      jq '.items[] | {memoryId, text}' <<< \"$RELATED\"\n      RELATED_CURSOR=$(jq -r '.nextCursor // empty' <<< \"$RELATED\")\n      [ -n \"$RELATED_CURSOR\" ] || break\n    done\n  done < <(jq -c '.items[]' <<< \"$LINKS\")\n  SOURCE_CURSOR=$(jq -r '.nextCursor // empty' <<< \"$LINKS\")\n  [ -n \"$SOURCE_CURSOR\" ] || break\ndone",
      "section_id": "trace-memories-to-their-sources"
    }
  ]
}