Operate the Identity Service

Mint, list, rotate, and revoke agent keys, rotate credentials, audit changes, upgrade, back up, and troubleshoot the Identity Service.

This page covers the daily work on an Identity Service release: managing agent keys for every service, rotating credentials, and finding who changed what. It also covers upgrades, backups of the metadata Redis, uninstalling, and troubleshooting.

The examples use release ids and a port-forward to its Service:

kubectl -n <namespace-name> port-forward svc/ids-iris-identity-service 9200:9200

export IRIS_IDENTITY_URL='http://localhost:9200'
export IRIS_IDENTITY_TOKEN='<control-token-or-operator-access-token>'

When operator sign-in is on, use your own access token instead of the control token, so the audit log names you. See Operator sign-in.

Check health

The health routes need no credential:

curl -sS "$IRIS_IDENTITY_URL/health/readiness"

Readiness returns 503 while the Identity Service can't reach the metadata Redis. helm test ids -n <namespace-name> runs the same check in the cluster.

Manage agent keys

The control API manages the keys of every service. The iris command-line interface (CLI) reads IRIS_IDENTITY_URL and IRIS_IDENTITY_TOKEN. To set it up, see Set up on a self-managed deployment.

Mint an agent key

Find the resource ID in the service's control plane first, for example an Agent Memory store ID. Then mint the key:

curl -sS -X POST "$IRIS_IDENTITY_URL/v1/api-keys" \
  -H "Authorization: Bearer $IRIS_IDENTITY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "support-agent",
    "grants": [{
      "tenant": "default",
      "product": "memory",
      "resourceType": "mem-store",
      "resourceId": "<store-id>",
      "actions": ["read", "write"]
    }]
  }'

The response returns keyId and token. The token is the agent key, and it's shown only once: give it to the agent through your secret store, and record only the keyId.

Optional fields:

• expiresAt: when the key stops working, in Unix seconds. Omitted or 0 means never.
• userId: the user that owns the key. It needs apiKeys.userBinding.enabled: true.

For the grant of each service, see Agent keys and grants. With the CLI, each service's own command mints its keys, because it knows the grant the service needs. For example, iris intent-router service key mints a key for a Router.

Before the Identity Service stores the key, it asks the service's control plane to validate the grant. A 400 means the resource doesn't exist in the tenant or an action isn't valid. A 424 means the Identity Service can't reach the control plane or its internal token is wrong.

List and inspect keys

curl -sS -H "Authorization: Bearer $IRIS_IDENTITY_TOKEN" \
  "$IRIS_IDENTITY_URL/v1/api-keys?product=memory&resourceId=<store-id>"
curl -sS -H "Authorization: Bearer $IRIS_IDENTITY_TOKEN" \
  "$IRIS_IDENTITY_URL/v1/api-keys/<key-id>"

With the CLI:

iris identity keys list --product memory --resource-id <store-id>
iris identity keys show <key-id>

iris identity keys list also filters by --resource-type. Neither the API nor the CLI ever returns a key's token after it's minted.

The list is paginated. Over HTTP, pass maxResults and then each response's nextToken. The CLI takes --limit, and --cursor with the nextCursor value it prints; --all reads every page.

Update, rotate, or revoke a key

# Change the grants, the name, or expiresAt. The token stays the same.
curl -sS -X PATCH "$IRIS_IDENTITY_URL/v1/api-keys/<key-id>" \
  -H "Authorization: Bearer $IRIS_IDENTITY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"grants": [{"product": "memory", "resourceType": "mem-store", "resourceId": "<store-id>", "actions": ["read"]}]}'

# Rotate: get a new key; the old one keeps working for graceSeconds.
curl -sS -X POST "$IRIS_IDENTITY_URL/v1/api-keys/<key-id>/rotate?graceSeconds=3600" \
  -H "Authorization: Bearer $IRIS_IDENTITY_TOKEN"

# Revoke: delete the key.
curl -sS -X DELETE "$IRIS_IDENTITY_URL/v1/api-keys/<key-id>" \
  -H "Authorization: Bearer $IRIS_IDENTITY_TOKEN"

With the CLI:

iris identity keys rotate <key-id> --grace 1h --write .env --env <variable-name>
iris identity keys revoke <key-id>

rotate writes the new key to the dotenv file with mode 0600 and never prints it. revoke asks for confirmation unless you pass --yes, and --product <product> refuses keys that grant another service.

Rotate credentials

The Identity Service reads the control token, the runtime credentials, and the internal tokens from their mounted files on each request. A changed Secret takes effect after Kubernetes updates the mounted file, in about a minute, without a restart. While the files update, some calls can fail.

Make every new value unique. The Identity Service doesn't start when two credentials have the same value.

Rotate a runtime credential without failed calls

The Identity Service accepts more than one entry for the same subject, so it can accept the old and the new value for a while. This example rotates the Agent Memory credential:

1. Create a Secret with a new value:

   kubectl -n <namespace-name> create secret generic memory-dp-credential-v2 \
     --from-literal=token="$(openssl rand -hex 32)"
   

2. Add a second entry with the same subject and a new name, and keep the old entry:

   runtime:
     serviceCredentials:
       - name: memory-dp
         subject: memory-dp
         allowedOperations: [api-key-introspect]
         allowedProducts: [memory]
         existingSecret: memory-dp-credential
       - name: memory-dp-v2
         subject: memory-dp
         allowedOperations: [api-key-introspect]
         allowedProducts: [memory]
         existingSecret: memory-dp-credential-v2
   

3. Run helm upgrade for the Identity Service release.

4. Point the data plane at the new Secret. For Agent Memory, set identityService.credential.existingSecret: memory-dp-credential-v2 and run helm upgrade for Agent Memory.

5. Wait until every data-plane pod uses the new Secret, and check that agent calls work.

6. Remove the old entry from values-ids.yaml and run helm upgrade for the Identity Service release.

7. Delete the memory-dp-credential Secret.

Find who did what

The Identity Service writes one log line with the message audit event for each control operation. It has no audit database, so collect the pod logs with your log system. To search the logs Kubernetes still keeps:

kubectl -n <namespace-name> logs -l app.kubernetes.io/instance=ids --tail=-1 \
  | grep '"audit event"' | grep '<key-id>'

The log never contains tokens. A request rejected before it reaches the service, such as one with a malformed body, has no audit line.

Upgrade

1. Read the release notes for the chart version.

2. Upgrade each service's control plane first. A newer Identity Service can send grant fields that an older control plane doesn't know.

3. Upgrade the Identity Service release with the same values file:

   helm upgrade ids redis-ai/iris-identity-service \
     --version <chart-version> \
     --namespace <namespace-name> \
     -f values-ids.yaml
   

4. Run helm test ids -n <namespace-name>, then mint and revoke a test key for each service.

helm rollback restores the workload and its values. It doesn't change generated Secrets or the keys and grants in the metadata Redis.

Back up and restore

The metadata Redis holds every agent key, grant, and subject grant. Back up database 0 of that Redis on a schedule, with persistence turned on. Also export the Secrets you created for the release, such as the control token, the runtime credentials, and the internal tokens, and store the export as you store database passwords.

A restore brings back the keys and grants as they were at backup time:

• Keys minted after the backup don't work. Mint them again and give them to their agents.
• Keys revoked after the backup work again. Revoke them again.

Move the metadata Redis

Stop the Identity Service during the move. If it keeps running after the copy, it changes only the old Redis.

1. Set replicaCount to 0 and run helm upgrade.
2. Copy the data to the new Redis.
3. Put the new host in metadata.urls in the metadata Secret.
4. Set replicaCount to its earlier value and run helm upgrade.

Rotate the metadata Redis credential

The Identity Service reads the metadata Redis connection only at startup.

1. In Redis, add the new user or password, and keep the old one valid.

2. Update the metadata Secret, and every service Secret that uses the same Redis.

3. Restart the Identity Service, and wait until it's ready:

   kubectl -n <namespace-name> rollout restart deployment/ids-iris-identity-service
   kubectl -n <namespace-name> rollout status deployment/ids-iris-identity-service
   

4. Restart the service control planes, then their data planes and workers, if they use the same Redis.

5. Run helm test ids -n <namespace-name>.

6. Remove the old credential from Redis.

Planned outage of the metadata Redis

While the metadata Redis is down, the Identity Service turns unready, and no agent key can be checked, minted, or changed. Kubernetes doesn't restart the pod. When Redis returns, readiness recovers without a restart.

Uninstall

helm uninstall ids --namespace <namespace-name>

Uninstalling keeps the Secrets that the chart generated, such as the control token, and keeps every key in the metadata Redis. Services can't check agent keys while the Identity Service is gone. To remove everything, delete the generated Secrets, the Secrets you created, and the metadata Redis data.

Troubleshooting

The Identity Service returns errors as RFC 7807 problem documents. The type field names the error.