Troubleshooting Insufficient Space for AOF Rewrite

Redis Enterprise Software
persistence
troubleshooting

Last updated 18, Apr 2024

Symptoms

I am getting errors related to Insufficient Space for AOF Rewrite.

Solution

As rewriting the AOF file can take a significant amount of disk space, we recommend sizing the persistent storage (default location /var/opt/redislabs/persist) to 3x-6x node memory.

Refer to the documentation for sizing recommendations.

Shards distribution

If possible, ensure slave shards are equally distributed across cluster nodes to spread the disk space usage between all the nodes. Verify the location of persistence files using an API request GET /v1/nodes, then check the value of persistent_storage_path. An example JSON body is:

[
    {
      "uid": 1,
      "status": "active",
      "uptime": 262735,
      "total_memory": 6260334592,
      "software_version": "0.90.0-1",
      "ephemeral_storage_size": 20639797248,
      "persistent_storage_path": "/var/opt/redislabs/persist",
      "persistent_storage_size": 20639797248,
      "os_version": "Ubuntu 14.04.2 LTS",
      "ephemeral_storage_path": "/var/opt/redislabs/tmp",
      "architecture": "x86_64",
      "shard_count": 23,
      "public_addr": "",
      "cores": 4,
      "rack_id": "",
      "supported_database_versions": [
        {
          "db_type": "memcached",
          "version": "1.4.17"
        },
        {
          "db_type": "redis",
          "version": "2.6.16"
        },
        {
          "db_type": "redis",
          "version": "2.8.19"
        }
      ],
      "shard_list": [1, 3, 4],
      "addr": "10.0.3.61"
    },
    {
      "uid": 1,
      "status": "active",
      "// additional fields..."
    }
]

Based on the result of the previous command, Verify enough disk space, for example using

df -h /var/opt/redislabs/persist

According to the requirements, the Persistent Storage size should be three times bigger than the RAM (Node RAM).

  • Check the existence of “*.aof. prev” files in the persistence directory. And delete them if not used.
  • In extreme writing scenarios, when AOF is enabled, the AOF rewrite process may require considerably more disk space for database persistence.

References

Refer to the following documents to learn more on the topic.