{
  "id": "svs_vamana",
  "title": "SVS-VAMANA Vector Search",
  "url": "https://redis.io/docs/latest/develop/ai/redisvl/0.11.0/user_guide/svs_vamana/",
  "summary": "",
  "content": "\n\nIn this notebook, we will explore SVS-VAMANA (Scalable Vector Search with VAMANA graph algorithm), a graph-based vector search algorithm that is optimized to work with compression methods to reduce memory usage. It combines the Vamana graph algorithm with advanced compression techniques (LVQ and LeanVec) and is optimized for Intel hardware.\n\n**How it works**\n\nVamana builds a single-layer proximity graph and prunes edges during construction based on tunable parameters, similar to HNSW but with a simpler structure. The compression methods apply per-vector normalization and scalar quantization, learning parameters directly from the data to enable fast, on-the-fly distance computations with SIMD-optimized layout Vector quantization and compression.\n\n\n**SVS-VAMANA offers:**\n- **Fast approximate nearest neighbor search** using graph-based algorithms\n- **Vector compression** (LVQ, LeanVec) with up to 87.5% memory savings\n- **Dimensionality reduction** (optional, with LeanVec)\n- **Automatic performance optimization** through CompressionAdvisor\n\n**Use SVS-VAMANA when:**\n- Large datasets where memory is expensive\n- Cloud deployments with memory-based pricing\n- When 90-95% recall is acceptable\n- High-dimensional vectors (\u003e1024 dims) with LeanVec compression\n\n\n\n**Table of Contents**\n\n1. [Prerequisites](#Prerequisites)\n2. [Quick Start with CompressionAdvisor](#Quick-Start-with-CompressionAdvisor)\n3. [Creating an SVS-VAMANA Index](#Creating-an-SVS-VAMANA-Index)\n4. [Loading Sample Data](#Loading-Sample-Data)\n5. [Performing Vector Searches](#Performing-Vector-Searches)\n6. [Understanding Compression Types](#Understanding-Compression-Types)\n7. [Hybrid Queries with SVS-VAMANA](#Hybrid-Queries-with-SVS-VAMANA)\n8. [Performance Monitoring](#Performance-Monitoring)\n9. [Manual Configuration (Advanced)](#Manual-Configuration-(Advanced))\n10. [Best Practices and Tips](#Best-Practices-and-Tips)\n11. [Cleanup](#Cleanup)\n\n---\n\n## Prerequisites\n\nBefore running this notebook, ensure you have:\n1. Installed `redisvl` and have that environment active for this notebook\n2. A running Redis Stack instance with:\n   - Redis \u003e= 8.2.0\n   - RediSearch \u003e= 2.8.10\n\nFor example, you can run Redis Stack locally with Docker:\n\n```bash\ndocker run -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest\n```\n\n**Note:** SVS-VAMANA only supports FLOAT16 and FLOAT32 datatypes.\n\n\n```python\n# Import necessary modules\nimport numpy as np\nfrom redisvl.index import SearchIndex\nfrom redisvl.query import VectorQuery\nfrom redisvl.utils import CompressionAdvisor\nfrom redisvl.redis.utils import array_to_buffer\n\n# Set random seed for reproducible results\nnp.random.seed(42)\n```\n\n\n```python\n# Redis connection\nREDIS_URL = \"redis://localhost:6379\"\n```\n\n## Quick Start with CompressionAdvisor\n\nThe easiest way to get started with SVS-VAMANA is using the `CompressionAdvisor` utility, which automatically recommends optimal configuration based on your vector dimensions and performance priorities.\n\n\n```python\n# Get recommended configuration for common embedding dimensions\ndims = 1024  # Common embedding dimensions (works reliably with SVS-VAMANA)\n\nconfig = CompressionAdvisor.recommend(\n    dims=dims,\n    priority=\"balanced\"  # Options: \"memory\", \"speed\", \"balanced\"\n)\n\nprint(\"Recommended Configuration:\")\nfor key, value in config.items():\n    print(f\"  {key}: {value}\")\n\n# Estimate memory savings\nsavings = CompressionAdvisor.estimate_memory_savings(\n    config[\"compression\"],\n    dims,\n    config.get(\"reduce\")\n)\nprint(f\"\\nEstimated Memory Savings: {savings}%\")\n```\n\n    Recommended Configuration:\n      algorithm: svs-vamana\n      datatype: float16\n      graph_max_degree: 64\n      construction_window_size: 300\n      compression: LeanVec4x8\n      reduce: 512\n      search_window_size: 30\n    \n    Estimated Memory Savings: 81.2%\n\n\n## Creating an SVS-VAMANA Index\n\nLet's create an index using the recommended configuration. We'll use a simple schema with text content and vector embeddings.\n\n\n```python\n# Create index schema with recommended SVS-VAMANA configuration\nschema = {\n    \"index\": {\n        \"name\": \"svs_demo\",\n        \"prefix\": \"doc\",\n    },\n    \"fields\": [\n        {\"name\": \"content\", \"type\": \"text\"},\n        {\"name\": \"category\", \"type\": \"tag\"},\n        {\n            \"name\": \"embedding\",\n            \"type\": \"vector\",\n            \"attrs\": {\n                \"dims\": dims,\n                **config,  # Use the recommended configuration\n                \"distance_metric\": \"cosine\"\n            }\n        }\n    ]\n}\n\n# Create the index\nindex = SearchIndex.from_dict(schema, redis_url=REDIS_URL)\nindex.create(overwrite=True)\n\nprint(f\"✅ Created SVS-VAMANA index: {index.name}\")\nprint(f\"   Algorithm: {config['algorithm']}\")\nprint(f\"   Compression: {config['compression']}\")\nprint(f\"   Dimensions: {dims}\")\nif 'reduce' in config:\n    print(f\"   Reduced to: {config['reduce']} dimensions\")\n```\n\n    ✅ Created SVS-VAMANA index: svs_demo\n       Algorithm: svs-vamana\n       Compression: LeanVec4x8\n       Dimensions: 1024\n       Reduced to: 512 dimensions\n\n\n## Loading Sample Data\n\nLet's create some sample documents with embeddings to demonstrate SVS-VAMANA search capabilities.\n\n\n```python\n# Generate sample data\nsample_documents = [\n    {\"content\": \"Machine learning algorithms for data analysis\", \"category\": \"technology\"},\n    {\"content\": \"Natural language processing and text understanding\", \"category\": \"technology\"},\n    {\"content\": \"Computer vision and image recognition systems\", \"category\": \"technology\"},\n    {\"content\": \"Delicious pasta recipes from Italy\", \"category\": \"food\"},\n    {\"content\": \"Traditional French cooking techniques\", \"category\": \"food\"},\n    {\"content\": \"Healthy meal planning and nutrition\", \"category\": \"food\"},\n    {\"content\": \"Travel guide to European destinations\", \"category\": \"travel\"},\n    {\"content\": \"Adventure hiking in mountain regions\", \"category\": \"travel\"},\n    {\"content\": \"Cultural experiences in Asian cities\", \"category\": \"travel\"},\n    {\"content\": \"Financial planning for retirement\", \"category\": \"finance\"},\n]\n\n# Generate random embeddings for demonstration\n# In practice, you would use a real embedding model\ndata_to_load = []\n\n# Use reduced dimensions if LeanVec compression is applied\nvector_dims = config.get(\"reduce\", dims)\nprint(f\"Creating vectors with {vector_dims} dimensions (reduced from {dims} if applicable)\")\n\nfor i, doc in enumerate(sample_documents):\n    # Create a random vector with some category-based clustering\n    base_vector = np.random.random(vector_dims).astype(np.float32)\n    \n    # Add some category-based similarity (optional, for demo purposes)\n    category_offset = hash(doc[\"category\"]) % 100 / 1000.0\n    base_vector[0] += category_offset\n    \n    # Convert to the datatype specified in config\n    if config[\"datatype\"] == \"float16\":\n        base_vector = base_vector.astype(np.float16)\n    \n    data_to_load.append({\n        \"content\": doc[\"content\"],\n        \"category\": doc[\"category\"],\n        \"embedding\": array_to_buffer(base_vector, dtype=config[\"datatype\"])\n    })\n\n# Load data into the index\nindex.load(data_to_load)\nprint(f\"✅ Loaded {len(data_to_load)} documents into the index\")\n\n# Wait a moment for indexing to complete\nimport time\ntime.sleep(2)\n\n# Verify the data was loaded\ninfo = index.info()\nprint(f\"   Index now contains {info.get('num_docs', 0)} documents\")\n```\n\n    Creating vectors with 512 dimensions (reduced from 1024 if applicable)\n    ✅ Loaded 10 documents into the index\n       Index now contains 0 documents\n\n\n## Performing Vector Searches\n\nNow let's perform some vector similarity searches using our SVS-VAMANA index.\n\n\n```python\n# Create a query vector (in practice, this would be an embedding of your query text)\n# Important: Query vector must match the index datatype and dimensions\nvector_dims = config.get(\"reduce\", dims)\nif config[\"datatype\"] == \"float16\":\n    query_vector = np.random.random(vector_dims).astype(np.float16)\nelse:\n    query_vector = np.random.random(vector_dims).astype(np.float32)\n\n# Perform a vector similarity search\nquery = VectorQuery(\n    vector=query_vector.tolist(),\n    vector_field_name=\"embedding\",\n    return_fields=[\"content\", \"category\"],\n    num_results=5\n)\n\nresults = index.query(query)\n\nprint(\"🔍 Vector Search Results:\")\nprint(\"=\" * 50)\nfor i, result in enumerate(results, 1):\n    distance = result.get('vector_distance', 'N/A')\n    print(f\"{i}. [{result['category']}] {result['content']}\")\n    print(f\"   Distance: {distance:.4f}\" if isinstance(distance, (int, float)) else f\"   Distance: {distance}\")\n    print()\n```\n\n    🔍 Vector Search Results:\n    ==================================================\n\n\n## Understanding Compression Types\n\nSVS-VAMANA supports different compression algorithms that trade off between memory usage and search quality. Let's explore the available options.\n\n\n```python\n# Compare different compression priorities\nprint(\"Compression Recommendations for Different Priorities:\")\nprint(\"=\" * 60)\n\npriorities = [\"memory\", \"speed\", \"balanced\"]\nfor priority in priorities:\n    config = CompressionAdvisor.recommend(dims=dims, priority=priority)\n    savings = CompressionAdvisor.estimate_memory_savings(\n        config[\"compression\"],\n        dims,\n        config.get(\"reduce\")\n    )\n    \n    print(f\"\\n{priority.upper()} Priority:\")\n    print(f\"  Compression: {config['compression']}\")\n    print(f\"  Datatype: {config['datatype']}\")\n    if \"reduce\" in config:\n        print(f\"  Dimensionality reduction: {dims} → {config['reduce']}\")\n    print(f\"  Search window size: {config['search_window_size']}\")\n    print(f\"  Memory savings: {savings}%\")\n```\n\n    Compression Recommendations for Different Priorities:\n    ============================================================\n    \n    MEMORY Priority:\n      Compression: LeanVec4x8\n      Datatype: float16\n      Dimensionality reduction: 1024 → 512\n      Search window size: 20\n      Memory savings: 81.2%\n    \n    SPEED Priority:\n      Compression: LeanVec4x8\n      Datatype: float16\n      Dimensionality reduction: 1024 → 256\n      Search window size: 40\n      Memory savings: 90.6%\n    \n    BALANCED Priority:\n      Compression: LeanVec4x8\n      Datatype: float16\n      Dimensionality reduction: 1024 → 512\n      Search window size: 30\n      Memory savings: 81.2%\n\n\n## Compression Types Explained\n\nSVS-VAMANA offers several compression algorithms:\n\n### LVQ (Learned Vector Quantization)\n- **LVQ4**: 4 bits per dimension (87.5% memory savings)\n- **LVQ4x4**: 8 bits per dimension (75% memory savings)\n- **LVQ4x8**: 12 bits per dimension (62.5% memory savings)\n- **LVQ8**: 8 bits per dimension (75% memory savings)\n\n### LeanVec (Compression + Dimensionality Reduction)\n- **LeanVec4x8**: 12 bits per dimension + dimensionality reduction\n- **LeanVec8x8**: 16 bits per dimension + dimensionality reduction\n\nThe CompressionAdvisor automatically chooses the best compression type based on your vector dimensions and priority.\n\n\n```python\n# Demonstrate compression savings for different vector dimensions\ntest_dimensions = [384, 768, 1024, 1536, 3072]\n\nprint(\"Memory Savings by Vector Dimension:\")\nprint(\"=\" * 50)\nprint(f\"{'Dims':\u003c6} {'Compression':\u003c12} {'Savings':\u003c8} {'Strategy'}\")\nprint(\"-\" * 50)\n\nfor dims in test_dimensions:\n    config = CompressionAdvisor.recommend(dims=dims, priority=\"balanced\")\n    savings = CompressionAdvisor.estimate_memory_savings(\n        config[\"compression\"],\n        dims,\n        config.get(\"reduce\")\n    )\n    \n    strategy = \"LeanVec\" if dims \u003e= 1024 else \"LVQ\"\n    print(f\"{dims:\u003c6} {config['compression']:\u003c12} {savings:\u003e6.1f}% {strategy}\")\n```\n\n    Memory Savings by Vector Dimension:\n    ==================================================\n    Dims   Compression  Savings  Strategy\n    --------------------------------------------------\n    384    LVQ4x4         75.0% LVQ\n    768    LVQ4x4         75.0% LVQ\n    1024   LeanVec4x8     81.2% LeanVec\n    1536   LeanVec4x8     81.2% LeanVec\n    3072   LeanVec4x8     81.2% LeanVec\n\n\n## Hybrid Queries with SVS-VAMANA\n\nSVS-VAMANA can be combined with other Redis search capabilities for powerful hybrid queries that filter by metadata while performing vector similarity search.\n\n\n```python\n# Perform a hybrid search: vector similarity + category filter\nhybrid_query = VectorQuery(\n    vector=query_vector.tolist(),\n    vector_field_name=\"embedding\",\n    return_fields=[\"content\", \"category\"],\n    num_results=3\n)\n\n# Add a filter to only search within \"technology\" category\nhybrid_query.set_filter(\"@category:{technology}\")\n\nfiltered_results = index.query(hybrid_query)\n\nprint(\"🔍 Hybrid Search Results (Technology category only):\")\nprint(\"=\" * 55)\nfor i, result in enumerate(filtered_results, 1):\n    distance = result.get('vector_distance', 'N/A')\n    print(f\"{i}. [{result['category']}] {result['content']}\")\n    print(f\"   Distance: {distance:.4f}\" if isinstance(distance, (int, float)) else f\"   Distance: {distance}\")\n    print()\n```\n\n    🔍 Hybrid Search Results (Technology category only):\n    =======================================================\n\n\n## Performance Monitoring\n\nLet's examine the index statistics to understand the performance characteristics of our SVS-VAMANA index.\n\n\n```python\n# Get detailed index information\ninfo = index.info()\n\nprint(\"📊 Index Statistics:\")\nprint(\"=\" * 30)\nprint(f\"Documents: {info.get('num_docs', 0)}\")\n\n# Handle vector_index_sz_mb which might be a string\nvector_size = info.get('vector_index_sz_mb', 0)\nif isinstance(vector_size, str):\n    try:\n        vector_size = float(vector_size)\n    except ValueError:\n        vector_size = 0.0\nprint(f\"Vector index size: {vector_size:.2f} MB\")\n\n# Handle total_indexing_time which might also be a string\nindexing_time = info.get('total_indexing_time', 0)\nif isinstance(indexing_time, str):\n    try:\n        indexing_time = float(indexing_time)\n    except ValueError:\n        indexing_time = 0.0\nprint(f\"Total indexing time: {indexing_time:.2f} seconds\")\n\n# Calculate memory efficiency\nif info.get('num_docs', 0) \u003e 0 and vector_size \u003e 0:\n    mb_per_doc = vector_size / info.get('num_docs', 1)\n    print(f\"Memory per document: {mb_per_doc:.4f} MB\")\n    \n    # Estimate for larger datasets\n    for scale in [1000, 10000, 100000]:\n        estimated_mb = mb_per_doc * scale\n        print(f\"Estimated size for {scale:,} docs: {estimated_mb:.1f} MB\")\nelse:\n    print(\"Memory efficiency calculation requires documents and vector index size \u003e 0\")\n```\n\n    📊 Index Statistics:\n    ==============================\n    Documents: 0\n    Vector index size: 0.00 MB\n    Total indexing time: 1.58 seconds\n    Memory efficiency calculation requires documents and vector index size \u003e 0\n\n\n## Manual Configuration (Advanced)\n\nFor advanced users who want full control over SVS-VAMANA parameters, you can manually configure the algorithm instead of using CompressionAdvisor.\n\n\n```python\n# Example of manual SVS-VAMANA configuration\nmanual_schema = {\n    \"index\": {\n        \"name\": \"svs_manual\",\n        \"prefix\": \"manual\",\n    },\n    \"fields\": [\n        {\"name\": \"content\", \"type\": \"text\"},\n        {\n            \"name\": \"embedding\",\n            \"type\": \"vector\",\n            \"attrs\": {\n                \"dims\": 768,\n                \"algorithm\": \"svs-vamana\",\n                \"datatype\": \"float32\",\n                \"distance_metric\": \"cosine\",\n                \n                # Graph construction parameters\n                \"graph_max_degree\": 64,           # Higher = better recall, more memory\n                \"construction_window_size\": 300,  # Higher = better quality, slower build\n                \n                # Search parameters\n                \"search_window_size\": 40,         # Higher = better recall, slower search\n                \n                # Compression settings\n                \"compression\": \"LVQ4x4\",          # Choose compression type\n                \"training_threshold\": 10000,      # Min vectors before compression training\n            }\n        }\n    ]\n}\n\nprint(\"Manual SVS-VAMANA Configuration:\")\nprint(\"=\" * 40)\nvector_attrs = manual_schema[\"fields\"][1][\"attrs\"]\nfor key, value in vector_attrs.items():\n    if key != \"dims\":  # Skip dims as it's obvious\n        print(f\"  {key}: {value}\")\n\n# Calculate memory savings for this configuration\nmanual_savings = CompressionAdvisor.estimate_memory_savings(\n    \"LVQ4x4\", 768, None\n)\nprint(f\"\\nEstimated memory savings: {manual_savings}%\")\n```\n\n    Manual SVS-VAMANA Configuration:\n    ========================================\n      algorithm: svs-vamana\n      datatype: float32\n      distance_metric: cosine\n      graph_max_degree: 64\n      construction_window_size: 300\n      search_window_size: 40\n      compression: LVQ4x4\n      training_threshold: 10000\n    \n    Estimated memory savings: 75.0%\n\n\n## Best Practices and Tips\n\n### When to Use SVS-VAMANA\n- **Large datasets** (\u003e10K vectors) where memory efficiency matters\n- **High-dimensional vectors** (\u003e512 dimensions) that benefit from compression\n- **Applications** that can tolerate slight recall trade-offs for speed and memory savings\n\n### Parameter Tuning Guidelines\n- **Start with CompressionAdvisor** recommendations\n- **Increase search_window_size** if you need higher recall\n- **Use LeanVec** for high-dimensional vectors (≥1024 dims)\n- **Use LVQ** for lower-dimensional vectors (\u003c1024 dims)\n\n### Performance Considerations\n- **Index build time** increases with higher construction_window_size\n- **Search latency** increases with higher search_window_size\n- **Memory usage** decreases with more aggressive compression\n- **Recall quality** may decrease with more aggressive compression\n\n## Cleanup\n\nClean up the indices created in this demo.\n\n\n```python\n# Clean up demo indices\ntry:\n    index.delete()\n    print(\"Cleaned up svs_demo index\")\nexcept:\n    print(\"- svs_demo index was already deleted or doesn't exist\")\n```\n\n    Cleaned up svs_demo index\n\n",
  "tags": [],
  "last_updated": "2026-04-08T12:21:52-07:00"
}