Encryption

Encryption in Data Store

The Encryption Capability provides transparent field-level encryption for data store chunks. It encrypts chunk content and metadata fields automatically during write operations and decrypts them during read operations, working seamlessly with fulltext and vector capabilities.

Encryption operates transparently—you don't need to access it directly. Once configured, it's automatically used by fulltext and vector capabilities whenever you create, update, or retrieve chunks.

Key Feature: When encryption is enabled, embeddings are generated from plaintext content before encryption. This ensures that semantic search works correctly with encrypted data, as embeddings represent the original content rather than encrypted ciphertext.

Prerequisites

This example specifically requires completion of all setup steps listed on the Prerequisites page.

You should be familiar with these concepts and components:

1. Data Store

2. Fulltext capability

3. Vector capability

How Encryption Works

Transparent Operation

Encryption integrates directly with fulltext and vector capabilities. When you enable encryption on a data store:

1. During Write Operations: Content and metadata fields specified in the encryption configuration are encrypted before being stored.

2. During Read Operations: Encrypted fields are automatically decrypted when chunks are retrieved.

3. Embedding Generation: For vector capability, embeddings are generated from plaintext content before encryption, ensuring semantic search accuracy.

Field-Level Configuration

You can encrypt specific fields:

1. Content field: Encrypt the chunk content using "content".

2. Metadata fields: Encrypt specific metadata fields using dot notation, e.g., "metadata.secret_key", "metadata.secret_value".

3. Nested metadata: Support for nested metadata fields, e.g., "metadata.nested.secret".

Choose an Encryptor

The data store supports multiple encryptor types:

AES-GCM Encryptor

Use AESGCMEncryptor for simple encryption with a direct key:

from gllm_datastore.encryptor.aes_gcm_encryptor import AESGCMEncryptor
import os

# Generate a 256-bit key (32 bytes)
encryption_key = os.urandom(32)
encryptor = AESGCMEncryptor(key=encryption_key)

Key Management: Store your encryption key securely. If you lose the key, you cannot decrypt your data. Consider using a key management service for production applications.

Key Rotating Encryptor

Use KeyRotatingEncryptor for scenarios requiring key rotation:

from gllm_datastore.encryptor.aes_gcm_encryptor import AESGCMEncryptor
from gllm_datastore.encryptor.key_ring.in_memory_key_ring import InMemoryKeyRing
from gllm_datastore.encryptor.key_rotating_encryptor import KeyRotatingEncryptor
import os

# Create a key ring and add multiple keys
key_ring = InMemoryKeyRing()
key_ring.add("key_v1", AESGCMEncryptor(key=os.urandom(32)))
key_ring.add("key_v2", AESGCMEncryptor(key=os.urandom(32)))

# Create encryptor with active key
encryptor = KeyRotatingEncryptor(
    key_ring=key_ring,
    active_key_id="key_v1"
)

Enable Encryption

Enable encryption using the with_encryption() method. This method can be chained with other capability registration methods.

Example: Chroma Data Store

from gllm_datastore.data_store.chroma.data_store import ChromaDataStore, ChromaClientType
from gllm_datastore.encryptor.aes_gcm_encryptor import AESGCMEncryptor
from gllm_inference.em_invoker.openai_em_invoker import OpenAIEMInvoker
import os

em_invoker = OpenAIEMInvoker(model_name="text-embedding-3-small")
encryptor = AESGCMEncryptor(key=os.urandom(32))

store = (
    ChromaDataStore(
        collection_name="secure-docs",
        client_type=ChromaClientType.MEMORY,
    )
    .with_encryption(
        encryptor=encryptor,
        fields={"content", "metadata.secret_key"}
    )
    .with_fulltext()
    .with_vector(em_invoker=em_invoker)
)

Using Encrypted Data Store

Once encryption is enabled, use the data store normally. Encryption and decryption happen automatically:

from gllm_core.schema import Chunk
from gllm_datastore.core.filters import filter as F

# Create chunks with sensitive data
chunks = [
    Chunk(
        id="doc-1",
        content="Sensitive information here",
        metadata={"secret_key": "secret_value", "public_field": "public_value"}
    ),
    Chunk(
        id="doc-2",
        content="Another sensitive document",
        metadata={"secret_key": "another_secret", "public_field": "another_public"}
    ),
]

# Store chunks (content and metadata.secret_key are encrypted automatically)
await store.fulltext.create(chunks)
await store.vector.create(chunks)

# Retrieve chunks (decryption happens automatically)
results = await store.fulltext.retrieve(filters=F.eq("id", "doc-1"))
print(results[0].content)  # Automatically decrypted
print(results[0].metadata["secret_key"])  # Automatically decrypted
print(results[0].metadata["public_field"])  # Not encrypted, returned as-is

# Semantic search works correctly (embeddings generated from plaintext)
semantic_results = await store.vector.retrieve(query="sensitive information")

Limitations

Filter Restrictions

Encrypted fields cannot be used in filters for update or delete operations. The filter values you provide are not encrypted, so they won't match the encrypted data stored in the database.

What works:

• Using non-encrypted fields like id in filters

• Using non-encrypted metadata fields in filters

• Retrieving all chunks and filtering in application code

What doesn't work:

• Filtering by encrypted content: F.eq("content", "encrypted_value")

• Filtering by encrypted metadata: F.eq("metadata.secret_key", "value")

Example:

# ✅ Works: Use non-encrypted field for filtering
await store.fulltext.update(
    filters=F.eq("id", "doc-1"),
    update_values={"metadata.status": "updated"}
)

# ❌ Doesn't work: Cannot filter by encrypted field
await store.fulltext.update(
    filters=F.eq("metadata.secret_key", "value"),  # Won't match encrypted data
    update_values={"metadata.status": "updated"}
)

Field Type Requirements

Encrypted fields must be serializable to strings. Non-string values are automatically converted to strings before encryption.