Deployment Guide

Audience: Developers

Deployment Guide

This guide provides comprehensive information for deploying GL Open DeepResearch in production environments.

Overview

GL Open DeepResearch is a containerized application that requires several infrastructure components to function properly. This guide covers the technology stack, environment configuration, and deployment considerations.

Technology Stack

Core Application

• Runtime: Python 3.12

• Framework: FastAPI

• Package Manager: uv

• Base Image: asia.gcr.io/gdp-labs/gl-base/python:3.12

Database

• PostgreSQL: Primary database for accounts, profiles, tasks, and task groups

  • Minimum version: PostgreSQL 12+

  • Connection pooling: SQLAlchemy with configurable pool size

Message Queue & Task Processing

• Celery: Asynchronous task processing

• RabbitMQ: Message broker for Celery (alternative: Redis)

• PostgreSQL: Celery result backend

Caching & Streaming

• Redis: Streaming event storage and caching

  • Used for storing task streaming events (24-hour TTL)

  • Required for task stream endpoints (GET /v1/tasks/{task_id}/stream)

External Services

• LLM APIs: Required for deep research execution

• Smart Search SDK: Web search and content retrieval

• Sea Lion API: Optional LLM service integration

Infrastructure Requirements

Minimum Resources

API Service:

• CPU: 2 cores

• Memory: 2GB (minimum), 4GB (recommended)

• Storage: 10GB for application code and dependencies

Worker Service:

• CPU: 4 cores (for concurrent task processing)

• Memory: 4GB (minimum), 8GB (recommended)

• Storage: 10GB for application code and dependencies

Database (PostgreSQL):

• CPU: 2 cores

• Memory: 4GB

• Storage: 100GB (scales with task volume)

Redis:

• CPU: 1 core

• Memory: 2GB

• Storage: 10GB (events expire after 24 hours)

RabbitMQ:

• CPU: 1 core

• Memory: 1GB

• Storage: 5GB

Network Requirements

• Outbound HTTPS access to:

  • LLM API endpoints

  • Smart Search SDK endpoints

  • External web resources (for web scraping)

• Inbound HTTP/HTTPS access on port 8000 (API service)

• Internal network access between:

  • API service ↔ Database

  • API service ↔ Redis

  • Worker service ↔ Database

  • Worker service ↔ Redis

  • Worker service ↔ RabbitMQ

Environment Variables

Required Environment Variables

Database Configuration

# PostgreSQL connection URL
GL_DEEP_RESEARCH_DB_URL=postgresql://user:password@host:port/database

# PostgreSQL database credentials (alternative to connection URL)
POSTGRES_DB=gl_deep_research
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres

# Database connection pool settings
DB_POOL_SIZE=50                    # Connection pool size
DB_MAX_OVERFLOW=10                 # Maximum overflow connections
DB_POOL_TIMEOUT=30                 # Timeout for getting connection (seconds)
DB_POOL_RECYCLE=3600               # Recycle connections after (seconds)
DB_POOL_PRE_PING=true              # Validate connections before use (boolean)

Authentication

# Master API key for administrative operations (account creation, profile management)
GL_DEEP_RESEARCH_MASTER_API_KEY=your-secure-master-api-key-here

# API key configuration
API_KEY_LENGTH=32                  # Length of generated API keys
API_KEY_PREVIEW_PREFIX_LEN=3       # Prefix length for key preview
API_KEY_PREVIEW_SUFFIX_LEN=3       # Suffix length for key preview
API_KEY_MIN_LENGTH=6               # Minimum API key length

LLM Configuration (Tongyi)

# LLM API settings
LLM_BASE_URL=https://your-llm-base-url
LLM_API_KEY=your-llm-api-key
MODEL_NAME=Alibaba-NLP/Tongyi-DeepResearch-30B-A3B

# Optional LLM parameters (can be overridden in profiles)
MAX_INPUT_TOKEN=320000
MAX_RETRIES=10
TEMPERATURE=0.5
TOP_P=0.95
PRESENCE_PENALTY=1.1

GPT-Researcher Configuration

# OpenAI API key for GPT-Researcher
OPENAI_API_KEY=your-openai-api-key

# Sea Lion API (optional LLM service)
SEA_LION_BASE_URL=https://api.sealion.ai/v1
SEA_LION_API_KEY=your-sea-lion-api-key

Smart Search SDK Configuration

SMART_SEARCH_BASE_URL=https://stag-be-smart-search.obrol.id
SMART_SEARCH_IDENTIFIER=your-smart-search-identifier
SMART_SEARCH_SECRET=your-smart-search-secret

Redis Configuration

# Redis connection settings
REDIS_HOST=127.0.0.1               # Default: localhost (use service name in K8s)
REDIS_PORT=6379
REDIS_PASSWORD=password
REDIS_DB="0"                       # Database number as string
REDIS_TLS_ENABLED=false           # Set to true for TLS connections (boolean)

# Redis cache TTL settings
DEFAULT_REDIS_CACHE_TTL=1800       # Default cache TTL (30 minutes)
DEFAULT_STREAM_REDIS_CACHE_TTL=86400  # Stream events TTL (24 hours)
DEFAULT_SMART_SEARCH_TOKEN_CACHE_TTL=3600  # Smart Search token cache TTL (1 hour)

Celery Configuration

# Celery broker and result backend
CELERY_BROKER_URL=amqp://user:password@rabbitmq-host:5672/vhost
CELERY_RESULT_BACKEND=db+postgresql://user:password@host:port/database

# Celery queue configuration
CELERY_QUEUE_NAME=deep_research

# Celery worker configuration
WORKER_PREFETCH_MULTIPLIER=1       # Prefetch multiplier for worker tasks
WORKER_CONCURRENCY=4               # Number of concurrent worker processes

Task Configuration

# Task execution limits
TASK_TIME_LIMIT_SECONDS=3600       # Hard time limit (1 hour)
TASK_SOFT_TIME_LIMIT_SECONDS=3300  # Soft time limit (55 minutes)
TASK_MAX_RETRIES=3                 # Maximum retry attempts
TASK_RETRY_BACKOFF_SECONDS=5       # Fixed delay between retries (seconds)

Webhook Configuration

# Webhook dispatch settings
WEBHOOK_MAX_RETRIES=3              # Maximum retry attempts for webhook dispatch
WEBHOOK_TIMEOUT_SECONDS=10         # Timeout for webhook HTTP requests
WEBHOOK_RETRY_BACKOFF_SECONDS=5    # Delay between webhook retries (seconds)

Table Cleanup Configuration

# Research tasks table cleanup
TABLE_RESEARCH_TASKS_TTL_HOURS=24              # TTL for research tasks (hours)
TABLE_RESEARCH_TASKS_CLEANUP_INTERVAL_MINUTES=60  # Cleanup interval (minutes)

# Task groups table cleanup
TABLE_TASKGROUPS_TTL_HOURS=24                  # TTL for task groups (hours)
TABLE_TASKGROUPS_CLEANUP_INTERVAL_MINUTES=60   # Cleanup interval (minutes)

# Accounts table cleanup
TABLE_ACCOUNTS_TTL_HOURS=24                    # TTL for accounts (hours)
TABLE_ACCOUNTS_CLEANUP_INTERVAL_MINUTES=60    # Cleanup interval (minutes)

# Cleanup lock configuration
CLEANUP_LOCK_KEY=gl_deep_research:cleanup_lock  # Redis key for cleanup lock
CLEANUP_LOCK_TTL_SECONDS=300                   # Lock TTL (seconds)

Application Configuration

# Environment
ENVIRONMENT=development            # Options: production, staging, development

# Default profiles and providers
DEFAULT_PROFILE=TONGYI
AVAILABLE_ORCHESTRATORS=tongyi,gptr
AVAILABLE_TOOLS=web_search,web_search_map,web_search_urls,fetch_web_page,web_page_snippets,web_page_keypoints

# Logging
LOG_FORMAT=""                      # Options: json, text (empty string defaults to json)
LOG_LEVEL=INFO                     # Options: DEBUG, INFO, WARNING, ERROR
LOG_QUERY_PREVIEW_LENGTH=50        # Query preview length in logs

# Secret encryption (for webhook secrets at rest)
GLDR_SECRET=gldr-secret            # Secret key for encryption

# Sentry (optional, for error tracking)
SENTRY_DSN=                        # Sentry DSN for error tracking
SENTRY_PROJECT=gl-deep-research
BUILD_NUMBER=build-number          # Retrieve from CI/CD Build Number for Staging and Production
VERSION_NUMBER=version-number       # Retrieve from CI/CD Version Number for Staging and Production

Profile Configuration

Profiles are loaded from profiles.yaml at application startup. The file path can be customized:

GL_DR_PROFILES_PATH=/path/to/profiles.yaml

If not set, defaults to profiles.yaml in the application root directory.

Deployment Architecture

Container Components

GL Open DeepResearch consists of three main containerized services:

1. API Service: FastAPI application serving HTTP endpoints

2. Worker Service: Celery workers for asynchronous task processing

3. Flower Service (optional): Celery monitoring dashboard

Container Configuration

API Service

• Port: 8000

• Health Check: GET /health

• Readiness Probe: HTTP GET /health (initial delay: 15s, period: 10s)

• Liveness Probe: HTTP GET /health (initial delay: 30s, period: 10s)

• Entrypoint: /app/entrypoint.sh

Worker Service

• Command: celery -A gl_deep_research.worker.celery_app worker --loglevel=INFO --concurrency=4 -Q deep_research

• Concurrency: Configurable via WORKER_CONCURRENCY environment variable (default: 4)

• Prefetch Multiplier: Configurable via WORKER_PREFETCH_MULTIPLIER environment variable (default: 1)

• Queue: deep_research (configurable via CELERY_QUEUE_NAME)

Flower Service (Optional)

• Port: 5555

• Purpose: Monitor Celery tasks and workers

• Access: Internal network only (recommended)

Database Migrations

Database migrations are handled automatically on application startup via Alembic. The entrypoint script runs migrations before starting the application.

Migration Files Location: /app/migration/versions/

Manual Migration (if needed):

alembic upgrade head

Kubernetes Deployment

Helm Chart Structure

The deployment uses Helm charts with the following structure:

deployment/kubernetes/helm-values/
└── stag/
    └── gl-deep-research/
        ├── api/
        │   └── values.yaml          # API service configuration
        ├── worker/
        │   └── values.yaml          # Worker service configuration
        ├── flower/
        │   └── values.yaml          # Flower service configuration
        ├── values-configmap.yaml    # Environment variables (non-sensitive)
        └── values-secret.yaml       # Environment variables (sensitive)

Key Kubernetes Resources

1. Deployments: API, Worker, Flower services

2. Services: ClusterIP services for internal communication

3. Ingress: ALB ingress for external API access

4. ConfigMaps: Non-sensitive configuration

5. Secrets: Sensitive configuration (API keys, passwords)

Resource Limits

API Service:

resources:
  limits:
    cpu: 164m
    memory: 3G
  requests:
    cpu: 11m
    memory: 2G

Worker Service:

resources:
  limits:
    cpu: 2000m
    memory: 8G
  requests:
    cpu: 500m
    memory: 4G

Ingress Configuration

Example ALB ingress configuration:

ingress:
  name: gl-deep-research
  className: alb
  groupname: lb-gdplabs-exploration-shared
  healthcheckpath: "/health"
  rules:
    - host: stag-gl-deep-research.obrol.id
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gl-deep-research-api-service
                port:
                  number: 8000

Security Considerations

API Key Management

• Master API key must be stored securely (Kubernetes Secrets)

• Account API keys are hashed using bcrypt before storage

• API keys are only returned once during account creation

Webhook Security

• Webhook secrets are encrypted at rest using encryption

• Webhook signatures use HMAC-SHA256 for verification

• Set GLDR_SECRET to a secure secret key

Network Security

• Use TLS for Redis connections in production (REDIS_TLS_ENABLED=True)

• Restrict database access to internal network only

• Use service mesh or network policies to limit inter-service communication

Secrets Management

• Store sensitive values in Kubernetes Secrets

• Use secret management tools (e.g., Sealed Secrets, External Secrets Operator)

• Rotate secrets regularly

• Never commit secrets to version control

Monitoring & Observability

Health Checks

• Endpoint: GET /health

• Response: {"status": "ok"}

• Use for Kubernetes readiness and liveness probes

Logging

• Structured JSON logging (when LOG_FORMAT=json)

• Log levels: DEBUG, INFO, WARNING, ERROR

• Include request IDs for traceability

Metrics (Future)

• Task execution metrics

• API request metrics

• Database connection pool metrics

• Redis cache hit/miss rates

Error Tracking

• Sentry integration for error tracking

• Configure SENTRY_DSN and SENTRY_PROJECT

• Set ENVIRONMENT for proper environment tagging

Scaling Considerations

Horizontal Scaling

API Service:

• Stateless design allows horizontal scaling

• Use load balancer to distribute requests

• Consider database connection pool limits when scaling

Worker Service:

• Scale workers based on task queue depth

• Monitor Celery queue metrics

• Adjust concurrency per worker based on task characteristics

Database Scaling

• Use connection pooling (configure DB_POOL_SIZE)

• Consider read replicas for read-heavy workloads

• Monitor connection pool usage

Redis Scaling

• Use Redis Cluster for high availability

• Monitor memory usage (events expire after 24 hours)

• Consider Redis persistence for critical data

Backup & Recovery

Database Backups

• Regular PostgreSQL backups (daily recommended)

• Test backup restoration procedures

• Consider point-in-time recovery for production

Configuration Backups

• Version control for Helm values

• Backup profiles.yaml configuration

• Document environment-specific configurations

Troubleshooting

Common Issues

Database Connection Errors:

• Verify GL_DEEP_RESEARCH_DB_URL is correct

• Check network connectivity between services

• Verify database credentials

• Check connection pool settings

Redis Connection Errors:

• Verify REDIS_HOST, REDIS_PORT, and REDIS_PASSWORD

• Check Redis service availability

• Verify network connectivity

Task Execution Failures:

• Check worker logs for errors

• Verify LLM API credentials and connectivity

• Check Smart Search SDK configuration

• Review task timeout settings

Streaming Events Not Available:

• Verify Redis is accessible from worker service

• Check DEFAULT_STREAM_REDIS_CACHE_TTL setting

• Verify events are being stored (check Redis keys)

Debug Mode

Enable debug logging for troubleshooting:

LOG_LEVEL=DEBUG
LOG_FORMAT=text  # Use text format for easier reading

Production Checklist

• All required environment variables are set

• Master API key is securely generated and stored

• Database migrations are up to date

• Redis is configured and accessible

• RabbitMQ/Celery broker is configured

• Health checks are configured correctly

• Resource limits are set appropriately

• Secrets are stored securely (Kubernetes Secrets)

• Ingress/load balancer is configured

• Monitoring and alerting are set up

• Backup procedures are in place

• Security policies are applied

• Logging is configured and centralized

• Error tracking (Sentry) is configured

• Profiles are configured correctly

• External service credentials are valid

Additional Resources

• Examples - Getting started with the API

• API Documentation - Detailed API endpoint documentation

• Architecture Overview - System architecture details

• Research Profiles - Profile configuration guide