# Deployment Guide

**Audience:** Developers

## Deployment Guide

This guide provides comprehensive information for deploying GL Open DeepResearch in production environments.

### Overview

GL Open DeepResearch is a containerized application that requires several infrastructure components to function properly. This guide covers the technology stack, environment configuration, and deployment considerations.

### Technology Stack

#### Core Application

* **Runtime**: Python 3.12
* **Framework**: FastAPI
* **Package Manager**: uv
* **Base Image**: `asia.gcr.io/gdp-labs/gl-base/python:3.12`

#### Database

* **PostgreSQL**: Primary database for accounts, profiles, tasks, and task groups
  * Minimum version: PostgreSQL 12+
  * Connection pooling: SQLAlchemy with configurable pool size

#### Message Queue & Task Processing

* **Celery**: Asynchronous task processing
* **RabbitMQ**: Message broker for Celery (alternative: Redis)
* **PostgreSQL**: Celery result backend

#### Caching & Streaming

* **Redis**: Streaming event storage and caching
  * Used for storing task streaming events (24-hour TTL)
  * Required for task stream endpoints (`GET /v1/tasks/{task_id}/stream`)

#### External Services

* **LLM APIs**: Required for deep research execution
* **Smart Search SDK**: Web search and content retrieval
* **Sea Lion API**: Optional LLM service integration

### Infrastructure Requirements

#### Minimum Resources

**API Service:**

* CPU: 2 cores
* Memory: 2GB (minimum), 4GB (recommended)
* Storage: 10GB for application code and dependencies

**Worker Service:**

* CPU: 4 cores (for concurrent task processing)
* Memory: 4GB (minimum), 8GB (recommended)
* Storage: 10GB for application code and dependencies

**Database (PostgreSQL):**

* CPU: 2 cores
* Memory: 4GB
* Storage: 100GB (scales with task volume)

**Redis:**

* CPU: 1 core
* Memory: 2GB
* Storage: 10GB (events expire after 24 hours)

**RabbitMQ:**

* CPU: 1 core
* Memory: 1GB
* Storage: 5GB

#### Network Requirements

* Outbound HTTPS access to:
  * LLM API endpoints
  * Smart Search SDK endpoints
  * External web resources (for web scraping)
* Inbound HTTP/HTTPS access on port 8000 (API service)
* Internal network access between:
  * API service ↔ Database
  * API service ↔ Redis
  * Worker service ↔ Database
  * Worker service ↔ Redis
  * Worker service ↔ RabbitMQ

### Environment Variables

#### Required Environment Variables

**Database Configuration**

```env
# PostgreSQL connection URL
GL_DEEP_RESEARCH_DB_URL=postgresql://user:password@host:port/database

# PostgreSQL database credentials (alternative to connection URL)
POSTGRES_DB=gl_deep_research
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres

# Database connection pool settings
DB_POOL_SIZE=50                    # Connection pool size
DB_MAX_OVERFLOW=10                 # Maximum overflow connections
DB_POOL_TIMEOUT=30                 # Timeout for getting connection (seconds)
DB_POOL_RECYCLE=3600               # Recycle connections after (seconds)
DB_POOL_PRE_PING=true              # Validate connections before use (boolean)
```

**Authentication**

```env
# Master API key for administrative operations (account creation, profile management)
GL_DEEP_RESEARCH_MASTER_API_KEY=your-secure-master-api-key-here

# API key configuration
API_KEY_LENGTH=32                  # Length of generated API keys
API_KEY_PREVIEW_PREFIX_LEN=3       # Prefix length for key preview
API_KEY_PREVIEW_SUFFIX_LEN=3       # Suffix length for key preview
API_KEY_MIN_LENGTH=6               # Minimum API key length
```

**LLM Configuration (Tongyi)**

```env
# LLM API settings
LLM_BASE_URL=https://your-llm-base-url
LLM_API_KEY=your-llm-api-key
MODEL_NAME=Alibaba-NLP/Tongyi-DeepResearch-30B-A3B

# Optional LLM parameters (can be overridden in profiles)
MAX_INPUT_TOKEN=320000
MAX_RETRIES=10
TEMPERATURE=0.5
TOP_P=0.95
PRESENCE_PENALTY=1.1
```

**GPT-Researcher Configuration**

```env
# OpenAI API key for GPT-Researcher
OPENAI_API_KEY=your-openai-api-key

# Sea Lion API (optional LLM service)
SEA_LION_BASE_URL=https://api.sealion.ai/v1
SEA_LION_API_KEY=your-sea-lion-api-key
```

**Smart Search SDK Configuration**

```env
SMART_SEARCH_BASE_URL=https://stag-be-smart-search.obrol.id
SMART_SEARCH_IDENTIFIER=your-smart-search-identifier
SMART_SEARCH_SECRET=your-smart-search-secret
```

**Redis Configuration**

```env
# Redis connection settings
REDIS_HOST=127.0.0.1               # Default: localhost (use service name in K8s)
REDIS_PORT=6379
REDIS_PASSWORD=password
REDIS_DB="0"                       # Database number as string
REDIS_TLS_ENABLED=false           # Set to true for TLS connections (boolean)

# Redis cache TTL settings
DEFAULT_REDIS_CACHE_TTL=1800       # Default cache TTL (30 minutes)
DEFAULT_STREAM_REDIS_CACHE_TTL=86400  # Stream events TTL (24 hours)
DEFAULT_SMART_SEARCH_TOKEN_CACHE_TTL=3600  # Smart Search token cache TTL (1 hour)
```

**Celery Configuration**

```env
# Celery broker and result backend
CELERY_BROKER_URL=amqp://user:password@rabbitmq-host:5672/vhost
CELERY_RESULT_BACKEND=db+postgresql://user:password@host:port/database

# Celery queue configuration
CELERY_QUEUE_NAME=deep_research

# Celery worker configuration
WORKER_PREFETCH_MULTIPLIER=1       # Prefetch multiplier for worker tasks
WORKER_CONCURRENCY=4               # Number of concurrent worker processes
```

**Task Configuration**

```env
# Task execution limits
TASK_TIME_LIMIT_SECONDS=3600       # Hard time limit (1 hour)
TASK_SOFT_TIME_LIMIT_SECONDS=3300  # Soft time limit (55 minutes)
TASK_MAX_RETRIES=3                 # Maximum retry attempts
TASK_RETRY_BACKOFF_SECONDS=5       # Fixed delay between retries (seconds)
```

**Webhook Configuration**

```env
# Webhook dispatch settings
WEBHOOK_MAX_RETRIES=3              # Maximum retry attempts for webhook dispatch
WEBHOOK_TIMEOUT_SECONDS=10         # Timeout for webhook HTTP requests
WEBHOOK_RETRY_BACKOFF_SECONDS=5    # Delay between webhook retries (seconds)
```

**Table Cleanup Configuration**

```env
# Research tasks table cleanup
TABLE_RESEARCH_TASKS_TTL_HOURS=24              # TTL for research tasks (hours)
TABLE_RESEARCH_TASKS_CLEANUP_INTERVAL_MINUTES=60  # Cleanup interval (minutes)

# Task groups table cleanup
TABLE_TASKGROUPS_TTL_HOURS=24                  # TTL for task groups (hours)
TABLE_TASKGROUPS_CLEANUP_INTERVAL_MINUTES=60   # Cleanup interval (minutes)

# Accounts table cleanup
TABLE_ACCOUNTS_TTL_HOURS=24                    # TTL for accounts (hours)
TABLE_ACCOUNTS_CLEANUP_INTERVAL_MINUTES=60    # Cleanup interval (minutes)

# Cleanup lock configuration
CLEANUP_LOCK_KEY=gl_deep_research:cleanup_lock  # Redis key for cleanup lock
CLEANUP_LOCK_TTL_SECONDS=300                   # Lock TTL (seconds)
```

**Application Configuration**

```env
# Environment
ENVIRONMENT=development            # Options: production, staging, development

# Default profiles and providers
DEFAULT_PROFILE=TONGYI
AVAILABLE_ORCHESTRATORS=tongyi,gptr
AVAILABLE_TOOLS=web_search,web_search_map,web_search_urls,fetch_web_page,web_page_snippets,web_page_keypoints

# Logging
LOG_FORMAT=""                      # Options: json, text (empty string defaults to json)
LOG_LEVEL=INFO                     # Options: DEBUG, INFO, WARNING, ERROR
LOG_QUERY_PREVIEW_LENGTH=50        # Query preview length in logs

# Secret encryption (for webhook secrets at rest)
GLDR_SECRET=gldr-secret            # Secret key for encryption

# Sentry (optional, for error tracking)
SENTRY_DSN=                        # Sentry DSN for error tracking
SENTRY_PROJECT=gl-deep-research
BUILD_NUMBER=build-number          # Retrieve from CI/CD Build Number for Staging and Production
VERSION_NUMBER=version-number       # Retrieve from CI/CD Version Number for Staging and Production
```

#### Profile Configuration

Profiles are loaded from `profiles.yaml` at application startup. The file path can be customized:

```env
GL_DR_PROFILES_PATH=/path/to/profiles.yaml
```

If not set, defaults to `profiles.yaml` in the application root directory.

### Deployment Architecture

#### Container Components

GL Open DeepResearch consists of three main containerized services:

1. **API Service**: FastAPI application serving HTTP endpoints
2. **Worker Service**: Celery workers for asynchronous task processing
3. **Flower Service** (optional): Celery monitoring dashboard

#### Container Configuration

**API Service**

* **Port**: 8000
* **Health Check**: `GET /health`
* **Readiness Probe**: HTTP GET `/health` (initial delay: 15s, period: 10s)
* **Liveness Probe**: HTTP GET `/health` (initial delay: 30s, period: 10s)
* **Entrypoint**: `/app/entrypoint.sh`

**Worker Service**

* **Command**: `celery -A gl_deep_research.worker.celery_app worker --loglevel=INFO --concurrency=4 -Q deep_research`
* **Concurrency**: Configurable via `WORKER_CONCURRENCY` environment variable (default: 4)
* **Prefetch Multiplier**: Configurable via `WORKER_PREFETCH_MULTIPLIER` environment variable (default: 1)
* **Queue**: `deep_research` (configurable via `CELERY_QUEUE_NAME`)

**Flower Service (Optional)**

* **Port**: 5555
* **Purpose**: Monitor Celery tasks and workers
* **Access**: Internal network only (recommended)

#### Database Migrations

Database migrations are handled automatically on application startup via Alembic. The entrypoint script runs migrations before starting the application.

**Migration Files Location**: `/app/migration/versions/`

**Manual Migration** (if needed):

```bash
alembic upgrade head
```

### Kubernetes Deployment

#### Helm Chart Structure

The deployment uses Helm charts with the following structure:

```
deployment/kubernetes/helm-values/
└── stag/
    └── gl-deep-research/
        ├── api/
        │   └── values.yaml          # API service configuration
        ├── worker/
        │   └── values.yaml          # Worker service configuration
        ├── flower/
        │   └── values.yaml          # Flower service configuration
        ├── values-configmap.yaml    # Environment variables (non-sensitive)
        └── values-secret.yaml       # Environment variables (sensitive)
```

#### Key Kubernetes Resources

1. **Deployments**: API, Worker, Flower services
2. **Services**: ClusterIP services for internal communication
3. **Ingress**: ALB ingress for external API access
4. **ConfigMaps**: Non-sensitive configuration
5. **Secrets**: Sensitive configuration (API keys, passwords)

#### Resource Limits

**API Service:**

```yaml
resources:
  limits:
    cpu: 164m
    memory: 3G
  requests:
    cpu: 11m
    memory: 2G
```

**Worker Service:**

```yaml
resources:
  limits:
    cpu: 2000m
    memory: 8G
  requests:
    cpu: 500m
    memory: 4G
```

#### Ingress Configuration

Example ALB ingress configuration:

```yaml
ingress:
  name: gl-deep-research
  className: alb
  groupname: lb-gdplabs-exploration-shared
  healthcheckpath: "/health"
  rules:
    - host: stag-gl-deep-research.obrol.id
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gl-deep-research-api-service
                port:
                  number: 8000
```

### Security Considerations

#### API Key Management

* Master API key must be stored securely (Kubernetes Secrets)
* Account API keys are hashed using bcrypt before storage
* API keys are only returned once during account creation

#### Webhook Security

* Webhook secrets are encrypted at rest using encryption
* Webhook signatures use HMAC-SHA256 for verification
* Set `GLDR_SECRET` to a secure secret key

#### Network Security

* Use TLS for Redis connections in production (`REDIS_TLS_ENABLED=True`)
* Restrict database access to internal network only
* Use service mesh or network policies to limit inter-service communication

#### Secrets Management

* Store sensitive values in Kubernetes Secrets
* Use secret management tools (e.g., Sealed Secrets, External Secrets Operator)
* Rotate secrets regularly
* Never commit secrets to version control

### Monitoring & Observability

#### Health Checks

* **Endpoint**: `GET /health`
* **Response**: `{"status": "ok"}`
* Use for Kubernetes readiness and liveness probes

#### Logging

* Structured JSON logging (when `LOG_FORMAT=json`)
* Log levels: DEBUG, INFO, WARNING, ERROR
* Include request IDs for traceability

#### Metrics (Future)

* Task execution metrics
* API request metrics
* Database connection pool metrics
* Redis cache hit/miss rates

#### Error Tracking

* Sentry integration for error tracking
* Configure `SENTRY_DSN` and `SENTRY_PROJECT`
* Set `ENVIRONMENT` for proper environment tagging

### Scaling Considerations

#### Horizontal Scaling

**API Service:**

* Stateless design allows horizontal scaling
* Use load balancer to distribute requests
* Consider database connection pool limits when scaling

**Worker Service:**

* Scale workers based on task queue depth
* Monitor Celery queue metrics
* Adjust concurrency per worker based on task characteristics

#### Database Scaling

* Use connection pooling (configure `DB_POOL_SIZE`)
* Consider read replicas for read-heavy workloads
* Monitor connection pool usage

#### Redis Scaling

* Use Redis Cluster for high availability
* Monitor memory usage (events expire after 24 hours)
* Consider Redis persistence for critical data

### Backup & Recovery

#### Database Backups

* Regular PostgreSQL backups (daily recommended)
* Test backup restoration procedures
* Consider point-in-time recovery for production

#### Configuration Backups

* Version control for Helm values
* Backup `profiles.yaml` configuration
* Document environment-specific configurations

### Troubleshooting

#### Common Issues

**Database Connection Errors:**

* Verify `GL_DEEP_RESEARCH_DB_URL` is correct
* Check network connectivity between services
* Verify database credentials
* Check connection pool settings

**Redis Connection Errors:**

* Verify `REDIS_HOST`, `REDIS_PORT`, and `REDIS_PASSWORD`
* Check Redis service availability
* Verify network connectivity

**Task Execution Failures:**

* Check worker logs for errors
* Verify LLM API credentials and connectivity
* Check Smart Search SDK configuration
* Review task timeout settings

**Streaming Events Not Available:**

* Verify Redis is accessible from worker service
* Check `DEFAULT_STREAM_REDIS_CACHE_TTL` setting
* Verify events are being stored (check Redis keys)

#### Debug Mode

Enable debug logging for troubleshooting:

```env
LOG_LEVEL=DEBUG
LOG_FORMAT=text  # Use text format for easier reading
```

### Production Checklist

* [ ] All required environment variables are set
* [ ] Master API key is securely generated and stored
* [ ] Database migrations are up to date
* [ ] Redis is configured and accessible
* [ ] RabbitMQ/Celery broker is configured
* [ ] Health checks are configured correctly
* [ ] Resource limits are set appropriately
* [ ] Secrets are stored securely (Kubernetes Secrets)
* [ ] Ingress/load balancer is configured
* [ ] Monitoring and alerting are set up
* [ ] Backup procedures are in place
* [ ] Security policies are applied
* [ ] Logging is configured and centralized
* [ ] Error tracking (Sentry) is configured
* [ ] Profiles are configured correctly
* [ ] External service credentials are valid

### Additional Resources

* [Examples](https://gdplabs.gitbook.io/gl-open-deepresearch/users-guide/quick-start-guide) - Getting started with the API
* [API Documentation](https://github.com/GDP-ADMIN/gl-deep-research/blob/main/applications/gl-deep-research/docs/api-contract/README.md) - Detailed API endpoint documentation
* [Architecture Overview](https://gdplabs.gitbook.io/gl-open-deepresearch/developers-guide/introduction) - System architecture details
* [Research Profiles](https://gdplabs.gitbook.io/gl-open-deepresearch/developers-guide/customization/profiles) - Profile configuration guide
