# Direct Preference Optimization (DPO)

{% hint style="info" %}
The DPO fine-tuning  is currently under pull-request review, so this technique is not yet available for use in the SDK.
{% endhint %}

## What is Direct Preference Optimization (DPO)?

Direct Preference Optimization (DPO) is a preference-based fine-tuning technique that aligns a model using paired comparisons between responses, rather than relying on reinforcement learning or reward models. For each input prompt, DPO uses a `chosen response` (preferred) and a `rejected response` (less preferred) to directly increase the likelihood of generating the chosen output while decreasing the likelihood of the rejected one. This is achieved through a closed-form optimization objective that simplifies training while still capturing preference signals effectively. DPO is particularly useful when you have datasets that express relative human preferences, and it typically produces stable, efficient, and preference-aligned model behaviors.

<details>

<summary>Prerequisites</summary>

Before installing, make sure you have:

1. [Python 3.11+](https://glair.gitbook.io/hello-world/prerequisites#python-v3.11-or-v3.12)
2. [Pip](https://pip.pypa.io/en/stable/installation/) or [Poetry](https://python-poetry.org/docs/)
3. [gcloud CLI](https://cloud.google.com/sdk/docs/install) - required because `gllm-training` is a private library hosted in a private Google Cloud repository

After installing, please run

```bash
gcloud auth login
```

to authorize gcloud to access the Cloud Platform with Google user credentials.

{% hint style="info" %}
Our internal `gllm-training` package is hosted in a secure Google Cloud Artifact Registry.\
You need to authenticate via `gcloud CLI` to access and download the package during installation.
{% endhint %}

4. The minimum requirements:
   1. CUDA-compatible GPU
   2. Recommendation GPU:&#x20;
      1. RTX A5000&#x20;
      2. RTX 40/50 series.&#x20;
   3. Windows/Linux, currently not support for macOS

</details>

## Installation

{% tabs %}
{% tab title="Linux or Windows WSL" %}

```bash
pip install --extra-index-url "https://oauth2accesstoken:$(gcloud auth print-access-token)@glsdk.gdplabs.id/gen-ai-internal/simple/" "gllm-training"
```

{% endtab %}

{% tab title="Windows Powershell" %}

```bash
pip install --extra-index-url "https://oauth2accesstoken:$(gcloud auth print-access-token)@glsdk.gdplabs.id/gen-ai-internal/simple/" "gllm-training"
```

{% endtab %}

{% tab title="Windows Command Prompt" %}

```bash
FOR /F "tokens=*" %T IN ('gcloud auth print-access-token') DO pip install --extra-index-url "https://oauth2accesstoken:%T@glsdk.gdplabs.id/gen-ai-internal/simple/"  "gllm-training"
```

{% endtab %}
{% endtabs %}

## Quickstart

Let's move on to a basic example fine-tuned using `DPOTrainer`. To run DPO fine-tuning, you need to specify a `model name`, `dpo_column_mapping` and `dataset path`. Make sure your data sets contained of `prompt`, `chosen` as a correct response and `rejected` as a rejected response.

```python
# Main Code
from gllm_training import DPOTrainer
from examples.llm_as_judge_reward_function import output_format_reward

dpo_trainer = DPOTrainer(
    model_name="Qwen/Qwen3-0.6b",
    datasets_path="examples/dpo_csv"
)
dpo_trainer.train()

```

## Fine tuning model using YAML file.

We can run experiments in a more structured way by using a YAML file. The current DPO fine-tuning SDK supports both online data from Google Spreadsheets and local data in CSV format.

### Example 1: Fine tuning using online data.

We can prepared our experiment using YAML file with the data trained and validation from google spreadsheet.

{% stepper %}
{% step %}

### Configure environment variables (.env)

Fill in the `GOOGLE_SHEETS_CLIENT_EMAIL` and `GOOGLE_SHEETS_PRIVATE_KEY` fields. If you don’t have these keys, please contact the infrastructure team.

```bash
GOOGLE_SHEETS_CLIENT_EMAIL="your-service-account@project.iam.gserviceaccount.com"
GOOGLE_SHEETS_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\nYOUR_PRIVATE_KEY\n-----END PRIVATE KEY-----\n"
```

{% endstep %}

{% step %}

### Share the spreadsheet

Share your Google Spreadsheet containing the training and validation data with the `GOOGLE_SHEETS_CLIENT_EMAIL`.

```
Share -> add your google sheet client email -> set as editor
```

{% endstep %}

{% step %}

### `Experiment configuration (dpo_experiment_config.yml)`

You can use a YAML file to plan your fine tuning experiments. To fine tuning with YAML, you need to define the required variables in the file.

```yaml
fine_tune_hyperparam_1: &fine_tune_hyperparam_1_conf
  hyperparameters_id: "1"
  max_seq_length: 16000
  torch_dtype: null
  load_in_16bit: true
  trust_remote_code: true
  full_finetuning: false
  enable_thinking: false
  thinking_mode: "off"
  add_generation_prompt: false
  gpu_memory_utilization: 0.9

  # LoraConfig
  r: 16
  lora_alpha: 16
  lora_dropout: 0 # Currently only supports dropout = 0
  bias: "none" # Currently only supports bias = "none"
  target_modules:
    - "q_proj"
    - "k_proj"
    - "v_proj"
    - "o_proj"
    - "gate_proj"
    - "up_proj"
    - "down_proj"
  use_gradient_checkpointing: "unsloth"
  random_state: 3407

  # DPO Config
  num_train_epochs: 1
  learning_rate: 5.0e-6
  weight_decay: 0.0
  warmup_ratio: 0.1
  lr_scheduler_type: "linear"
  optim: "adamw_8bit"
  logging_steps: 2
  per_device_train_batch_size: 2
  gradient_accumulation_steps: 4
  save_steps: 50

  # Output directory
  model_output_dir: "data/dpo/model"

  # Resume training
  resume_from_checkpoint: false
  model_cache_dir: ".hf_cache"

experiment_1:
  experiment_id: "1"
  hyperparameters: *fine_tune_hyperparam_1_conf
  storage_config: *s3_storage_config
  topic: "DPO Fine Tuned"
  model_name: "Qwen/Qwen3-0.6b"
  framework: "unsloth"
  multimodal: false
  spreadsheet_id: "1pibHZZCmyc4NHaXGkMxYhBcHwaOLJeC-3hUa9gIcN9g"
  train_sheet: "training_data"
  prompt_sheet: "prompt"
  prompt_name: "prompt_default"
  dpo_column_mapping_config:
    input_columns:
      prompt: "prompt"
    chosen: "chosen"
    rejected: "rejected"
  save_processed_dataset: true
  output_processed_dir: "data/dpo/dataset"
```

{% endstep %}

{% step %}

### Fine tuning

To run your fine-tuning, you need to load the YAML data using the `YamlConfigLoader` function, and select the experiment ID when executing the load function.

```python
from dotenv import load_dotenv
from gllm_training import YamlConfigLoader
from gllm_training import DPOTrainer
load_dotenv(override=True)

config_loader = YamlConfigLoader(base_dir="./")
config = config_loader.load("dpo_experiment_config.yaml", "experiment_1")
dpo_trainer = DPOTrainer(
    **config
)
dpo_trainer.train()
```

{% endstep %}
{% endstepper %}

### Example 2: Fine tuning using local data.

The remaining hyperparameter configurations for fine-tuning are the same as when using online data. Below is an example YAML configuration for using local data for training and validation.

```yaml
experiment_2:
  experiment_id: "2"
  hyperparameters: *fine_tune_hyperparam_1_conf
  storage_config: *s3_storage_config
  topic: "DPO Fine Tuned"
  model_name: "Qwen/Qwen3-0.6b"
  framework: "unsloth"
  multimodal: false
  datasets_path: "examples/dpo_csv"
  train_filename: "training_data.csv"
  prompt_filename: "prompt_data.csv"
  prompt_name: "prompt_default"
  dpo_column_mapping_config:
    input_columns:
      prompt: "prompt"
    chosen: "chosen"
    rejected: "rejected"
  save_processed_dataset: true
  output_processed_dir: "data/dpo/dataset"
```

## Upload model to cloud storage

When running experiments, we don’t always save the model directly to the cloud. Instead, we may first evaluate its performance before uploading it to cloud storage. To support this workflow, we provide a `save_model` function that allows you to upload the model as a separate step after fine tuning.

{% stepper %}
{% step %}

### Configure environment variable (.env)&#x20;

Fill in the `AWS_ACCESS_KEY`, `AWS_SECRET_KEY` and `AWS_REGION` fields. If you don’t have these keys, please contact the infrastructure team.

```
AWS_ACCESS_KEY="<AWS_ACCESS_KEY>"
AWS_SECRET_KEY="<AWS_SECRET_KEY>"
AWS_REGION="<AWS_REGION>"
```

{% endstep %}

{% step %}

### Upload model

To upload the model, you need to configure the storage configuration and specify the `model path` on `save_model` function. The model path should point to the directory of your best adapter model.

```python
from gllm_training import DPOTrainer
from gllm_training.schema import StorageConfig

finetuner = DPOTrainer(
    model_name="Qwen/Qwen3-0.6b",
    storage_config=StorageConfig(
        provider="s3",
        upload_to_cloud=True,
        object_prefix="fine-tuned-models",
        bucket_name="glair-gen-ai-llm-model",
    ),
)
finetuner.save_existing_model(
    model_path="data/fine_tuned/exp_1/Qwen3-1.7b/exp_id_1:dpo_fine_tune_hyperparam_1:prompt_1:Qwen3-0.6b:adapter"
)
```

{% endstep %}
{% endstepper %}
