slicktune.objectives

Training objectives (what the model learns).

Classes

DPOObjective	Direct Preference Optimization (TRL DPOTrainer).
GRPOObjective	Group Relative Policy Optimization (TRL GRPOTrainer).
KTOObjective	Kahneman–Tversky Optimization (TRL KTOTrainer).
ORPOObjective	Odds Ratio Preference Optimization (TRL experimental ORPO).
SFTObjective	Supervised fine-tuning on instruction / chat pairs.

Package Contents

class slicktune.objectives.DPOObjective

Bases: slicktune.types.Objective

Direct Preference Optimization (TRL DPOTrainer).

Parameters:

• beta (float, optional) – KL penalty coefficient, by default 0.1.

• loss_type (str, optional) – TRL DPO loss type, by default "sigmoid".

__slots__ = ()

beta: float = 0.1

loss_type: str = 'sigmoid'

name: str = 'dpo'

required_columns() → list[str]

Return required preference columns.

Returns:

list[str] – Preference triple column names.

class slicktune.objectives.GRPOObjective

Bases: slicktune.types.Objective

Group Relative Policy Optimization (TRL GRPOTrainer).

Uses a verifiable substring reward on must_contain by default (see slicktune.rewards.substring_must_contain_reward()).

Parameters:

• beta (float, optional) – KL penalty coefficient, by default 0.0.

• num_generations (int, optional) – Completions sampled per prompt (must be >= 2), by default 4.

• max_completion_length (int, optional) – Max new tokens per completion, by default 128.

• temperature (float, optional) – Sampling temperature, by default 1.0.

__slots__ = ()

beta: float = 0.0

max_completion_length: int = 128

name: str = 'grpo'

num_generations: int = 4

required_columns() → list[str]

Return required GRPO columns.

Returns:

list[str] – Prompt + verifiable substring column names.

temperature: float = 1.0

class slicktune.objectives.KTOObjective

Bases: slicktune.types.Objective

Kahneman–Tversky Optimization (TRL KTOTrainer).

Parameters:

• beta (float, optional) – KL penalty coefficient, by default 0.1.

• desirable_weight (float, optional) – Weight for desirable (label=True) examples, by default 1.0.

• undesirable_weight (float, optional) – Weight for undesirable (label=False) examples, by default 1.0.

__slots__ = ()

beta: float = 0.1

desirable_weight: float = 1.0

name: str = 'kto'

required_columns() → list[str]

Return required KTO columns.

Returns:

list[str] – Unpaired preference column names.

undesirable_weight: float = 1.0

class slicktune.objectives.ORPOObjective

Bases: slicktune.types.Objective

Odds Ratio Preference Optimization (TRL experimental ORPO).

Parameters:

beta (float, optional) – Odds-ratio penalty coefficient, by default 0.1.

__slots__ = ()

beta: float = 0.1

name: str = 'orpo'

required_columns() → list[str]

Return required preference columns.

Returns:

list[str] – Preference triple column names (same shape as DPO).

class slicktune.objectives.SFTObjective

Bases: slicktune.types.Objective

Supervised fine-tuning on instruction / chat pairs.

__slots__ = ()

name: str = 'sft'

required_columns() → list[str]

Return required dataset columns for SFT.

Returns:

list[str] – Chat messages column name.