slicktune.eval

Phase-2 evaluation: holdout perplexity and judges.

Classes

HoldoutEvalResult	Holdout negative-log-likelihood / perplexity summary.
Judge	Score model generations for quality / correctness.
JudgeReport	Aggregate judge outcomes.
JudgeResult	Outcome of judging one generation.
LLMJudge	Ask an LLM to score a completion from 0–10, normalized to [0, 1].
SubstringJudge	Deterministic judge: pass if must_contain appears in the generation.

Functions

compute_holdout_perplexity(→ HoldoutEvalResult)	Compute mean NLL and perplexity on a holdout SFT JSONL / dataset.
run_judge_on_probes(→ JudgeReport)	Generate replies for probe prompts and score them with judge.

Package Contents

class slicktune.eval.HoldoutEvalResult

Holdout negative-log-likelihood / perplexity summary.

Parameters:

• eval_loss (float) – Mean token NLL on the holdout set.

• perplexity (float) – exp(eval_loss).

• num_examples (int) – Number of evaluated examples.

eval_loss: float

num_examples: int

perplexity: float

class slicktune.eval.Judge

Bases: abc.ABC

Score model generations for quality / correctness.

__slots__ = ()

abstract judge(*, prompt: str, generation: str, **context: Any) → JudgeResult

Score a single generation.

Parameters:

• prompt (str) – User prompt.

• generation (str) – Model completion.

• **context (Any) – Judge-specific extras (e.g. must_contain).

Returns:

JudgeResult – Score and rationale.

class slicktune.eval.JudgeReport

Aggregate judge outcomes.

Parameters:

results (list[JudgeResult]) – Per-example judgments.

property mean_score: float

Return mean score across judged examples.

Returns:

float – Mean in [0, 1], or 0.0 when empty.

results: list[JudgeResult] = []

class slicktune.eval.JudgeResult

Outcome of judging one generation.

Parameters:

• prompt (str) – User prompt.

• generation (str) – Model completion.

• score (float) – Score in [0, 1].

• rationale (str) – Short explanation from the judge.

generation: str

prompt: str

rationale: str

score: float

class slicktune.eval.LLMJudge

Bases: Judge

Ask an LLM to score a completion from 0–10, normalized to [0, 1].

Uses digit-constrained decoding so small models cannot wander into Yes / True instead of a numeric score.

Parameters:

• model (Any) – Causal LM used as the judge (often the same trained model).

• tokenizer (PreTrainedTokenizerBase) – Matching tokenizer.

• max_new_tokens (int, optional) – Unused for constrained scoring (kept for API compatibility), by default 2.

__slots__ = ()

judge(*, prompt: str, generation: str, **context: Any) → JudgeResult

Score via an LLM rubric prompt.

Parameters:

• prompt (str) – User prompt.

• generation (str) – Model completion.

• **context (Any) – Optional criteria string.

Returns:

JudgeResult – Normalized score parsed from the judge reply.

max_new_tokens: int = 2

model: Any

tokenizer: transformers.PreTrainedTokenizerBase

class slicktune.eval.SubstringJudge

Bases: Judge

Deterministic judge: pass if must_contain appears in the generation.

__slots__ = ()

case_sensitive: bool = False

judge(*, prompt: str, generation: str, **context: Any) → JudgeResult

Score via substring match.

Parameters:

• prompt (str) – User prompt.

• generation (str) – Model completion.

• **context (Any) – Must include must_contain.

Returns:

JudgeResult – Score 1.0 or 0.0.

slicktune.eval.compute_holdout_perplexity(*, model: Any, tokenizer: transformers.PreTrainedTokenizerBase, data: str | pathlib.Path | datasets.Dataset, max_length: int = 512) → HoldoutEvalResult

Compute mean NLL and perplexity on a holdout SFT JSONL / dataset.

Parameters:

• model (Any) – Causal LM (base or PEFT).

• tokenizer (PreTrainedTokenizerBase) – Tokenizer.

• data (str or Path or Dataset) – Holdout SFT data with messages (or loadable JSONL).

• max_length (int, optional) – Truncation length, by default 512.

Returns:

HoldoutEvalResult – Loss / perplexity summary.

Raises:

ValueError – If no examples are available.

slicktune.eval.run_judge_on_probes(*, model: Any, tokenizer: transformers.PreTrainedTokenizerBase, probe_path: str | pathlib.Path, judge: Judge, max_new_tokens: int = 128) → JudgeReport

Generate replies for probe prompts and score them with judge.

Parameters:

• model (Any) – Model under evaluation.

• tokenizer (PreTrainedTokenizerBase) – Tokenizer.

• probe_path (str or Path) – Probe JSONL (prompt, must_contain).

• judge (Judge) – Scoring strategy.

• max_new_tokens (int, optional) – Generation length, by default 128.

Returns:

JudgeReport – Aggregate scores.