| Title: | Paraphrase-Drift Scoring for LLM Eval Sets |
|---|---|
| Description: | Paraphrase-heavy eval sets drift over time: answers get rewritten, questions get reworded, and nobody re-runs the comparison that justified keeping both versions. eval_drift() scores two texts with three plain similarity measures (word-shingle Jaccard, character-trigram Jaccard, normalized Levenshtein similarity) and one blended drift score. No model calls, no network access, pure base R. Companion to the eval-methodology writing on PastAGI <https://pastagi.com>. |
| Authors: | James Verlander [aut, cre] |
| Maintainer: | James Verlander <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.0 |
| Built: | 2026-10-22 00:08:35 UTC |
| Source: | https://github.com/scrapertweeter3-prog/evaldrift-r |
Deterministic, dependency-free paraphrase-drift scoring so drifted eval pairs get flagged before a human ever looks at them.
eval_drift(a, b, weights = c(shingle = 0.40, trigram = 0.35, lev = 0.25), n = 3)eval_drift(a, b, weights = c(shingle = 0.40, trigram = 0.35, lev = 0.25), n = 3)
a |
First text (e.g. the original answer). |
b |
Second text (e.g. the rewritten answer). |
weights |
Blend weights for shingle, trigram, Levenshtein. |
n |
Word-shingle size. |
A list: shingle_jaccard, trigram_jaccard, levenshtein_sim, drift.
eval_drift("The model refuses the request.", "The model declines to answer the request.")eval_drift("The model refuses the request.", "The model declines to answer the request.")