INPUT
INFERENCE-TIME SIGNALS
Offer metadata, structured V4 analysis and canonical raw title plus description are available before review.
SYSTEM R1
OFFER PERSONALIZATION ML R&D
REPRODUCIBLE ML / PREFERENCE LEARNING / MODEL EVALUATION
Controlled machine-learning research into predicting real APPLY / MAYBE / REJECT decisions from automatically available software-offer signals.
PYTHONSCIKIT-LEARNLOGISTIC REGRESSIONTF-IDFMODEL EVALUATIONHUMAN-IN-THE-LOOP
SYSTEM IDENTITY
SYSTEM IDOFFER-PERSONALIZATION-ML-R1
STATUSCOMPLETED EXPERIMENT / R&D
TYPEMACHINE LEARNING RESEARCH
A completed, reproducible research experiment — not a production ML system.
01 / OVERVIEW
PREFERENCE LEARNING AS A CONTROLLED EXPERIMENT
This study tests whether signals available before manual review can predict one user's real APPLY, MAYBE or REJECT decision for a software offer. Its value is the experimental discipline: a frozen dataset, shared folds, out-of-fold predictions, leakage-safe preprocessing and a formal paired benchmark.
The result is deliberately non-binary. Adding lexical text improved several discrimination and probability-quality metrics, but did not improve balanced multiclass performance. MAYBE remained the central failure mode.
02 / RESEARCH QUESTION
WHAT CAN BE KNOWN BEFORE HUMAN REVIEW?
Can automatically observable offer features predict APPLY, MAYBE or REJECT for a specific user and workflow?
TARGET
HUMAN SUPERVISION
Human APPLY / MAYBE / REJECT decisions provide the supervised target.
EXCLUDED
NO REVIEW LEAKAGE
Human notes, rationales, ratings, workflow state and client identity are not model features.
03 / DATASET
SMALL, FROZEN AND EXPLICITLY IDENTIFIED
REVIEWED OFFERS
146
TOTAL FROZEN REVIEWED SET
TRAINABLE EXAMPLES
123
23 EXCLUDED WITHOUT USABLE V4 ANALYSIS
APPLY SUPPORT
54
MAYBE SUPPORT
11
MINORITY / AMBIGUOUS CLASS
REJECT SUPPORT
58
FINGERPRINT / SCHEMA V2 93bac8da8aeeb2899af63fe8d6ed860f903732dbc2c329cc3d91420e43e80244
Schema v2 includes the raw offer title and description in dataset identity. Human review notes and rationales remain supervision evidence, not inference-time model features.
04 / EXPERIMENT DESIGN
ONE DATASET, ONE SPLIT CONTRACT, TWO REPRESENTATIONS
HISTORICAL OFFERS
↓FROZEN DATASET
↓SHARED DETERMINISTIC FOLDS
E1 STRUCTURED31 FEATURES↓OOF PREDICTIONS
E2 STRUCTURED + TEXT31 FEATURES + TF-IDF↓OOF PREDICTIONS
FORMAL PAIRED BENCHMARK
E1 and E2 use the same 123 samples, the same prepared fold memberships, the same classifier configuration and out-of-fold evaluation. The comparison isolates the value of lexical text representation rather than changing several variables at once.
05 / E1 STRUCTURED BASELINE
INTERPRETABLE STRUCTURED BASELINE
STRUCTURED FEATURES
31
OOF ACCURACY
0.691057
OOF MACRO-F1
0.553961
OOF WEIGHTED-F1
0.696732
ROC-AUC
0.682106
LOG LOSS ↓
0.923421
MODEL
ONE-VS-REST LOGISTIC REGRESSION
C=1.0 / class_weight=balanced / solver=liblinear / max_iter=2000 / random_state=42
VALIDATION
5-FOLD STRATIFIED OOF
StratifiedKFold with shuffle enabled and random_state=42.
REPRESENTATION
31 STRUCTURED FEATURES
Only automatically available offer and structured-analysis signals enter the model.
06 / SPLIT SENSITIVITY
VALIDATION ASSUMPTIONS CHANGE THE PICTURE
STRATIFIED MACRO-F1
0.553961
5-FOLD REFERENCE EVALUATION
CLIENT-GROUPED MACRO-F1
0.554107
PRACTICALLY UNCHANGED
TEMPORAL MACRO-F1
0.469055
MATERIAL DEGRADATION
TEMPORAL MAYBE F1
0.000000
VISIBLE CLASS COLLAPSE
Client-grouped macro-F1 was practically unchanged, providing little evidence that repeated client identity drives performance. This is not proof that leakage is absent.
Temporal expanding-window evaluation covered 117 OOF samples and degraded materially; MAYBE F1 collapsed to 0.0. It used the current frozen normalized offer snapshot, so it measures temporal split sensitivity and generalization — not a strict point-in-time reconstruction.
The grouped and temporal artifacts were executed under the previous fingerprint contract. E1 was retrained under fingerprint schema v2 and reproduced the stratified metrics exactly; the grouped and temporal studies were not rerun under v2.
07 / E2 STRUCTURED + TF-IDF
LEXICAL TEXT ADDED WITHOUT LEAKAGE
STRUCTURED FEATURES
31
OOF ACCURACY
0.723577
OOF MACRO-F1
0.552036
OOF WEIGHTED-F1
0.714806
ROC-AUC
0.694671
LOG LOSS ↓
0.843437
TEXT
WORD N-GRAMS
lowercase=true / ngram_range=(1,2) / min_df=2 / max_df=0.95
SPACE
BOUNDED VOCABULARY
max_features=5000 / sublinear_tf=true / norm=l2 / strip_accents=None
LEAKAGE
FIT INSIDE TRAINING FOLD
Vocabulary and IDF are learned inside each training fold, never on the full dataset.
08 / PAIRED BENCHMARK
FORMAL E1 VS E2 COMPARISON
Benchmark: offer_personalization_e1_vs_e2_stratified. All deltas are E2 - E1, including lower-is-better metrics. Macro-F1 is the ranking metric.
| METRIC | E1 | E2 | DELTA / E2 - E1 |
|---|---|---|---|
| MACRO-F1 | 0.553961 | 0.552036 | -0.001925NEARLY FLAT |
| ACCURACY | 0.691057 | 0.723577 | +0.032520IMPROVEMENT |
| WEIGHTED-F1 | 0.696732 | 0.714806 | +0.018074IMPROVEMENT |
| ROC-AUC | 0.682106 | 0.694671 | +0.012566IMPROVEMENT |
| AVERAGE PRECISION | 0.523307 | 0.538060 | +0.014753IMPROVEMENT |
| LOG LOSS ↓ | 0.923421 | 0.843437 | -0.079984IMPROVEMENT |
| BRIER SCORE ↓ | 0.505426 | 0.462909 | -0.042517IMPROVEMENT |
| ECE ↓ | 0.112989 | 0.128552 | +0.015562REGRESSION |
09 / PER-CLASS ANALYSIS
THE IMPROVEMENT DID NOT REACH EVERY CLASS
| METRIC | E1 | E2 | DELTA / E2 - E1 |
|---|---|---|---|
| APPLY | 0.792793 | 0.796460 | +0.003667NEARLY FLAT |
| MAYBE | 0.160000 | 0.105263 | -0.054737REGRESSION |
| REJECT | 0.709091 | 0.754386 | +0.045295IMPROVEMENT |
APPLY STABLE
MAYBE WORSENED / PRIMARY FAILURE MODE
REJECT MATERIALLY IMPROVED
10 / KEY FINDING
BETTER TEXT SIGNAL, UNRESOLVED BALANCED PERFORMANCE
11 / REPRODUCIBILITY
THE EXPERIMENT CAN BE INSPECTED AND REPEATED
REP-01
FROZEN DATASET FINGERPRINT
Immutable dataset identity makes the evaluated sample set explicit.
REP-02
DETERMINISTIC FOLD MEMBERSHIP
Prepared 5-fold membership is stable and reproducible.
REP-03
IDENTICAL E1 / E2 FOLDS
Both experiments are compared on exactly the same validation rows.
REP-04
PREPROCESSING INSIDE FOLDS
TF-IDF vocabulary and IDF are fitted only on each training fold.
REP-05
OOF PREDICTIONS
Every evaluated training row is predicted by a model that did not train on it.
REP-06
JOBLIB ROUNDTRIP
Serialized models are reloaded and validated against expected predictions.
REP-07
FILE-ONLY EVALUATION
Evaluation and comparison operate on immutable experiment files.
REP-08
DATABASE UNCHANGED
Training and evaluation do not mutate the application database.
REP-09
SEPARATE ARTIFACTS
E1, E2 and the formal BenchmarkResult remain independently inspectable.
12 / LIMITATIONS
WHAT THIS RESULT DOES NOT ESTABLISH
- 01
Only 123 examples are trainable, including just 11 MAYBE examples.
- 02
Human labeling ambiguity is a central source of uncertainty.
- 03
Temporal validation degrades materially, including MAYBE F1 of 0.0.
- 04
The temporal study is not a strict point-in-time historical backtest.
- 05
No hyperparameter search or decision-threshold tuning was performed.
- 06
No embeddings or transformer experiment was performed.
- 07
The experiment does not establish production readiness.
- 08
The reported metrics do not establish statistical significance.
13 / NEXT DATA STRATEGY
MORE INDEPENDENT EVIDENCE BEFORE MORE MODEL COMPLEXITY
The highest-value next step is not an E3 model. It is collecting more independently reviewed offers, especially cases near the APPLY ↔ MAYBE and MAYBE ↔ REJECT decision boundaries.
Historical feature snapshots should also be preserved if a strict point-in-time backtest becomes necessary. After the dataset grows, the same frozen benchmark contract can be rerun under a new fingerprint.