Why a two-line rule beats seven tuned models

The Kaggle Titanic competition: 891 passengers to learn from, 418 to predict. Seven model families with nested cross-validation reach 84% accuracy in CV and land at 77% on the leaderboard. A rule built on the gender baseline plus one family correction scores 79.9%. This page explains the gap.

Code on GitHub Kaggle competition

0.799best Kaggle score (woman-boy rule by family, 334 of 418)

0.775best machine-learning submission (7-model ensemble)

79% → 56%accuracy of the ensemble's bets against the gender rule, CV vs test

0.986 → 0.641CV vs Kaggle with a leaking group feature

01Cross-validation vs the leaderboard

Nested CV (outer 5-fold x 3 repeats, inner grid search) for the models, 5-fold x 10 repeats for the rules. Kaggle scores all 418 test passengers, so one passenger is 0.24 points.

Submission	CV	Kaggle
Gender rule	0.787	0.76555
Woman-boy rule, family groups (chosen by CV)	0.840	0.79904
Woman-boy rule, ticket groups	0.837	0.79665
Ensemble (soft vote, 7 models) + ticket groups	0.846	0.77511
Logistic regression + ticket groups	0.841	0.77272
Ensemble (soft vote), no groups	0.835	0.76794
XGBoost, no groups	0.837	0.76076
LightGBM with a leaking group feature	0.986	0.64114

Dot plot of CV accuracy per model, all between 83% and 85%, with Kaggle scores as diamonds between 76.6% and 79.7%; the woman-boy rule has the highest Kaggle score. — Light dots are the three CV repeats, the dark dot their mean. Every submitted model falls six to seven points from CV to Kaggle; the rule falls four, the gender baseline two.

02The models' bets are a coin flip on new passengers

Sex alone explains most of the outcome, so every model is the gender rule plus a few dozen bets: a man predicted to survive, or a woman predicted to die. Since the gender rule was submitted too, the value of the bets on the test set follows from the two scores, net correct = (score − gender score) × 418, without any test label.

Bar chart of the share of bets that are right: logistic regression 78% in CV vs 55% on test, ensemble 79% vs 56%, woman-boy rule 97% vs 93%. — The models' bets come from patterns in 891 passengers (first-class men, large third-class families) that do not transfer. The rule bets only on direct evidence from ticket mates, and 14 of its 15 test bets are right.

The gender rule has nothing to fit and still drops two points from CV to Kaggle, one standard error on 418 passengers. The models drop six to seven, so roughly four points are bets that do not generalize. Hyperparameter tuning adds only 0.5 points of optimism on average.

03The useful feature is also the one that leaks

596 of the 1,309 passengers share one of 216 tickets, and 152 test passengers share a ticket with someone in training. A passenger's ticket-group feature is the survival rate of their ticket mates, and the labels it may use decide what CV reports.

Grouped bars of CV accuracy: using training-fold labels only gives 83.7 to 84.1%, all labels except own 84.2 to 84.3%, all labels including own 89.9 to 98.9%. — With the passenger's own label included, a passenger travelling alone carries the answer in the feature. That model predicts 57% survival on the test set and scores 0.641, below the gender rule.

04With 891 rows, model choice is noise

Fold-to-fold standard deviation is about 2 points, and XGBoost moves from 0.826 to 0.854 between CV repeats. All seven tuned families land between 0.829 and 0.841; on the leaderboard the ensemble beats logistic regression by one passenger.

Left: survival by sex and class, women 97%, 92%, 50% across classes, men 37%, 16%, 14%. Right: survival by title, Mrs 80%, Miss 70%, Master 57%, Rare 29%, Mr 16%. — Exploratory analysis: sex and class dominate, and the title separates boys (Master, 57%) from men (Mr, 16%).

05Family, not ticket

A ticket links companions who are not family and splits families who bought separate tickets. On the same 100 folds (5-fold x 20 repeats), grouping by family key (surname + class + ticket without its last digit + fare + port) lifts the rule's CV from 0.836 to 0.840, better in all 20 repeats; linking by either key gives 0.840 too.

Group	Rule CV	Bet precision (CV)	Kaggle
Same ticket	0.836	97.3%	0.79665
Same family	0.840	98.0%	0.79904
Linked by either	0.840	97.5%	not submitted

The family grouping was chosen on CV and submitted once. It changes 3 test predictions and gains one passenger on Kaggle (15 of its 16 test bets right): the same direction as CV, but one passenger is within the noise.

06Caveats

418 test passengers make any leaderboard difference below about four points fragile; the bet analysis shows the direction of the gap, not its exact size.
No model was chosen with leaderboard feedback: the first seven files came from one pipeline run and were submitted together; the family rule was picked by CV and submitted once.
The public score covers the whole test set, so there is no private score to check against.