Predicted auto claims at 0.887 ROC-AUC with plain logistic regression

Blanket premiums misprice individual drivers, so insurers need a score at quote time

Most auto premiums are set by broad rating classes such as age band, postal code and vehicle class, and everyone in a class pays the same. That pricing selects against the insurer. A safe driver charged above their real risk leaves for a competitor who prices them correctly; a risky driver charged below it stays. Each renewal cycle the book gets worse. The countermeasure is a per-policy claim score at quote time. Our team’s question was: from the answers on an intake form, how well can we predict whether a policy will produce a claim, and can an underwriter explain the score? Success meant beating the trivial baseline on ROC-AUC with a model whose drivers can be read off directly.

10,000 policies, a 31% claim rate and two columns 10% empty

The data is car_insurance.csv from a DataCamp project brief: 10,000 customers, 18 columns, one row per policyholder. The columns cover the driver (age band, gender, driving experience, education, income class, credit score, marital status, children), the record (speeding violations, DUIs, past accidents) and the vehicle (ownership, year before or after 2015, sedan or sports car, annual mileage). The target is whether the policy produced a claim: 31.3% did. A model that always predicts “no claim” is right 68.7% of the time, which makes accuracy a weak yardstick.

credit_score was missing for 9.8% of rows and annual_mileage for 9.6%. The team median-imputed both and added a missing-credit flag, expecting a thin credit file to signal risk. The data did not support that: the flag showed no link to claims (χ² = 0.05, p = 0.82).

Four decisions shaped a model an underwriter can read and call

My EDA showed driving experience separates claimants more than any other variable

I profiled all 18 variables: distributions, missingness, and the claim rate by category. Driving experience had the strongest association with claims (χ² = 2,810), followed by age (2,309), income (1,798) and vehicle ownership (1,434). Gender was significant but far weaker (χ² = 114). The EDA also surfaced a pattern that looks wrong: policyholders who claimed had fewer recorded past accidents on average (0.29 against 1.40), and speeding violations ran the same way. The team kept those columns, and I flagged the pattern as a sign that this is a teaching dataset whose relationships should not be carried over to a real book without checking.

The team ranked models on ROC-AUC because accuracy rewards ignoring claimants

With a 69/31 split, accuracy flatters a model that rarely predicts a claim. The team compared logistic regression with a decision tree, a random forest and gradient boosting on one stratified 80/20 split (8,000 training and 2,000 test policies), and ranked them on ROC-AUC with precision, recall and F1 alongside. Every model scored between 0.82 and 0.83 on accuracy, so accuracy could not separate them anyway.

The team kept logistic regression because it matched the trees and explains itself

Logistic regression scored 0.8865 against 0.8814 for gradient boosting. That gap is too small to call a win on one split. With performance tied, the team chose the model an underwriter or a regulator can audit: each standardised coefficient states how much one factor moves the odds of a claim. Gradient boosting was rejected because its gain, if any, did not pay for the loss of a readable model.

I built score_customer() to accept the answers an intake form actually produces

A fitted model in a notebook expects encoded, scaled features. An intake form produces strings such as "high school" or "before 2015", and sometimes no credit score at all. I could have asked whoever calls the model to pre-encode inputs; I rejected that because any mismatch with training would silently change scores. Instead score_customer() applies the training encodings, fills a missing credit score with the training median and sets the flag, scales the features in training column order, and returns a claim probability and a risk tier. On the notebook’s test applicants it returns 0.94 (Very High) for a young driver with no credit history and prior incidents, and 0.0085 (Low) for an experienced owner with a clean record. The tier cut-offs are equal-width placeholders at 0.25, 0.50 and 0.75.

0.887 ROC-AUC on 2,000 held-out policies, level with the trees

• Validation: one stratified 80/20 split (claim rate 31.3% in training, 31.4% in test). No cross-validation was run, so the spread across splits is not measured and the ranking among the top four models is not settled.
• Logistic regression: ROC-AUC 0.8865, accuracy 0.831, precision 0.73, recall 0.73, F1 0.73.
• What that means in policies: of 627 test-set claimants it flagged 455; of the 622 policies it flagged, 455 claimed (73%, against a 31% base rate). It missed 172 claimants and wrongly flagged 167 safe drivers. These counts follow from the reported precision and recall.
• Simpler options: A 6-input version gives 0.868 ROC-AUC. Age alone gives 77.2% accuracy against 68.7% for always predicting no claim.

Recommendation: use the score to route quotes, and set tiers from costs

An underwriting team could use the score to route quotes: price the low tier automatically and send high-tier quotes for review. The model exists as a deployable scoring function and has not been deployed. Before any live use, the tier cut-offs should come from the cost of each error: a missed risky driver costs a claim payout, and a wrongly flagged safe driver may leave for a competitor. If a short form matters more than the last 0.019 of ROC-AUC, the 6-input version is the practical choice. Track the claim rate within each tier against the predicted rate every quarter.

Limitations and what I’d do next

• One split, no cross-validation. The 0.005 gap over gradient boosting could reverse on another split. Next: 5-fold stratified cross-validation with the spread reported for all models.
• Teaching data with odd relationships. Claimants have fewer recorded accidents and violations than non-claimants. Next: validate on a real claims history before any pricing use.
• Imputation saw the test rows. The medians were computed before the split, a small leak. Next: fit the imputer on training rows only, inside a pipeline.
• Fairness review is not done. Gender carries a coefficient of +0.45. Next: a fairness and regulatory review, and a version without gender to measure what removing it costs.
• Default threshold. All precision and recall figures use a 0.5 cut-off. Next: choose the threshold from claim-cost and churn-cost estimates.

Team and credits

Five people split the pipeline into phases: data cleaning, feature engineering, hypothesis testing and modelling. I ran the EDA on all 18 variables at the start and built the score_customer() scoring function at the end. Teammates owned cleaning, feature engineering, hypothesis testing and the four-model comparison.

def score_customer(record: dict, model=logit, scaler=scaler, ...):
    r = dict(record)
    # ordinal encoding for driving_experience / education / income
    # credit_score: flag, then fill with the training median if missing
    # one-hot encoding for vehicle_year and vehicle_type
    # build the vector in training column order, scale, predict
    ...
    return {"claim_probability": prob, "risk_tier": tier}