Lessons from Applied ML Projects

The distance between a notebook reporting 95% and a system somebody can use is mostly unglamorous work, and it is where the actual engineering lives.

The metric is a claim, so say what it is a claim about

An accuracy number without a described evaluation set is decoration. What was held out, how was it split, and does the split leak? On eye-tracking data, splitting by frame rather than by participant will quietly inflate every number you report, because the model learns the person instead of the condition.

Split perception from decision

In the autism-screening work, ResNet18 does the perception and a Random Forest makes the call. A single end-to-end network might have matched the accuracy, but nobody could have looked at its output and said why. Keeping the decision stage small buys you an explanation, and in a clinical context the explanation is the point.

Know your failure cases by name

• ›Which inputs does this get confidently wrong?
• ›What happens when a recording is short, noisy, or truncated?
• ›What is the cost asymmetry — is a false negative worse than a false positive here?

A model whose failures you cannot describe is not finished, regardless of its accuracy.