25 original quant ML and data questions with answers on point-in-time joins, leakage, validation, calibration, regularisation and production monitoring.
A financial model can fail before fitting begins if its features use revised data or its joins create records that never existed at the decision time. These examples move from data contracts and time-aware evaluation to model choice and production diagnostics. This is an expanded original public guide; it is not limited to the smaller ML section currently present in our dashboard bank.
Define when each value became available, what one row represents and whether join keys are unique. A correct-looking table can still contain future information or duplicated outcomes.
Use evaluation periods, label gaps, costs and metrics that match the intended decision. A high score is useful only in relation to a baseline and an operational objective.
Record preprocessing, feature rules, data versions and monitoring responses alongside model parameters. Changes upstream can invalidate a model even when its saved weights stay fixed.
Try each question before reading its answer. These original teaching examples remain free to read on this page.
Start with information timing and join contracts; model quality cannot repair leaked or duplicated observations.
What must be defined before constructing features for a trading model?
Answer: The exact decision timestamp and which information is available then.
Explanation: A feature can be statistically predictive but unavailable when the trade must be placed. Availability is part of the feature definition.
How should a model join a historical decision to a company report?
Answer: Use the most recent report actually published by the decision time, accounting for revisions.
Explanation: Joining by reporting-period end can use information published later. Point-in-time availability prevents that look-ahead error.
Why might joining two tables by date unexpectedly increase row count?
Answer: The key may be nonunique on both sides, creating a many-to-many join.
Explanation: Check key uniqueness and intended granularity before joining. A larger dataset can reflect duplicated combinations rather than new observations.
Should missing values always be replaced with zero?
Answer: No; first identify what missingness means and choose an appropriate treatment.
Explanation: Zero may be a meaningful economic value. A missing indicator, model-based imputation or exclusion has different assumptions and consequences.
When should a standard scaler's mean and variance be estimated?
Answer: On training data only, then applied unchanged to validation and test data.
Explanation: Fitting preprocessing on all data leaks information about the evaluation distribution. Put learned transformations inside the training pipeline.
Reserve unseen data, handle overlapping labels and compare alternatives on the same task.
What evaluation split matches a model that will predict tomorrow from historical data?
Answer: Train on earlier periods and evaluate on later periods, often in rolling or expanding windows.
Explanation: The split should imitate deployment. Consider label overlap and changes in market conditions when choosing gaps and windows.
Daily examples predict the next 10 days of returns. Why might a simple adjacent train/test split leak information?
Answer: Labels near the boundary share future return observations with the test period.
Explanation: Remove or gap overlapping training labels around the boundary. A chronological feature split alone does not remove label overlap.
What happens if you repeatedly tune a model after checking its test-set score?
Answer: The test set becomes part of model selection and loses its role as an untouched evaluation.
Explanation: Repeated choices adapt to test noise. Keep a separate final holdout or use a justified nested evaluation design.
Why compare a complex forecast model with a simple baseline?
Answer: To measure whether the added complexity improves the task enough to justify its cost.
Explanation: Use the same information, evaluation periods and cost assumptions. A weak or mismatched baseline makes improvements uninformative.
Only 1% of labels are positive. Is 99% accuracy enough to demonstrate a useful classifier?
Answer: No; always predicting negative achieves 99%.
Explanation: Examine precision, recall, calibration and decision cost. Select metrics that reflect what the model is used to do.
Choose measures and penalties that reflect the target decision rather than a convenient headline score.
A detector flags 20 cases, 8 truly positive; there are 10 positives overall. Find precision and recall.
Answer: Precision 40%; recall 80%.
Explanation: Precision divides true positives by flagged cases; recall divides by all actual positives. The denominators answer different questions.
What does it mean for a binary forecast to be calibrated?
Answer: Among cases assigned probability p, the event occurs about a fraction p of the time.
Explanation: Calibration concerns probability reliability, not just ranking. A model can rank well while giving overconfident probabilities.
Should every calibrated binary model use a classification threshold of 0.5?
Answer: No; the threshold should reflect the relative costs and benefits of decisions.
Explanation: With false-positive cost C_FP and false-negative cost C_FN, the simple loss-minimising threshold is C_FP/(C_FP + C_FN).
What does an L2 penalty encourage in a linear model?
Answer: Smaller coefficients through a penalty proportional to their squared magnitude.
Explanation: It trades training fit for controlled complexity. Feature scaling matters because the penalty acts on coefficient size.
Why can an L1 penalty produce exactly zero coefficients?
Answer: Its absolute-value penalty has a corner at zero.
Explanation: The optimum can sit at that corner, inducing sparsity. L2 generally shrinks without producing exact zeros in ordinary unconstrained fits.
Understand what each method optimises and what could change when new observations arrive.
Why can a very deep decision tree overfit?
Answer: It can isolate small training subsets and fit noise with highly specific rules.
Explanation: Restrict depth or leaf size and validate out of sample. Perfect training accuracy is not evidence of generalisation.
What problem does bagging primarily try to reduce?
Answer: Prediction variance by averaging models fit to perturbed training samples.
Explanation: Gains depend on base-model instability and diversity. Highly correlated model errors limit the benefit of averaging.
What does the first principal component maximise?
Answer: Variance of the projected centred data among unit-length directions.
Explanation: PCA is unsupervised: a high-variance direction need not be most predictive of a target. Scaling changes the objective.
What should you monitor when a model moves from research into production?
Answer: Input and label distributions, missingness, calibration, performance and execution outcomes.
Explanation: Stable code does not imply stable data. Tie alerts to a response plan and distinguish unavailable labels from poor performance.
Why is testing a historical stock strategy on today's index members misleading?
Answer: It excludes firms that left the index, including failed or deteriorating firms.
Explanation: The historical investment universe must reflect what was selectable then. A current constituent list uses future survival information.
Revisions, costs and pipeline changes connect research results to actual decisions.
Why can today's downloaded macroeconomic history exaggerate a historical model's performance?
Answer: Historical values may include revisions unavailable at the original decision time.
Explanation: Use vintage releases or explicitly model publication and revision timing. Chronological splitting cannot repair a revised input history.
Does a higher prediction score guarantee a more profitable trading strategy?
Answer: No.
Explanation: Turnover, transaction costs, capacity and position sizing affect economic value. Evaluate the decision rule as well as the forecast.
Does a high model feature-importance score prove that the feature causes the target?
Answer: No.
Explanation: Importance is conditional on the model, data and metric. Correlated features and leakage can distort it; causality requires additional identification.
What must accompany a saved model to reproduce its predictions?
Answer: The feature definitions, preprocessing, versions, input schema, availability rules and model parameters.
Explanation: A model file is only one stage. Unrecorded transformations or data revisions can change outputs without changing its weights.
How would you investigate a model with good average performance but large occasional losses?
Answer: Slice errors by regime, exposure, data quality and decision size, then inspect tail cases.
Explanation: An aggregate metric can hide economically important failure modes. Keep analysis separate from post-hoc tuning on the final holdout.
Each question has its own page with a short answer and a credit to its source. Hints and full worked solutions are in the question bank with a free account.
Our bank summary dated contains 7 ML/Data entries in the bank. Counts describe entries in our curated practice bank, including its source and classification choices. They do not measure employer interview frequency, candidate success rates or how often any one question is asked.
The public examples above were written for these guides. They illustrate the techniques and are not a reproduction or ranking of the bank's sourced questions. The dashboard's broader bank combines sourced questions, worked answers where available and references for book entries without a reproduced solution.
| Subtopic | Bank entries |
|---|---|
| Data Wrangling | 5 |
| Machine Learning | 2 |
Bank difficulty labels: 6 easy · 1 hard.
Use these lesson previews to find the definitions and derivations behind the worked examples.
Continue with 7 ml/data questions in the practice bank.
Open the ml/data bank