Data Quality Dimensions
The reliability of any predictive model is fundamentally bounded by the quality of the data used to train and evaluate it. Data quality is not a single property but a composite of several distinct dimensions, each of which can independently degrade model performance if left unaddressed.
Accuracy and Completeness
Accuracy refers to the degree to which recorded values correctly represent the real-world entities or events they describe. Completeness measures the absence of missing values across relevant fields. Incomplete records can bias a model if the missingness is not random — for example, if health records are more likely to omit values for specific demographic groups, models trained without accounting for this pattern may produce systematically different outcomes for those groups.
Consistency and Timeliness
Consistency ensures that the same entity is represented in the same way across different data sources, time periods, or collection methods. Merging records from multiple systems without resolving conflicts in naming conventions, unit systems, or coding standards introduces noise that is difficult to detect and correct downstream. Timeliness addresses how current the data is relative to the prediction target — a model trained on three-year-old customer behavior may not reflect current patterns.
Volume Considerations
The appropriate volume of training data depends on the complexity of the model, the dimensionality of the feature space, and the signal-to-noise ratio of the underlying patterns. No universal minimum sample size applies across model types. However, several practical guidelines emerge from the structure of the problem.
For linear models, a rough heuristic is that stable coefficient estimates require at least ten observations per predictor variable. For tree-based ensembles, performance tends to plateau after a sufficient volume is reached, with returns diminishing as dataset size doubles. For neural architectures, large volumes are typically necessary, though transfer learning can reduce this requirement substantially when the source and target domains share structure.
In Canadian public-sector and healthcare contexts, data scarcity is common for rare events such as specific disease presentations, regulatory violations, or infrastructure failures. Techniques including oversampling, synthetic data generation, and cross-validation with careful stratification are often necessary to produce useful models from limited samples.
Feature Engineering Fundamentals
Feature engineering is the process of transforming raw variables into representations that expose the patterns relevant to the prediction task. Even when a model is capable of learning non-linear relationships, well-designed features can reduce the data volume required for effective learning and improve interpretability.
Encoding Categorical Variables
Categorical features — such as province codes, industry classifications, or product types — must be converted to numerical representations before most algorithms can process them. One-hot encoding creates a binary indicator for each category, which is appropriate when categories carry no ordinal relationship. For high-cardinality categoricals, target encoding or frequency encoding reduces dimensionality but introduces the risk of target leakage if not handled with cross-validation.
Temporal Feature Extraction
Timestamps contain multiple separable signals. Extracting day of week, time of day, month, fiscal quarter, or days since a reference event often reveals cyclical patterns invisible in raw datetime values. Lag features — the value of a variable at one or more prior time steps — are essential for time-series prediction and are discussed further in the Forecasting Systems guide.
Labeling and Annotation
Supervised learning requires that training examples are labeled with ground-truth outcomes. In practice, labels are often derived rather than directly measured. Churn labels in customer retention models are typically defined as absence of activity for a fixed window; fraud labels are derived from investigation outcomes rather than real-time detection; clinical outcome labels depend on follow-up intervals and documentation completeness.
Label quality is often harder to assess than feature quality but has at least as large an effect on model reliability. Systematic errors in labeling — for example, if a diagnostic code is consistently applied with a delay — create patterns that are learned by the model but do not reflect the intended predictive relationship.
Data Pipeline Structure
Predictive models require not only historical training data but also a reliable pipeline that delivers features in the correct format at prediction time. A training-serving skew occurs when the feature computation logic applied during training differs from the logic applied at inference, producing a discrepancy that can degrade production performance even when the model evaluated correctly during development.
Documenting and version-controlling feature transformations alongside model artifacts reduces the risk of training-serving skew. The Implementation Workflows guide covers how these pipeline components are structured during deployment.
Privacy and Governance in Canada
Data collection and retention for predictive modeling in Canada is subject to federal and provincial privacy legislation. PIPEDA governs private-sector organizations operating across provincial boundaries. Quebec's Act respecting the protection of personal information in the private sector (Law 25) introduced additional requirements including mandatory privacy impact assessments for personal data use in automated decision systems.
Data minimization principles — collecting only the data necessary for the specified purpose — reduce both regulatory risk and the surface area for security incidents. This has direct implications for feature engineering: variables collected for one purpose may not be permissible for use in a predictive model without separate consent or a documented legitimate interest assessment.
Common Data Gaps
Several recurring gaps appear across predictive analytics projects regardless of industry.
- Proxy variables: When a direct measurement is unavailable, a correlated proxy is often used. Understanding the conditions under which the proxy breaks down is critical to assessing model reliability under distribution shift.
- Historical depth: Models intended to predict rare or periodic events may require several years of history to capture sufficient examples. Short observation windows produce models with limited reliability for low-frequency phenomena.
- Cross-system linkage: Organizations frequently hold relevant data in siloed systems that cannot be joined without persistent entity identifiers. The absence of reliable identifiers limits the feature richness available to models trained on aggregated or anonymized datasets.
- Representation gaps: Training data that systematically underrepresents specific subgroups produces models that perform differently across those subgroups, often with no visible signal in aggregate evaluation metrics.
For further context on how data requirements interact with the model evaluation process, see Model Selection for Predictive Analytics.