Data drift Data drift is similar to concept drift, but rather than the target variable, it refers to the shifts in the input data’s statistical properties over time. If the model’s input data changes significantly from what it was trained on, it can lead to inaccurate predictions.
Model decay Even if the underlying data remains consistent, a model’s performance can degrade over time. This could be attributed to alterations in the surrounding environment, affecting the model’s operations, or to untracked variables.
Tracing the root cause of drift To find the source of drift, MM needs to consider several factors, namely feature importance, data input, and bias.
FEATURE IMPORTANCE
Feature importance in ML models can change for various reasons, such as evolving data patterns or alterations in relationships between variables. It can also be influenced by changes in data sources, like the unavailability or modification of previously essential variables. Ensuring the accurate detection of concept drift is vital for optimal monitoring of feature importance. Concept drift refers to the unexpected fluctuation in the statistical properties of the target variable, that can significantly impact the model’s predictive performance. Regular monitoring of feature importance helps identify these changes early, enabling adjustments, retraining, or model rebuilding before significant performance degradation occurs. Techniques like permutation importance, SHAP values, or LIME can be employed to understand and monitor feature importance effectively.
DATA INPUT
Data input monitoring is a vital aspect of MM, focusing on detecting changes in input data that can disrupt original training assumptions that can otherwise lead to model drift, bias, accuracy decline, and privacy concerns.The core of data input monitoring lies in regularly evaluating real-world data to identify shifts in feature distribution. It goes beyond noting changes and aims to understand their potential impact, signaling the need for further analysis or model adjustments. Measuring the probability of changes in data distribution and their effect on outcomes is crucial for identifying accuracy, fairness, and reliance on specific features. This process is similar to analyzing model performance without having access to ground truth data.While accounting for potential dependencies among features can be complex, data input monitoring often treats each feature as independent. This simplifies the analysis while providing an understanding of drift magnitude.
© 2023 Fractal Analytics Inc. All rights reserved
03
Made with FlippingBook - PDF hosting