Statistical Applications of Split and Compare Quantiles
Salient Features in Action
Model Agnostic
SCQ for Regression Problem
Quantile-based Analysis
Data-Agnostic
Customizable Binning
Post-Deployment Analysis
Split Groups
100.0%
100.0%
100.0%
93.3% 100.0%
100.0%
80.0% 86.7% 93.3% 100.0%
93.3% 100.0%
86.7% 93.3% 100.0%
100.0%
100.0%
100
86.7%
86.7% 80.0%
80.0%
80.0%
66.7% 73.3% 80.0%
80.0%
80
The SCQ plot can provide valuable insights into the accuracy of machine learning models post-deployment when actual values are not available. For instance, consider a prediction of 52000 for which no actual value is available. In this scenario, the Split & Compare Quantile Plot can be used to evaluate the accuracy of the prediction and its associated risks. By examining the plot, we can see that the prediction falls into the bin marked with an orange color quantile in the legend and the 2nd quantile as per the x-axis. Further analysis reveals that in the second quantile, the orange color is accurate only ~30% of the time, while ~13% of the time the prediction falls in value bins that are lower than the actual value, and the remaining ~55% of the time, it falls in the bins above the actual value.
73.3%
60.0% 66.7%
66.7%
60.0% 53.3%
60.0%
60.0%
60.0%
60
53.3%
53.3%
37345.3 - 50909.8 51184.6 - 57559.8 57767.0 - 62276.9 62455.8 - 65108.9 65295.7 - 69573.8 70117.6 - 73683.1 73989.0 - 77836.4 78318.8 - 82656.3 82700.2 - 87736.3 87800.5 - 127222.7
40.0% 46.7%
46.7%
37.5%
40
33.3%
26.7% 20.0%
26.7%
26.7%
25.0%
20.0%
20.0%
20.0%
20
13.3%
6.7% 13.3%
12.5%
6.7%
6.7%
0
Tag
SCQ for classification problem
While machine learning models can be highly accurate, there's inevitably some error associated with them, potentially leading to incorrect predictions. Consider the example of identifying mortgage defaulters in financial services. Even the most sophisticated models can't achieve 100% accuracy and may misidentify non-defaulters as defaulters. In such scenarios, stakeholders can benefit from using Split & Compare Quantile (SCQ) charts, which provide a clear picture of the degree of error associated with a model's predictions. By breaking down the data into deciles, this chart highlights all the labels within each decile, enabling stakeholders to establish the error level the model will likely have at a given threshold.
Assuming a decision threshold of 67% has been identified, which will enable the bank to service 65.25% of customers. Interestingly, this threshold will also result in 11.79% of customers being identified as having a probability of default. In such cases, stakeholders would want to minimize the risk reflected by the model by selecting a threshold that minimizes the error. The first part represents the potential commercial loss resulting from incorrect labeling of defaulters as not likely (false positives), and the second part represents a potential loss of opportunity resulting from false negatives. The stakeholders should use this information to determine the optimal decision boundary that minimizes false positives and negatives. By doing so, the bank can minimize its overall risk exposure while maximizing its potential revenue opportunities.
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
70
1.18 1.18
9.23
20
1.82 1.09
60
7.92
1.51
8.31
50
15
2.18
8.32
40
2.51
7.49
10
2.41
30
7.56
3.70
6.71
20
5
5.92
10
5.72
4.98
4.80
0
0
© 2023 Fractal Analytics Inc. All rights reserved
05
Made with FlippingBook - PDF hosting