A loan default model: evaluation, governance and communication

Tick each point that your answer made clearly and only once. Points here are this site's own. A real examiner may accept other valid points.

• Accuracy = (70 + 1,810) / 2,000 = 94.0%.
• Precision = 70 / (70 + 90) = 43.8%.
• Recall = 70 / (70 + 30) = 70.0%.
• Accuracy looks high mainly because most loans do not default, so it is not a good measure here.
• Only a little over two in five loans predicted to default actually do, so automatic rejection would turn away many good customers.
• The model finds 70% of the defaults, so 30% of defaults would still be approved.
• The cost of each error should be compared: a missed default loses principal, and a wrongly declined loan loses margin and a customer.
• The threshold can be changed to shift the balance between precision and recall.

Marks counted: 0 of 8

Tick each point that your answer made clearly and only once. Points here are this site's own. A real examiner may accept other valid points.

• Fairness and bias: check that the model does not treat groups unfairly because of the training data, for example by region or by sector.
• Explainability: decisions need reasons, so that applicants can be told why and can challenge a decision.
• Human oversight: a person should review borderline and adverse decisions.
• Validation and approval before use, independent of the builders, and documented.
• Monitoring for drift and regular review, because economic conditions change.
• Data protection and privacy: lawful use of data, minimisation and security.
• Accountability: an owner of the model and an entry in the model inventory.

Marks counted: 0 of 7

Tick each point that your answer made clearly and only once. Points here are this site's own. A real examiner may accept other valid points.

• Start with the decision and the recommendation, not the method.
• Use plain words, for example 'of 160 loans flagged, about 70 would default'.
• Show a simple chart or table, with clear labels and honest scales.
• Express uncertainty as a range and explain that the figures come from a sample of past loans.
• Describe the limits and what the model should not be used for, and ask for the decision needed.

Marks counted: 0 of 5