John Caskey | Payday Lending: New Research and the Big Question [PDF]This is true not just of payday lending, but all lending.
In an ideal experiment, one would randomly grant payday loans to a group of applicants and randomly deny the loans as well as close substitutes to a similar group of applicants. One would then track indicators of financial stress over time across the two groups.
Predicting who is a good credit risk for a mortgage is devilishly complicated by the fact that your training set is highly skewed. You have for more examples of people who got loans and paid them back than you do people who defaulted. We could have much better models of credit risk if there were as many of the latter as the former, but who's going to give bad credit risks money just for the sake of better data analysis and machine learning? And of course you have no way of knowing the difference between people who were turned down and would have paid and people who were turned down and would have defaulted, so there is no way of differentiating the true negatives from the false negatives.