When statistical methods are applied to evaluate fair lending compliance, one of the metrics of interest is the statistical significance of measured differences in treatment of applicants. Such differences may be measured by such things as the interest rates charged on loans or the rates of denial for one group versus another (such as males versus females).
When disparities (differences in treatment) are apparent, the question often becomes “are the differences statistically significant?” In this two-part post, we attempt a complete but simple explanation of what this term means and the context by which findings should be interpreted.
There are two concepts that must be understood with regard to application of statistical inference, in general, and fair lending, specifically. The first is that statistical methods are applied in order to estimate something that is unknown and not directly measureable.
Using fair lending compliance as an example, the general question we are usually asking is did/does a lender discriminate with respect to a prohibited factor (such as race or gender) in its lending practices? To evaluate this in the statistical realm, we conduct an analysis of a sample of data. From this data, we draw a conclusion, of which statistical significance is a component.
The key point – we are only interested in the sample of data because of what it tells us about the unknown and what cannot be measured. If this were not the case, then there would be no need to apply statistical methods.
Secondly, in the world of statistics, whether using multivariate methods such as regression or simpler univariate statistics, scientific methods are being applied. The standard of proof scientifically is extremely rigorous and, therefore, rarely (if ever) attainable. Thus, we rely on probabilities in order to determine what conclusions are drawn from data. In fact, most of the knowledge that has been created by science and what is known about the world were derived from the probabilities of something being true as opposed to absolute certainty.
With these two concepts in mind, let’s consider a simplistic example of how statistics can be applied to answer a question and the role of “statistical significance.” Earlier this year the New England Patriots defeated the Atlanta Falcons at Super Bowl 51 in overtime.
Since the overtime period is “sudden death” meaning the first team to score a touchdown wins, the winner of the initial coin toss can be very important. Since a coin has two sides (which we call heads or tails), it is easy to see that the chances of heads on any given toss are the same as tails which means a 50% chance of head and a 50% chance of tails.
So, back to the Super Bowl, as we know, New England won the toss and went on to score and win the game. An Atlanta Falcons fan may ask, is there a possibility that the coin could have been “rigged” or was it a normal and thus a fair coin? And, how would we determine that?
These questions will help us introduce the concept of statistical significance – which we will do in our next post.