Omitted Variable Bias’s Cousin: How Measurement Error Can Also Bias the Results of Pay Equity Studies
When tasked to quantify potential pay gaps between gender or race groups, economists usually rely on multivariate regression analysis. In pay equity studies, this regression estimates the average pay difference between male and female employees after accounting for important factors which affect pay. However, if any of those factors (known as control variables) are measured with error, pay differences might erroneously be attributed to gender and biased results can emerge.
Omitted and Mismeasured Control Variables: Sources of Bias
Depending on the population being analyzed and a company’s compensation policies, control variables may include pay grades, departments, and job families as well as employees’ tenure, performance measures, and prior experience. Omitting any of these factors from a pay model can lead to omitted variable bias.[1] For instance, if a company values greater prior experience with a salary premium and female employees have less experience on average, then a pay regression that omits prior experience as a control would result in a downward-biased estimate[2] of the female coefficient. In this example, the female regression coefficient would conflate the true female effect and the positive effect of prior experience.
However, biased regression coefficients can also arise from error in the measurement of control variables. For example, prior work experience may be measured with some degree of error or “noise.” Available HR data snapshots typically do not capture the actual amount of relevant prior experience.[3] Economists often proxy for actual prior labor market experience with potential prior experience to mitigate concerns about omitted variable bias. Often calculated as the time elapsed between an employee’s 22nd birthday and their hire date, potential prior experience measures the value of actual prior experience, but, critically, with some error.[4]
How Measuring the Effect of Prior Experience With Error Can Affect Estimates of the Gender Pay Gap
The following simplified example illustrates the effect that an imprecise measure of prior experience can have on estimates of the gender pay gap. In the stylized setting below, actual prior experience functions as the only true determinant of variation in salary levels, so that the true female coefficient would be exactly zero if one controlled for actual prior experience.
Chart 1 plots generated data of 28 employees with actual prior experience values of 10 – 13 years. As the blue line shows, their salaries vary between $70,000 and $82,000. Each additional year of actual prior experience yields an additional $4,000 in salary (the slope of the blue line). The potential prior experience measures (red dots and blue triangles) contain some error ranging between 0 - 1.5 years. As seen in the graph, the spread of the proxy measures around the true values (blue squares) is equal across all salary levels with an average measurement error of zero.[5] Nonetheless, a simple linear regression of pay on observed potential experience estimates an experience premium of about $2,400 per year (the slope of the red line), underestimating the true relationship between experience and compensation by $1,600. This so-called “attenuation bias” towards zero is a well-established result in statistics[6]. Given the noise/error in potential prior experience, it can only partially explain the variation in pay, hence the pay-for-experience relationship becomes weaker. Attenuation bias is proportional to the noise in the measurement. If the spread of the measurement errors was larger, then the red line would be flatter.
Chart 1: The effect of mismeasured prior experience on its relationship with salary levels
Note: The blue line indicates the true relationship between prior work experience and salary. Each additional year of actual experience adds exactly $4,000 to an employee’s salary. The generated data contains seven employees at $70,000 and 10 years of actual work experience and seven employees each at $74,000 (11 yrs), $78,000 (12 yrs) and $82,000 (13 yrs). Since measurement error averages zero in this classical example, the two lines intersect at the mean value of salary and prior experience.
Implications of Measurement Error in Pay Equity Analyses
Importantly, should the effect of prior experience on pay be biased due to measurement error, when average experience levels vary across genders, some of its effect will be misattributed to gender. In Chart 1, data points for female employees tend to be concentrated in the bottom-left corner. Women on average have 1.4 fewer years of prior experience than men and consequently earn about $5,600[7] less, on average. A regression of salaries on potential prior experience and gender results in a negative female coefficient of -$3,000, meaning that females on average are estimated to earn $3,000 less than men with the same amount of potential prior experience.
Knowing the true relationship between salaries, experience and gender in this example, at least two implications are clear:
- Including an imperfect measure of potential prior experience is better than excluding the measure altogether.
- Including the proxy of potential experience mitigates omitted variable bias.
- The estimated raw gender pay difference of -$5,600 moves to
-$3,000 once potential prior experience is included as a control variable.
- However, measurement error inherent in potential prior experience would still lead to the erroneous conclusion that females on average earn less than men.[8]
- In other words, gender picks up some of the effect that is masked due to noise in the prior experience measure.
- The degree of the noise determines the severity of the bias: If the error was 0 to 3 years, the measured return to prior experience would be lower and more of its effect would be misattributed to gender.
Additional Considerations
While often overlooked relative to its more prominent cousin, omitted variable bias, measurement error can also lead to misleading results in pay equity regression analyses. Economists analyzing pay equity should be aware of the issue and take appropriate steps to ensure their results are not sensitive to bias due to measurement error.
While every company and study is different, several diagnostic exercises can be undertaken to gauge the extent of a potential issue. The available HR data itself will generally give a good impression of the distribution between gender, ages, and hire dates and, hence, of the variation in potential prior experience within analysis groups. Further, if most employees spent most of their professional life within the organization, the additional effect of their prior experience on pay may be negligible. Similarly, policies determining (starting) pay variation may give qualitative information on the true effect size of actual experience on salary levels. A random sample of resumes may be supplemented to spot-check the degree of variation between actual (relevant) and potential prior experience among men and women. Finally, the econometrics literature provides additional avenues for remedies or robustness checks if measurement error is suspected.
[1] Wooldridge, M. (2013) Introductory Econometrics: A Modern Approach. 5th Edition, South-Western, Mason, p.88;https://www.edgewortheconomics.com/insight-Regression-In-Antitrust-Litigation.
[2] The term downward-biased estimate refers to an estimate that may be lower than the true value it is attempting to estimate.
[3] It is likely that the value and relevance of prior experience has been more precisely assessed as a key determinant of starting pay at the time of hire.
[4] Because age at hire and age at hire minus 22 are perfectly correlated, the regression coefficient on either of these proxies for potential experience will be identical.
[5] The example roughly follows the canonical example of the classical errors-in-variable model. Stock, J.H. and Watson, M.W. (2006) Introduction to Econometrics. 2nd Edition, Addison Wesley, Boston, p. 320.
[6] Stock and Watson, p.321. The direction of the bias may change if classical errors-in-variable assumptions are violated.
[7] 1.4 x $4,000 = $5,600
[8] Conversely, if females held more actual experience as men in the example but were indeed underpaid, measurement error may lead to the erroneous conclusion of pay parity.
Experts
Principal Consultant