Reliability of Occupational Requirements Survey

 Some vocational witnesses and ALJs have taken to the air, talking about something that they don't really know anything about.  The Occupational Requirements Survey is not done and is therefore not reliable.  The ORS publishes the final first wave data (2018) and the preliminary second wave data (2021).  

The Bureau of Labor Statistics publishes its Handbook of Methods for the ORS.  The Handbook of Methods describes is calculations.  The chapter on calculations contains a subchapter, Reliability of ORS estimates.  In that subchapter, BLS states:

To assist users in confirming the reliability of ORS estimates, the ORS publishes standard errors. Standard errors provide users with a measure of the precision of an estimate to ensure that it is within an acceptable range for their intended purpose. The standard errors are calculated from collected and imputed data. The ORS program is researching methods for estimating the variance excluding imputed values. Examples on how to build confidence intervals using standard errors are included in the standard error section of the ORS website.

 At the bottom of the subchapter, BLS states:

Before publishing any estimate, the ORS program reviews it to make sure that it meets specified statistical reliability and confidentiality requirements. See data review and estimation tabs on the research section of the ORS website. Estimates that are consistent with these procedures are designated as fit for use and released in BLS publications.

Following the research section hyperlink, we get to the statement of estimation:

• Estimation▾
  • Imputation methodology (PDF), 2016
  • Variance estimation (PDF), 2016
  • Estimation processes used in the Occupational Requirements Survey (PDF), 2015
  • ORS pre-production estimation and validation report (PDF), 2015
  • Estimation considerations (PDF), 2014

 Follow the imputation of methodology.  In 2016, the authors speaking for BLS state:

The survey plans to publish all estimates that meet the reliability and confidentiality criteria.

 The collection surveys have OMB clearance #1220-0189.  The design evaluation studied the methodology to "accurately and reliably capture data that are relevant to the SSA's disability program."  The Dynamics of Occupational Change: Implications of the Occupational Requirements Survey states:

While exclusive reliance on adjusting occupational employment weights is not a realistic prospect for ORS to provide up-todate information on job skill requirements, SSA has been unable to incorporate any occupational employment weights in its determinations of available work because of the nature of the DOT. Because the ORS uses standard occupational codes, linking occupation weights and skill scores will not be a problem, and the ORS will capture effects of changes in occupational size over time for SSA’s eligibility determination process for the first time. 

Does the ORS have limitations and problems?  Probably but that begs the economics question:

Compared to what?

Challenge the vocational witness about his/her methodology - compare to the BLS.  Challenge the vocational witness's ability to accumulate and synthesize data -- compare to the BLS.  How does not knowledge about work in nation and the ability to collect and report data about work in the nation compare to the BLS?  

___________________________

Suggested Citation:

Lawrence Rohlfing, Reliability of Occupational Requirements Survey, California Social Security Attorney (May 27, 2022)  https://californiasocialsecurityattorney.blogspot.com