Skip to site content
U.S. Department of Health and Human Services
The Division of Planning, Evaluation, and Research

Primer: Strengthening a Learning Culture at the Department of Labor

Innovation Exchange 2015

Over the last six years, the U.S. Department of Labor (DOL) has made significant progress in institutionalizing a culture of evidence and learning. This primer highlights the main components of DOL's approach.

Key factors

  • Commitment from leadership. The commitment of the Secretary and Deputy Secretary to build a culture of evidence and learning has been critical. That includes their support for the role of the Chief Evaluation Officer (CEO) as an "honest broker" about evidence issues; the requirement from leadership that operating agencies create learning agendas; and top leadership's inclusion of the CEO in key policy and management discussions (e.g., agency performance reviews) so that the CEO is knowledgeable about leadership's priorities.
  • 0.5% set aside for program evaluation. The DOL Secretary can set aside up to 0.5% of appropriated funds from across the Department for evaluation. Once set aside, these funds get transferred to the Chief Evaluation Officer's budget. Many of the agencies also have separate evaluation funds, so this (up to) half percent supplements those funds.
  • Learning agendas. Learning agendas are an important planning tool at DOL. Each operating agency within the Department (there are 15) is required to create a five-year learning agenda, which gets updated every year. The learning agendas highlight priority questions and/or priority studies that the agencies would like to have done. They may also convey themes for their upcoming evaluation efforts or analysis that might be needed. They draw on a range of learning tools, including rigorous impact evaluation (i.e., randomized controlled trials or well designed quasi-experiments), basic analysis or research, and performance analysis (looking at factors that are associated with outcomes). Learning agendas are a catalyst for setting priorities for studies and for conceptualizing studies that need to be done. Evaluations that Congress has required of agencies are also included. Importantly, the learning agendas communicate and engage operating agencies to help focus the resources and services of the Chief Evaluation Office.
  • Chief Evaluation Officer (and office). The role of the CEO is to coordinate, encourage and build the capacity and understanding around evaluation throughout the department. As noted above, the CEO's budget includes the set-aside funds for evaluation, which she can then allocate to advance agencies' learning agendas. The CEO role is not designed to direct or centrally control all evaluation at DOL, but rather to encourage good evaluation. By early 2014 there were 50 studies underway and 40 more being planned. The CEO is a political appointee but is also an experienced researcher and evaluator. 1
  • Building strong relationships with operating agencies around relevance of evaluation. The CEO and her office have worked to create strong and productive relationships with the operating agencies-particularly in terms of showing that evaluation can be useful to those agencies-in at least two ways. First, they have framed their office's work around a spirit of customer service, rather than emphasizing requirements (although learning agenda's are required). Second, the focus of the CEO, just like the learning agendas, is primarily on learning and performance improvement, not an "up or down" verdict on particular programs.2 That has helped agencies to reduce their skittishness about evaluation and to see the CEO's office as useful to them.
  • Connecting performance and evaluation efforts. Performance and evaluation efforts can sometimes be siloed within Departments. DOL has been able to build bridges between these analytical approaches (and between staff in each area). In particular, the CEO also sits in all quarterly performance reviews with agencies run by the Deputy Secretary. In most meetings, there is often some discussion of evaluations underway. The CEO also provides input and assistance about existing or proposed performance measures, including identifying ways to add outcome measures or build more knowledge about whether an agency's (or program's) performance measures are correlated with impacts. That, in turn, can lead to updates to the agency's learning agenda.
  • Departmental evaluation policy statement. The statement (http://www.dol.gov/asp/evaluation/EvaluationPolicy.htm Exit Disclaimer: You Are Leaving www.ihs.gov ), posted on DOL's website in 2014, presents the principles that guide DOL's planning, conduct, and use of program evaluations. It emphasizes a commitment to conducting rigorous, relevant evaluations and to using evidence from evaluations to inform policy and practice. The statement also addresses the topics of rigor, relevance, transparency, independence, and ethics in the conduct of evaluations.
  • Clearinghouse of evidence-based approaches. DOL's Clearinghouse for Labor Evaluation and Research (CLEAR) (http://clear.dol.gov/ Exit Disclaimer: You Are Leaving www.ihs.gov ) is designed to make research on labor topics more accessible to practitioners, policymakers, researchers, and the public more broadly so that it can inform their decisions about labor policies and programs.3

1 For another federal agency model of an "chief evaluation office," check out the Institute of Education Sciences (IES) at the U.S. Department of Education (ED): https://ies.ed.gov Exit Disclaimer: You Are Leaving www.ihs.gov 

2 For example, particularly for agency leaders who have limited experience with evaluation, the CEO may emphasize a "differential treatment" (aka planned variation) approach that test whether some approaches within a program is more effective than others, rather than a treatment/no treatment design.

3 CLEAR is one of several federal agency clearinghouses. One of the most well known is the What Works Clearinghouse Exit Disclaimer: You Are Leaving www.ihs.gov  at ED. For a list of federal clearinghouses, see the My Brother's Keeper page on the ED website: http://www2.ed.gov/rschstat/statistics/surveys/mbk/index.html Exit Disclaimer: You Are Leaving www.ihs.gov .

Evidence of culture change

One piece of evidence that a culture of learning and evidence is being institutionalized at DOL is that an increasing number agency staff are coming to the CEO's office for help on analytic issues and evaluations. (In fact, that office has expanded its staff in order to meet the demand.) The fact that agency staff are initiating these conversations, rather than the CEO's office having to always do so, is a sign of progress. Second, several agencies within DOL now include their learning agendas in their operating plans (which describe agencies' performance goals for the year), even though that is not required. That suggests that agencies see the agendas as useful.