6.Measuring the Impact of Labor Market Information (The results of three field experiments, funded by the Canadian government – Human Resources and Skills Development Canada – Mr. Michel Turcotte, Ms. Tannis Goddard, Dr. Bryan Hiebert, Ms. Sareena Hopkins, Canada

6.1 Key Issues

  • The session introduced a research program undertaken by the Canadian government to obtain a better match of skills and employment. Measuring career development services and exploring new methodologies to collect evidence about them is becoming increasingly important at the policy level in Canada. Concrete action was taken to obtain a mechanism to evaluate the impact of government-produced labor market information (LIMA) and also find a better methodology which could be applied to other issues at policy level.
  • A multi-disciplinary team was assembled and six projects were funded. Two of the studies were presented in the session. These isolate impacts of LIMA by using small-scale experiments. Whereas most current methods focus on the delivery mode, the studies focused on problem-solving and decision-making.
  • A Career Motion Project undertaken by Social Research and Demonstration Corporation and Training Solutions tested the efficiency of an innovative Web-based career guidance tool, designed to help underemployed, post-secondary education (PSE) graduates make better use of their skills in the labor market.
  • A study undertaken by the Canadian Career Development Foundation (CCDF) and the Canadian Research Working Group in Evidence-Based Practice (CRWG) assessed to what extent assistance by a service provider enhanced the effective use of LIMA, and to what extent independent self-help is a sufficient process for clients to use LIMA effectively. The framework developed by CRWG and used to link process and outcomes was presented, as well as findings.

6.2Challenges

  • The process of recruiting participants for the Career Motion study improved substantially once the marketing message was changed from negative to positive and inspirational.
  • A major self-assessment problem in the CCDF/CRWG project was how clients tended to overstate their own level of skill or knowledge at the start of the intervention. Testing was conducted and some data was found to support this hypothesis. To get around the problem, a Post-Pre Assessment was developed which produced quite an accurate indication of change over time.

6.3Discussion

  • Since the CCDF/CRWG study was undertaken with government funding, the LIMA packages used and the final reports will be in the public domain and accessible on the CRWG website.
  • Marketing questions raised concerned word-of-mouth and peer marketing, and the incentive of 100 dollars offered to clients who completed the assessments. However, clients said the money was not important to them.
  • On methodological issues, further validation of results was suggested since a participant in the program might not say they did not learn anything. Several papers have been written on the Post-Pre Assessment and summarize observations around the inflated self-assessments. Different ways do exist to deal with the problem, one being “concurrent validity checks”.
  • Regarding the impact of the intervention, a question is how the study ruled out additional factors that might play a role, when they assessed participants a year later.
  • The frontline practitioners involved in the CCDF/CRWG study integrated the training and became more structured in their work subsequently, and the involvement of agency managers was a key success factor. Another important point about the study was selecting clients from everyone who walked into participating agencies.
  • The CRWG would welcome the opportunity to further assess how practices are advancing and outcomes are sustained over time.
  • A basic screen was implemented to deal with issues of language. Special LIMA packages were built for people with a low literacy rate.
  • Correlational analysis was done on the three outcomes of knowledge, skills and personal attributes. Knowledge and skills were highly correlated, but personal attributes seem to be closer related to skills than to knowledge.
  • There are issues underpinning LIMA, much of which attempts to promote certain industries or has an agenda. Canada has been different in that a wealth of LIMA is not sectorally but publicly driven. In addition, in building the website for the study, an extensive analysis was conducted of each external link.

6.4Way Forward

  • Career Motion had a greater effect on users who accessed it more frequently, so a question to explore is how to motivate greater use of the program. Another issue is how to incorporate one-on-one interactions with a trained professional or advisor as well as dialogue and networking with peers.
  • The CCDF/CRWG researchers are convinced of the need to get agency managers on board in future, considering that their engagement and support to the counselors providing services was a factor in the success of the project.