Methodology & Results

Methodology

This study was conducted under the governance of the SRS Task Force and the Steering Committee. The role of the SRS Task Force was to provide policy and administrative guidance to the study. The role of the Steering Committee was to provide intellectual guidance to the conduct of the study, and to ensure that the study results will meet the information needs of CCDA in its effort to enhance the Red Seal program standard. This section describes the methodological approaches used to address the project objectives and to provide answers to the research questions.

A three-pronged approach was used to gather information for this study, namely:

  1. a national and international literature review;
  2. a case study of exemplary occupational standards; and
  3. an online consultation of national and international key informants.

Review of Literature

WDM project staff conducted a comprehensive review of Canadian and international literature focusing on innovative and best practices in current use for structuring and developing occupational standards. The literature and research publications to be reviewed were carefully selected in order to address the research objectives and provide answers to the research questions formulated for this study.

Relevant materials (government documents, research reports, technical reports, journal articles and other published materials) were accessed from specialized databases, the Internet and library facilities using key word searches. The search was restricted to literature published between 2005 and 2010. The searches were also restricted to the literature published in the context of more developed countries, as well as those available in English or French. When it was absolutely necessary to review materials published in other languages, WDM project staff sought the collaboration of international collaborators, as was the case with some documents from China and Germany.

Requests for information were posted on specialized mailing lists and were also sent to specialized agencies including United Nations Educational, Scientific and Cultural Organization (UNESCO), ILO, and research centers. To provide diversity in perspective, the material selected for review was drawn not only from empirical research, but also from the following additional sources:

  • National and international reports;
  • Legislation pertaining to occupational standards;
  • Manuals, handbooks and guides for developing standards;
  • Country reports;
  • Academic literature;
  • Policy reports; and
  • Articles dealing with practitioners' opinions.

WDM project staff read the abstracts of all publications identified by the initial searches to further restrict the literature review to those publications that specifically examine the aspects of occupational standards that were directly related to the study objectives and the research questions. In addition, the reference lists of publications selected for review was scanned for other pertinent sources that might not have identified by the database searches.

Case Study of Best Practices and Exemplary Occupational Standards

The second stage of the literature review consisted of an in-depth case study of exemplary national and transnational occupational standards. The purpose was to identify innovative and best practices for structuring and developing occupational standards. The standards used for the case study were identified through a literature review and discussions with several international expert-practitioners. Input from the Project Steering Committee was also sought for the selection of the cases to be investigated.

Ten national standards and three transnational standards were selected for the review and analysis process.

These standards were from the following geographical areas: Australia; the Asia Pacific Region; China; the European Union; France; Germany; New Zealand; Quebec; South Africa; Texas, USA; the United Kingdom; and the USA (SkillsNET). All the information generated by the analysis process was entered into the respective cells of a template that was developed in collaboration with members of the Project Steering Committee. The assistance of key informants from the source countries was sought to address information gaps in the published literature. Input from social media networks was also used for the same purpose.

Key Informants' Consultation

The purpose of the key informants' consultation was to elicit useful information from experts and users of occupational standard regarding the structure and development processes of the standards. An online survey developed with the assistance of the Project Steering Committee was used for gathering information from key informants. A "Call to Participate" in an online consultation was sent to 200 national and international key informants. A follow-up friendly reminder was also sent to non-respondents two weeks after the original mailing. Telephone and Skype conference calls were used to clarify responses and to elicit more in depth information. Twenty-four (24) respondents completed the online survey, almost half of whom were from Canada (11). Representatives from seven Canadian jurisdictions, as recommended by the CCDA, participated in the consultation. Other key informants were from various other countries, including, India, Indonesia, Germany, United States, and United Kingdom. Follow up with non-respondents suggests that the low response rate (12 per cent) is due to the highly technical nature of the questionnaire. Several key informants responsible for the development of occupational standards were not familiar with certain technical aspects, which are usually handled by consultants.

The consultation process was designed to generate information on the following key issues and concerns:

  • Importance of structuring and developing occupational standards to meet the current and future needs of the Red Seal program;
  • Time required for developing and updating standards;
  • Advantages and disadvantages of various occupational standard structures and development processes;
  • Ensuring stakeholders' acceptance for occupational standards;
  • Rationale for using a particular standard structure;
  • Desirable improvements to the structure and development processes of standards;
  • Strategies to engage stakeholders in the design and modification of the structure of standards;
  • Strategies to ensure that occupational standards comply with all applicable legal frameworks;
  • Standard quality assurance and quality control measures;
  • Specific approaches used to ensure the validity of occupational standards; and
  • Strategies used for updating occupational standards

The text analysis features built into the Survey Monkey platform were used for analyzing responses to the open-ended questions. This text analysis option enabled scanning the text for most commonly cited words, establishing categories, assigning categories to the responses, sorting categories, and examining categories for recurrent themes.

Results

A well-educated workforce is a key ingredient to becoming and remaining competitive in the globalized, knowledge-based economy. Higher skills increase productivity, enhance the returns on capital investment, and increase adaptability to new markets and fuel innovations. These economic performance indicators have been the driving forces of several policy responses to achieve a world-class workforce. Results of the literature review revealed that after years of neglect, there is now a renewed interest in Technical and Vocational Education and Training (TVET). This regained interest for TVET is motivated by the growing need for highly skilled workers to cope with the dramatic economic changes and technological developments. TVET is now perceived as an important instrument for developing the skilled workforce that all nations need to facilitate economic development, national prosperity, social progress and environmental protection. The waves of reforms that have taken countries by storm have focused on strengthening the link between training and the labour market. Australia has initiated the most ambitious and successful TVET reforms in recent years. Over 100 countries have implemented, or are in the process of implementing, their National Qualification Framework (NQF) in an effort to improve the relevance and flexibility of TVET; to facilitate labour mobility and portability of qualifications across sectors, regions and countries; and to promote the recognition and validation of informal and non-formal learning. These trends were captured in a recommendation of the recent UNESCO Third International Congress on TVET: "Technical and vocational education and training: should support flexible pathways and the accumulation, recognition and transfer of individual learning through transparent, well-articulated outcome-based qualifications systems; reliable measures for assessment, recognition and validation of qualifications, including at the international level; exchange of information and development of trust; and partnerships among all relevant stakeholders. Quality assurance mechanisms should be integrated into all parts of the qualifications system" (UNESCO, 2012, p. 3). The development of occupational standards has been the cornerstone of these reform efforts. Most countries are now norm referencing and benchmarking their occupational standards to those of other countries, in an effort to ensure the development of a competitive and world-class workforce.

It is noteworthy that because of the paucity of research in TVET, there is also a dearth of evidence-based information in TVET for making important policy decisions. The conceptualization of the architecture of occupational standards and the processes used for their development are primarily driven by experts' opinion, heuristics, and conventional wisdom. No evidence-based information was found to support that some particular structural features, or components of an occupational standard can contribute to meet any of the current and future needs of the Red Seal program. Similarly, no single evidence-based information was found to support that some distinctive features of occupational standard development methodologies can contribute to meet any of the current and future needs of the Red Seal program. Only one experimental research study assessing the general effectiveness of four standard methodologies was found. An empirical study conducted in Taiwan to assess occupational standard development methodologies (Functional Job Analysis, DACUM, Position Analysis Questionnaire, and CODAP), revealed that Functional Analysis was more suitable than others however researchers found considerable variations between standards developed through different methodologies (Tien, Chou, & Ven, 2001).

Best practices in the medical profession are derived from repeated peer-reviewed evidence collected from controlled experimental research. Given the absence of this research-based evidence in the field of occupational standard, best practices used to answer the research questions for this study were exclusively based on experts' opinions gathered from the review of literature and from the key informants' consultation. Similarly, the selection of exemplary standards for review was based on the international reputation and leadership role being played by the sponsoring agencies. Therefore, the paradigm of inquiry used for conducting this study was emergent from a connoisseurship model, resting on the fundamental assumption that expert practitioners spend considerable amount time reflecting in action while resolving problems in the workplace. This reflection and hypothesis testing of events help them to develop a set of useful practical knowledge, which provides credibility to their opinions.

The report of the literature review, the study of exemplary occupational standards and the consultation of national and international key informants can be found in later sections of this report. Following is a summary of the key findings emerging from the analysis of the study data:

Key Findings

  • Skills development is an important priority on most government agendas. Occupational standards has become an important policy tool to support the skill development and qualifications recognition, especially in the context of reforms, leading to the implementation of national qualification frameworks, that have taken nations by storm. The role of occupational standards is to establish the link between labour market needs, training, assessment and certification.
  • Pressure to reform the Technical and Vocational Education and Training (TVET) systems to gain a competitive advantage in a global economy has forced many nations to borrow skills development policies from other countries as a quick fix. Countries with more experience in TVET reform have also been actively engaged in "lending" their policies to other countries. There is a growing concern regarding policy borrowing and policy lending because: "they are not necessarily useful in satisfying the policy aspirations for which they were adopted in the first place" (Chakroun B. , 2010, p. 199). In addition to the possible shortcomings of policy borrowing and policy lending there are also some critical issues regarding the soundness and effectiveness of the policies being borrowed or loaned. WDM project staff conducted a follow up inquiry with the research coordinator of an ILO study investigating the implementation and impact of national qualification framework in 16 countries, regarding the format and development processes being used by these countries for developing their occupational standards. The comments were most revealing and provide some cautions regarding policy borrowing or lending:

    "The formats were fairly similar across countries, and I would not say there were not any which were extremely satisfactory. In fact, in many countries there were constant attempts to change the format as the existing one was criticised. The processes were even more problematic. Particularly, involvement of industry was very limited in most cases. In many instances both education institutions and employers were unhappy with both formats and processes, and both saw it as an imposition of the other party. Probably the best example in terms of both is Australia, but you are probably aware that the competency-based training system has been highly criticised and reviewed, and there is a lot of unhappiness even there. A further problem in many countries where standards were designed but never used" (Allais, 2012).

  • The ILO report indicated that there is a growing interest for developing occupational standards and national qualification frameworks. Several countries have embarked on the bandwagon and the outcome appeared to be disappointing, as evidenced by the conclusion of the ILO report:

    Expectations that qualifications frameworks can achieve the ambitious policy objectives claimed for them in relatively limited time periods seem to be ill-founded. This research found little evidence that NQFs are achieving their goals....some specific evidence of qualifications frameworks having failed to achieve their goals was found (International Labour Organization, 2010b, p. 2).

  • Workers' competence is becoming increasingly complex because of the changing workplaces. Occupational standards have become a policy tool to link skills development to labour market requirements. Consequently, occupational standards is becoming more detailed and complex in order to meet emerging needs;
  • There is an increased interest for developing international and regional occupational standard to facilitate foreign credential recognition and labour mobility;
  • Occupational standards are used as a basis for making important personnel decisions such as employment selection, training, testing and certification. Occupational standards must therefore be legally defensible and must be free from bias and discrimination against women, people with disability or minority groups. Various frameworks can be used to ensure the validity of occupational standards. Accreditation of the Red Seal program by a standard setting organization such as the Standard Council of Canada can ensure product quality and validity;
  • Changing labour market requirements are calling for workers with non-traditional skill profiles, who have the versatility to perform job functions from multiple occupations. The policy response to this need has been to track generic skills and occupational skills that are broadly transferable across trades. This policy has direct implications for the flexibility and adaptability of the workforce. There is also the potential to achieve an economy of scale by consolidating resources and activities related to training and assessment. There is a renewed interest in the transferability of skills pioneered by Employment and Immigration Canada as early as 1979. The concept has been refined by the International Labour Organization and promoted as the Module of Employable Skills. Australia, the Asia Pacific region, the United Kingdom, and the United States (SkillsNET) are now coding the units of their occupational standard to facilitate the traceability of transferable skills across occupations;
  • Canada has an unacceptably high rate of workplace fatalities. Identifying hazards in the workplace and using that information for developing safety training, supported by performance feedback, can significantly reduce workplace injuries. A rigorous and systematic procedure for assessing hazards associated with Red Seal occupations is lacking and should constitute an integral component of the occupational standard structure and development methodology;
  • Reversing environmental degradation has become a global priority. Major efforts are being undertaken by several countries for greening the economy by reducing the use of fossil fuels, decreasing pollution and greenhouse gas emissions, increasing the efficiency of energy usage, recycling materials, and developing and adopting renewable sources of energy. The greening of the economy implies changing skills demands and needs. Both UNESCO and the ILO have called for the greening of TVET to facilitate the development of competences that support the goals of sustainable development;
  • Occupational standard structures and development methodologies in current use have no scientific foundations. They have rather been developed using heuristics and conventional wisdom. If the structures and processes used for developing occupational standards are weak, the products are less than perfect. Not surprisingly, even countries with long-established traditions in developing occupational standards, such as the United Kingdom and Australia, are constantly reviewing and revising both the structure and processes employed for developing their standards.
  • There is also no one best occupational standard structure and development process. As one key informant puts it: "from my point of view, beyond technical matters, occupational standards are also social construct, that are shaped by national context, stakeholders' vision and understanding, tensions, powers, etc. The best structure and development process is the fit for purpose; one that helps respond to the needs of stakeholders". However, research indicated that it is difficult for key stakeholders to agree on an appropriate standard structure and development process. As a result, in many countries standards were developed, but never used.
  • Research indicates that the emphasis of standards can be on occupations, training, assessment or a combination of these components. Results of the analysis indicated that while the major emphasis of the standards reviewed is on occupational requirements, nearly 50 per cent also specify training and assessment standards. Although training and assessment specifications are not part of some standards, all encapsulate some critical elements that are extremely useful for generating training programs and assessment instruments. The analysis revealed that the great majority of the standards reviewed support exam development, identify critical elements to be assessed, and support on-the-job and in-school training. A substantial number of these standards also allow for the use of a range of assessment options, and support the assessment of informal and non-formal learning. Only four of the standards reviewed (Australia, Asia Pacific Region, SkillsNET, and UK) support transferability of skills across trades. Two of the exemplary standards (Australia and SkillsNET) also support the assessment of multiple competencies simultaneously. It is noteworthy that several of the standards reviewed were in compliance with all applicable legal frameworks.
  • There is a major wave to reform skills development from an input to an output emphasis. Occupational standards facilitate the shift from an input to an output based training;
  • Comparative reviews of national occupational standards indicated that there are more similarities than differences among occupational standards structure and development methodology across countries, principally because of the increased trend for policy learning and policy borrowing among nations;
  • National occupational analysis documents such as the NOA are being phased out, and replaced by national occupational standards. This trend is primarily due to the need for more valid and reliable information to facilitate workers' training, assessment and certification;
  • It is generally agreed that an occupational standard has three facets of competence: a technical content, a social context and a person facet. Most occupational standards focus on the analysis of technical content with complete disregard for the social and person facets (Stewart & Hamlin, 1992);
  • While the assessment of workplace performance involved three critical factors, namely
    1. Quantity;
    2. Quality and
    3. Methods,
    occupational standards tend to focus exclusively on methods, leaving exam developers with little or no information for building the quantity and quality factors in assessment tools;.
  • There is an emerging interest in the progressive assessment and certification of individual key competencies;
  • There is an international trend for using a common template for developing national occupational standards. There is also a consensus emerging regarding the key elements that must be included in a national occupational standard to support training and assessment of formal, informal and non-formal learning. These elements are:
    • Unit Standard: A basic building block of a standard describing a discrete area of work performed by a competent worker that is used for the assessment of competence.
    • Performance Criteria: Performance Criteria are statements that describe in detail the skills applied when achieving the work outcomes outlined in a Skill Standard. They also specify in measurable terms how well the work should be performed in order to be called "competent" in the trade. A Skill Standard will have a number of Performance Criteria.
    • Evidence of Attainment: Evidence describes work outcomes in terms of work products and/or services provided. It is used for assessing a worker's competence against agreed standards.
    • Required Knowledge: Required Knowledge describes the knowledge that a competent worker must possess to be able to achieve the work outcomes to the standards required. Knowledge requirements are based on facts, principles, rationale and methods, and are focused on the "whats", "whys" and "hows".
    • Range Variables: The range variables provide useful information for assessing a worker's competence to achieve the work outcomes in the normal range of contexts they are likely to encounter on the job. They include rich contextual information about workplace requirements, the work environment, the resources used and the working conditions.
  • The NOA, as the name suggests, is an occupational analysis and does not meet the criteria of an occupational standard by world standards. Some key components need to be integrated into the NOA to transform it from an occupational analysis to an occupational standard. The OPS, which was patterned after the Australian Training Packages, encapsulates all the essential features of a modern occupational standard;
  • Canada is perceived as a world leader in the field of occupational standards. DACUM is a made in Canada standard development methodology that is used all over the world;
  • DACUM, developed in Canada, and Functional Analysis, developed in the UK, are the two most widely used methodologies for developing occupational standards. Both have strengths and weaknesses. The strength of FA resides in the conceptual lens it provides for deconstructing the functions performed by workers and linking them to work specific outcomes. DACUM provides a rigorous and systematic approach for capturing and analyzing occupational skills and knowledge. International experts' input suggested that a hybrid approach combining the most desirable features of FA and DACUM would yield optimum results;
  • To compensate for some inherent weaknesses in occupational development methodologies, the Australian Training Packages and the ILO Regional Model Competency Standards employed a combination of approaches in developing standards in an effort to improve the outcomes and products of quality;
  • According to the review of best practices, standard development time can vary between 3 to 18 months. The SkillsNET methodology, with a history of more than 20 years of service, has the shortest development time (3 months). There is no evidence indicating whether there is a relationship between standard development time and the quality and validity of the standard;
  • There is a trend for importing and tailoring skills standards from other occupations, within and between countries;
  • Several key actors played an active role in the development of the standards reviewed. The groups most commonly involved included
    1. employers/industry representatives/human resource managers;
    2. workers;
    3. union representatives; and
    4. government agencies/bodies.
    There is an increasing convergence of beliefs among industry and policy makers that occupational standards should not only meet the current skill requirements, but should also focus on future skill needs. This evolutionary perspective promotes employers' participation in the standard development process and deemphasizes the importance of workers' contributions to the process. This tendency is fueled by the assumption that workers can only identify skills that they are currently using at the workplace, and that employers on the other hand have a better perspective on future skills requirements.
  • Workers' knowledge and skills have a diminishing shelf life due to the ongoing technological and socio-technical changes at the workplace. Occupational standards must be updated regularly to prevent skills obsolescence. In the past, occupational standards were updated every five years. Some countries (Australia and the UK) with exemplary standards are adopting an updating model that is focused on continuous improvement;
  • Harsh economic realities have raised awareness that current approaches for developing occupational standards are expensive and protracted and must be replaced by more sustainable strategies. Several key actors involved in standards development are turning to digital technology not only to reduce costs of development but also to improve quality and widen the range and number of key stakeholders who provide input to the process;
  • Research indicated that the emphasis of standards can be on occupations, training, assessment or a combination of these components. The European Union International Standard Development Model includes
    1. the employment specification (employment standard);
    2. the learning specifications (learning standard); and
    3. the assessment specifications (assessment standard).
    With the application of digital technology the users are empowered to customize the content of standards to include only information that is pertinent to their own needs;
  • An appropriate structure and a rigorous development methodology pave the way for the development of sound standards. However, the competence of the facilitator and the experience of the experts participating in the development of the standards are critical success factors to produce standards of high quality;
  • The strongest occupational standard structure, with a sound and rigorous development methodology, cannot ensure the quality, the validity and the integrity of the content of a standard. The consensus which is built around the validation of a standard is the defining outcome which provides the ultimate quality control and quality assurance mechanism;
  • The best occupational standards having the strongest structure with a sound and rigorous development process cannot ensure an equitable and fair assessment. Equitable and fair assessment will also depend on skills and expertise of practitioners for developing and administering competency exams;
  • There are no metrics for measuring good occupational standards. A connoisseur can make a qualitative judgement on the quality of a standard based on some key indicators; and
  • There are hundreds, perhaps thousands, of users who struggle and become frustrated in applying occupational standards for development purposes. A developmental testing of the prototype standard, involving expert and user verification and revision can improve the usability of the document.

Extent to which the NOA and the OPS meet the current and future needs of the Red Seal program

The review of literature, examination of exemplary occupational standards, and consultation with key informants resulted in the identification of a number of critical elements that are commonly included in occupational standards that have the potential to meet the current and future needs of the Red Seal program. This section of the report provides an analysis of the extent to which these elements are encapsulated in the NOA and the OPS. It is important to note that the analysis is based on published evidence, insights gained from the research findings, and the expertise of the research team. There were no empirical studies or formal evaluations of the NOA and the OPS from which to draw evidence to support the analysis.

The NOA, as the name suggests, is an occupational analysis document that has been adapted over the years to facilitate the development of multiple-choice exams for the Red Seal program. Our analysis reveals that the NOA supports nine of the identified needs partially, and fails to support four others. Our analysis reveals that the OPS, which was modelled after the Australian Training Packages, includes all the essential elements of an exemplary standard, fully supports two of the identified needs, and to various extents partially supports ten of the identified needs. The analysis of the extent to which the NOA and the OPS meet the current and future needs of the Red Seal program was a difficult task to undertake because several of these needs are interrelated and interdependent. Additionally, in most cases, the results of the analysis are not categorically absolute. They are spanned on a degree continuum and the results were influenced by the conceptual lens used in the analysis. The following analyses were used to assess the extent to which the NOA and the OPS meet the current and future needs of the Red Seal program.

Need: Be credible within industry

The jurisdictions used an informal process to ensure that there were no trade infringements in the NOA. The consultation with key informants indicated that a more formal peer review process should be implemented to eliminate trade infringements.

The review of best practices indicated that the development of occupational standards should be governed by a set of guiding principles.

The NOA is validated by each jurisdiction using a purposive sample of experts while the OPS is validated using a focus group of job incumbents. (ITA, 2011a). The report of the Operational Review of the Red Seal Development Process recommended that the NOA be validated through a job analysis survey, using a nationally representative sample of job incumbents (Atlantic evaluation and research consultants, 2007). This recommendation is supported by evidence gathered through the literature review. Formal validation of the occupational standards not only enhances their industry credibility, but also ensures the defensibility of their validity for exam development purposes.

Need: Reflect workplace realities

The NOA is updated at fixed periods or in response to industry requests; the OPS would likely be updated with the same frequency as the NOA. The review of exemplary standards revealed an emerging trend that calls for a constant monitoring of changes in skill requirements and the continual updating of occupational standards to ensure they reflect workplace realities.

Anecdotal evidence suggested that there are some informal attempts at reviewing the occupational standards of other countries to inform the development of the NOA. National and international best practices suggested that more systematic norm referencing and benchmarking processes are necessary for developing standards that can inform the training and assessment of a world-class and competitive workforce.

The review of best practices and the consultation with key informants indicated that it is necessary to involve employers and human resource managers in occupational standard development workshops. For the most part, participants in NOA and OPS standard development workshops currently consist of expert workers.

The review of best practices indicated that occupational standard development workshops should have a higher employer to employee ratio.

Jurisdictional consultations are conducted to determine common and non-common core skills (CCDA, 2011). The consultation with key informants indicated that the common/non-common core concept should be substituted with a "mandatory" and "elective' designation. The OPS designates skills as "mandatory" and "elective".

Human competence is conceptualized in a framework that includes three essential elements:

  1. skills;
  2. knowledge; and
  3. attitudes.

The NOA and OPS addresses only the skill and knowledge components. Employers and technical instructors have to determine the attitude requirements. The absence of standard requirements for attitudes results in a lack of consistency in expectations.

Need: Represent industry needs

The NOA and OPS are developed and validated using input from expert workers. The NOA is recognized by industry and has become the flagship of the Red Seal program. Over the years, the CCDA has implemented several improvements to the NOA development process to ensure that the NOA accurately reflects industry needs. No formal survey of industry perceptions of the NOA or OPS was found.

The review of best practices and the consultation with key informants indicated that employers and human resource managers must be involved in occupational standard development workshops. While expert workers can describe their current skills, employers and managers can describe both current and future skill requirements. There is a need to involve employers more formally in the NOA development process.

Jurisdictional representatives provide intellectual and practical guidance for the development of the NOA. National and international best practices suggest that occupational standard development should be conducted under the guidance of a steering committee composed of key stakeholders.

Need: Allow the identification of critical skills to be assessed

The NOA does not include all the elements needed for developing valid assessment instruments. The review of exemplary standards and the consultation with key informants indicated that the NOA could provide more detail regarding critical skills to be assessed. The following issue needs to be addressed:

  • The specification of performance criteria, range variables and evidence of attainment can greatly support assessment.

On the other hand, the OPS encapsulates all the essential elements required for developing assessment instruments. However, for both the NOA and OPS, the following issues and concerns need to be addressed:

  • Unlike in other nations, the Red Seal program does not have a competency framework that identifies all the critical elements that must be assessed;
  • Taxonomy level of competencies is not specified;
  • Lack of a systematic process for conducting hazard analysis; and
  • Workers' attitude requirements are not systematically analyzed.
Need: Provide a basis for on-the-job training and in-school training

The best practices reviewed suggested that occupational standards should specify the skills apprentices need to achieve the work outcomes. This information is used to determine on-the-job-training.

The standard should also specify the knowledge apprentices need to achieve the work outcome. This information constitutes in-school training.

The consultation with key informants indicated that there is a lack of consistency regarding on-the-job training of apprentices. In Germany, this issue is addressed by implementing a standard that includes elements of occupational standards, training standards and assessment standards.

Curriculum developers and exam developers complained that occupational standards do not contain the critical information needed for developing curriculum and assessment tools. Lessons learned and best practices gleaned from the ITA field research conducted in Australia, New Zealand and South Africa suggested that it is highly desirable to develop competency standards and assessment and moderation systems and materials simultaneously.

NOA: The NOA is used for developing the Interprovincial Guide and Curriculum for apprenticeship training. It is also used for determining on-the-job training requirements. Although the NOA can be used as a basis for developing on-the-job training and in-school training, the inferences that must be made in the process can compromise the validity of the decision-making process.

OPS: The OPS indicates that Unit Competencies "focus on what is expected of a competent individual in the workplace as an outcome of learning [...] and they do not describe the procedures necessary to perform a particular role" (Canada, no date, p. 11). This raises the question of whether the OPS is an occupational standard or simply a document crafted to facilitate assessment (i.e. an assessment standard). Besides this concern, the OPS constitutes a rich source of information for developing on-the-job training and in-school training.

Learning hierarchy and taxonomy of psychomotor skills:

The NOA and OPS provide critical information that can be used as a basis for developing on-the-job training and in-school training. The value of that information for curriculum development and training could be enhanced if the knowledge requirements were classified according to Gagne's learning hierarchy (Gagne, 1985). Bloom's Taxonomy is very often inappropriately used for developing training.

Hazard analysis:

NOA: The development of the NOA does not include a systematic process for identifying training and assessment requirements in the area of occupational health and safety.

OPS: Although the OPS makes several references to occupational health and safety, there is no evidence that the methodology used for developing the OPS includes a systematic process for identifying training and assessment requirements in the area of occupational health and safety.

This condition is problematic given Canada's high rate of workplace fatalities among OECD countries, and considering that knowledge of occupational health and safety is not assessed by the Red Seal exam.

Hazard analysis information can be used to establish safety standards and to develop training and assessment tools.

Need: Clearly specify expected performance (transparency)

The consultation with key informants revealed that there is a lack of clarity in specifying expected performance in the NOA.

The CCDA has a well-developed and detailed manual for developing the Red Seal occupational standards: Guide for Developing National Occupational Analyses (CCDA, 2011)). However, this guide does not provide an indication of the fundamental principles guiding the development of the NOA.

The Alliance of Sector Councils has developed 11 guiding principles to frame the planning, development, validation, implementation and maintenance of occupational standards. These principles are:

  • Accessible, equitable and fair;
  • Current, relevant and valid;
  • Coherence and rigour;
  • Confidentiality;
  • Consensus;
  • Harmonization;
  • Impartiality and independence;
  • Openness and transparency;
  • Representative;
  • Sustainability; and
  • Voluntary. (The Alliance of Sector Councils, no date)

These guiding principles can be applied to the planning, development, validation and updating of the Red Seal standard to ensure its transparency in terms of expected performance.

The case study of exemplary standards revealed that these standards are developed using a template to ensure transparency and consistency in terms of expected performance.

The development of occupational standards is more an art than a science. All the processes used are based on expert opinions, heuristics and conventional wisdom. It is therefore difficult to determine if the standard is transparent in terms of expected performance. One of the suggestions from the consultation with key informants is for the standard to be verified and revised by users in order to ensure its transparency in terms of expected performance.

The NOA is validated by each jurisdiction using a purposive sample of experts and the OPS is validated using a focus group of job incumbents. (ITA, 2011a). Some countries seek input from the general public as part of the process for validating occupational standards. This additional step contributes to increasing the credibility of occupational standards.

Need: Support development of multiple choice exams

NOA: As the name suggests, the NOA is in essence an occupational analysis document. It is the flagship for the Red Seal program and its main use is for developing multiple choice exams for the Red Seal examination. It is also used for developing training.

OPS: The OPS was modelled after the Australian Training Packages, which was designed to support various forms of assessment, including multiple-choice exams. Therefore, by design, the OPS includes the essential components to meet the needs.

Desirable features:

Several key elements of occupational standards that facilitate exam development have been identified through the review of literature, the analysis of exemplary occupational standards and the consultation with key informants. The NOA encapsulates some key elements that are necessary for developing multiple-choice exams such as blocks, tasks, sub-tasks, key competencies, knowledge and ability statements, tools and equipment, context, and related components. However, as demonstrated in our review of exemplary occupational standards, by world standards the NOA has some information gaps, described below, that can compromise the transparency and validity of examinations developed using the document.

The NOA's blocks, tasks and sub-tasks are occupational analysis categories that are of critical importance for test development. They help test developers to generate test blueprints so that exam questions are appropriately distributed.

A review of international best practices failed to show other instances where the term "Block" was used in an occupational analysis or occupational standards. Use of the term block appears to have a direct impact on the richness of the description of the competency. See for example BLOCK D from the NOA for the Heavy Duty Equipment Technician trade (2009), p. III: "DRIVETRAIN SYSTEM". This label describes a block rather than a competency.

In occupational analysis jargon the NOA Block is the equivalent of a job duty. Several other agencies use the term "main function" to describe the same unit of work. The CTHRC uses "key competency" instead of block to describe the same unit of work in an attempt to make its standard more meaningful to its key stakeholders. The CTHRC also uses the terms "skill" and "sub-skills" instead of tasks and sub-tasks for the same reason.

The OPS, which, as mentioned above, was modelled after the Australian Training Packages, includes "units of competency". The OPS defines these components as follows:

Unit of competency: "An aspect of work in a particular occupation or industry that is used as a benchmark for assessment of competence" (Canada, no date, p. 5).

Elements: "Demonstrable and assessable tasks that make up the unit of competency. They describe actions or outcomes and the significant skills that are required to perform the task" (Canada, no date, p. 3).

The review of exemplary standards revealed that it is good practice to include units of competency and elements in occupational standards. Although the term "unit standard" is more commonly used, research conducted by the Industry Training Authority cautioned against the use of this term because it was the cause of considerable confusion among key stakeholders. The term "unit" is commonly used in the modularization of workers' competencies and is associated with training (ITA, 2008). Not surprisingly, the OPS indicates that units of competency "focus on what is expected of a competent individual in the workplace as an outcome of learning [...] and they do not describe the procedures necessary to perform a particular role" (Canada, no date, p. 11). This raises the question of whether the OPS is an occupational standard or simply a document crafted to facilitate assessment (i.e. an assessment standard).

The level of granularity of the elements of the NOS may be appropriate for an assessment standard, but is too detailed for an occupational standard. See, for example, Elements for Unit standard B4C: 1. Prepare for task; 4. Complete task. The level of breakdown is suitable for a time and motion study and not for an occupational analysis.

Performance Criteria:

The newer version of the NOAs, such as the NOA for the Heavy Duty Equipment Technician trade (2009), includes an element labelled "key competency", which is described as "activities that a person should be able to do in order to be able to be called 'competent' in the trade". By definition, the main focus of this element is on performance only and no reference is made to conditions under which the worker must perform and the minimum level of acceptable performance.

A review of some key competencies from the Heavy Duty Equipment Technician NOA (2009) revealed little consistency in the statements. For instance:

A-1.01.01 - clean and lubricate tools and equipment. (Does not include conditions under which worker must perform and standard of performance.)

A-2.02.06 - torque fasteners to manufacturers' specified torque rating. (Does not include conditions under which worker must perform.)

A-2.02.12 - install gaskets according to manufacturers' specifications in order to ensure tight seal and prevent damage to the gasket. (Includes all elements.)

To develop valid written exam questions, exam developers need clear and precise information on the nature of the performance, the conditions under which the worker must perform and the minimum level of acceptable performance. The review of exemplary standards revealed that it is good practice to include performance criteria in occupational standards. Since the NOA does not include the full performance criteria, this information gap increases the level of guesswork for test developers. Guessing may compromise the validity of test questions.

If the purpose of the key competency is to specify the performance criteria, it must be redefined and relabelled. Additionally, the specification of performance criteria is too important for test development to be hit or miss; there must be consistency in the formulation of performance criteria.

The OPS includes performance criteria. These are defined as "evaluative statements that specify what is to be assessed and the required level of performance" (Canada, no date, p. 5). Although the OPS does not include in the definition of performance criteria the conditions under which the worker must perform, several of the performance criteria reviewed in the OPS for the Heavy Duty Equipment Technician trade do describe these conditions.

Required Knowledge:

It appears that the newer NOAs, such as the NOA for the Heavy Duty Equipment Technician (2009), have been modeled on the United Kingdom format in that they specify required knowledge at the unit/task level. Older versions of the NOA specified required knowledge at the sub-task level. The specification of required knowledge at the task level raises several issues and concerns:

  • Given that the Red Seal exam questions are developed at the sub-task level, required knowledge specified at the task level puts considerable pressure on test developers to transpose knowledge requirements from a task to a sub-task;
  • Knowledge specification for test development at the sub-task level requires a finer level of granularity than at the task level; and
  • Specifying knowledge requirements in a block format at the task level, with no connection to the sub-tasks, increases guesswork and the likelihood of errors in determining knowledge to be tested. This may compromise the validity of test questions.

If a test taker or challenger should question the validity of an exam question, it would be difficult for the CCDA to link a test item back to a specific sub-task.

The OPS specifies the "knowledge to be assessed" rather than the critical knowledge required to support efficient and effective performance. Additionally, the knowledge statements are not linked to the elements or performance criteria.

This information gap increases the level of guesswork for test developers, which may compromise the validity of test questions.

Need: Support multiple assessment strategies to assess performance

The development of occupational standards is more of an art than a science. All the processes used are based on expert opinions, heuristics and conventional wisdom. It is therefore difficult to determine if the standard will ensure a level of consistency regardless of method of assessment without conducting a proof of concept. This is especially important in the context of informal and non-formal learning assessments.

However, the review of best practices suggested that a greater level of consistency can be achieved by integrating the following elements in the standard: performance criteria, knowledge and skills, range, evidence, and taxonomy levels, as well as level of criticality, difficulty, frequency of use, etc.

NOA: The NOA and the OPS are not subjected to any developmental testing procedures to assess their usability. The review of best practices and the consultation with key informants indicated that the occupational standard should be user verified and revised during its developmental stages. Research conducted by Chinien indicated that material verified and revised from the input of only a single user is significantly more effective that the unrevised version of the material (Chinien, 1990).

As an occupational analysis the NOA does not fully support assessment. Several inferences must be made in using the NOA for developing assessment tools. The review of exemplary standards indicated that additional components can be added to the NOA to ensure a level of consistency in the appraisal of competencies regardless of method of assessment. These components include performance criteria, required knowledge, range, evidence and taxonomy levels, as well as level of criticality, difficulty, frequency of use, etc.

OPS: The OPS was developed for assessment purposes. More specifically, the OPS was developed to facilitate the use of a "variety of forms of assessment beyond the multiple-choice exam to evaluate whether a challenger can demonstrate the skills that meet workplace standards set by industry" (Canada, no date, p. 3).

The review of exemplary standards indicated that OPS encapsulates the most critical components for ensuring a level of consistency in the appraisal of competencies regardless of method of assessment. Specifying the taxonomy levels of the competencies would further improve the OPS in this regard.

If several individuals were to use an occupational standard for developing assessment tools, there must be a level of internal consistency among the tools, regardless of the method of assessment. A lack of specificity in the formulation of occupational standards can result in considerable variations in the interpretation of competency requirements for assessment purposes. This variance can be magnified when using various methods of assessment.

Need: Support alternative forms of assessment

The NOA is specifically designed to support the development of multiple-choice examinations while on the other hand, the OPS was designed to support alternative forms of assessment. The increasing demands for assessing and recognizing knowledge and skills acquired through informal and non-formal learning call for occupational standards that support alternative forms of assessment. Skill standards must be formulated to support alternative forms of assessment.

The specification of performance criteria, range variables and evidence of attainment can significantly increase an occupational standard's ability to support alternative forms of assessment. These elements are already embedded in the OPS, but not in the NOA.

Evidence of Attainment:

While developing test questions, exam developers must know the nature of performance requirements, the conditions under which the worker must perform, and the minimum level of acceptable performance. They also need to know what evidence is required to demonstrate competence against agreed standards. The review of exemplary standards revealed that it is good practice to include Evidence of Attainment in occupational standards. The Evidence element can facilitate the development of multiple forms of assessment of key skills.

NOA: Since the NOA does not include assessment criteria/evidence, this information gap increases the level of guesswork for test developers, which may compromise the validity of test questions.

OPS: The OPS includes the Evidence element, which is described as "information that is used to demonstrate competence against agreed standards" (Canada, no date, p. 5). However, the evidence statements are not linked to the Elements or Performance Criteria. This information gap increases the level of guesswork for test developers, which may compromise the validity of test questions.

Range Variables:

NOA: Related components are described as products, items, materials and other elements relevant to the Block. The NOA lists the related components at the Block level, and they are not related to any specific sub-task. This condition increases the level of guesswork for test developers trying to associate a related component to a sub-task. Guessing may compromise the validity of test questions.

The NOA also includes a Context element, which is described as "information to clarify the intent and meaning of tasks" (HRSDC, no date, p. VII).

OPS: The OPS includes Range Variables. A range variable is described as "part of a unit of competency that sets out a range of contexts in which performance can take place" (Canada, no date, p. 5). The Range Variables are associated with both the Elements and Performance Criteria.

The NOA and the OPS do not include the taxonomy level of required knowledge, which is of critical importance for examination development. The main drawbacks of this information gap are:

  • The information is not available to users of the standard;
  • Since the information is not included in the standard, it is not subject to the same validation rigor and process; and
  • It is difficult for an outsider to determine if an examination developed from the standard was fair.

The NOA and the OPS do not include the taxonomy level of psychomotor skills. None of the exemplary occupational standards reviewed specified the taxonomy level of psychomotor skills. It does not make sense to devote a considerable amount of effort to determine the taxonomy levels for required knowledge and not apply the same effort to psychomotor skills, which constitute a higher proportion of job content for most trades.

The NOA and the OPS do not include a worker's level of autonomy. Performance expectation is strongly linked to level of autonomy. Knowledge of a worker's level of autonomy is useful for developing performance assessment tools.

Although the NOA and OPS have a well-developed and rigorous development methodology, there is no indication that the methodologies adhere to a specific competency assessment standard and guidelines. This condition can compromise the validity of assessment tools developed based on the NOA and OPS.

Occupational standards should be developed according to the guiding principles of applicable competency assessment standards (CTHRC, 2011).

Accreditation by the Standards Council of Canada (SCC), specifically the Personnel Certification Body Accreditation Program (Standards Council of Canada, 2011, p. 8), ensures conformity with all applicable requirements for developing standards and assessment tools.

Testing Emphasis:

NOA: Weights are globally assigned to the NOA blocks, tasks and sub-tasks by considering the number of exam questions that should be assigned to each. This decision is not made through a systematic examination of the criticality, difficulty, importance and frequency of the performance.

OPS: There is no evidence that the OPS has a system for determining testing emphasis.

Without a systematic approach for determining testing emphasis, insufficient testing priority may be attributed to critical knowledge and too much emphasis may be given to less essential knowledge components.

The occupational standard should specify testing emphasis based on an analytical examination of the criticality, difficulty, importance and frequency of use of the element of performance (Sage, J. & Rose, G., 1985), (CTHRC, 2011), (Norton B. , 1997).

Need: Provide flexibility for adapting standards to labour market demands

NOA: The Red Seal NOA has so far focused on the specification of competencies for a single specific occupation. International best practice suggests that the formulation of skill standards should be broad enough to be used across a range of settings, but flexible enough to be useful in any specific context. This issue has important implications for the transferability of skills.

OPS: In order to facilitate transferability, the unit standards of the OPS are formulated to be broad enough to be used across a range of settings, but flexible enough to be useful in any specific context (ITA, 2011a, p. 19).

Need: Support methods of assessment that evaluate multiple competencies simultaneously

Information related to the interdependence of competencies is useful for determining competencies that should be assessed together and to determine methods of assessment that facilitate the assessment of multiple competencies simultaneously (Australian Government, 2007). The NOA and OPS do not identify the interdependence of competencies.

Identifying the interdependence of competencies that can be assessed simultaneously is a time-consuming and tedious task. The review of international best practices indicated that an information management system can facilitate the process.

Need: Support the assessment and recognition of experiential learning (non-formal and informal learning)

NOA: As an occupational analysis the NOA does not fully support assessment. Several inferences must be made in using the NOA for developing assessment tools.

OPS: The OPS was developed for assessment purposes. More specifically, the OPS was developed to facilitate the use of a "variety of forms of assessment beyond the multiple-choice exam to evaluate whether a challenger can demonstrate the skills that meet workplace standards set by industry" (Canada, no date, p. 3). Therefore, there is a claim by design definition the OPS is supposed to support Prior Learning Assessment and Recognition (PLAR).

Need: Allow the transferability of skills across occupations

NOA: The NOA has a numbering system for tasks and sub-tasks, key competencies and required knowledge. However, that trailing system does not facilitate the traceability and transferability of skills across occupations. The consultation with key informants indicated that it is difficult to track the transferability of skills with the NOA

OPS: The OPS is modeled after the Australian standard, which by design supports the transferability of skills. The coding system adopted for the unit standard can facilitate transferability.

The format used to formulate standards has important implications for the transferability of skills across occupations and sectors.

Manual processing of information to determine and manage the transferability of skills can be a tedious and extremely time-consuming task. The review of best practices and exemplary standards indicated that digital technology platforms can be successfully applied to automate the process.

Summary of the extent to which the NOA and the OPS meet the current and future needs of the Red Seal program

Needs NOA OPS Recommendation Number
Be credible within industry Partially met Partially met 4, 9, 17, 19, 22, 24, 27, 28, 32
Reflect workplace realities Partially met Partially met 9, 14, 17, 18, 19, 20, 22, 23, 26, 27, 28, 36
Represent industry needs Partially met Partially met 9, 17, 22, 27, 28, 33, 35
Allow the identification of critical skills to be assessed Partially met Partially met 1, 2, 11, 23, 34
Provide a basis for on-the-job training and in-school training Partially met Partially met 2, 11, 12, 13, 16
Clearly specify expected performance (transparency) Partially met Partially met 7, 9, 17, 21, 22, 24, 28
Support development of multiple choice exams Partially met Partially met 1
Support multiple assessment strategies to assess performance Not met Partially met 1, 2, 5, 6, 10, 11, 15, 21, 23, 25, 34
Support alternative forms of assessment Partially met Partially met 1, 2,3, 4, 10, 21, 25
Provide flexibility for adapting standards to labour market demands Not met Met 29
Support methods of assessment that evaluate multiple competencies simultaneously Not met Not met 3
Support the assessment and recognition of experiential learning (non-formal and informal learning) Partially met Met 1, 2, 5, 6, 8, 10, 11, 15, 23, 34
Allow the transferability of skills across occupations Not met Partially met 5, 9, 17, 22, 28, 30, 31, 32

Rationale Supporting the Study Recommendations

No evidence-based information was found to support that some components of an occupational standard can contribute to meet the current and future needs of the Red Seal program. Similarly, no evidence-based information was found to support that some distinctive features of occupational standard development methodologies can contribute to meet these same needs. In spite of the lack of empirical evidence related to occupational standards, information gathered from the literature review and key informants was buttressed with the experience of WDM project staff in TVET, occupational standard development, and principles of qualitative research to elaborate supporting rationales from which appropriate recommendations for a new standard structure and development methodology were drawn.

This section of the report proposes the recommendations deemed necessary for elaborating an occupational standard structure and development methodology to meet the current and future needs of the Red Seal program. It also includes an analytical discussion, which provides the rationales for justifying the recommendations. These recommendations are deemed workable, because they were driven from insights gained from current practice.

Recommendations Need addressed Rationale

Recommendation 1

That the Red Seal NOA be restructured to include skill standards. Each skill standard should include the following main components (additional components are identified in the proposed Standard Template):

  • Key Competencies: Key performances deployed to achieve a major set of work outcomes.
  • Skills: Major performances deployed to achieve a specific set of work outcomes.
  • Sub-Skills: Specific performances deployed to achieve a specific work outcome.
  • Range Variables: Provide rich background contextual information about the work environment and working conditions that is useful for training and assessment.
  • Performance Criteria: Evaluative statements that specify what is to be assessed and the required level of performance in order to be called 'competent' in the trade.
  • Required Knowledge: Knowledge that a person must acquire to perform a unit skill effectively and efficiently. Knowledge requirements are based on facts, principles/rationale and methods, and are focused on the "whats", "whys" and "hows".
  • Evidence of Attainment: Describes work outcomes in terms of work products and/or services provided. Evidence needed to demonstrate competence against agreed standards.

Support development of multiple choice exams

Support methods of assessment that evaluate multiple competencies simultaneously

Allow for the identification of critical skills to be assessed

Support multiple assessment strategies to assess performance

Support the assessment and recognition of experiential learning (non-formal and informal learning)

The great majority of the key informants from the jurisdictions indicated that it is either extremely important (71.4%) or very important (14.3%) for occupational standards to support the development of multiple-choice questions.

The review of best practices and examination of exemplary occupational standards revealed that there are some critical elements that can be included in a standard to support the development of assessment tools. These elements are:

Skills to be performed: Major performances deployed to achieve a specific set of work outcomes.

Performance criteria: When correctly formulated, performance criteria can easily be turned into test questions.

Required knowledge: Required knowledge is used for determining knowledge to be tested.

Range variables: The range variables provide useful information for assessing workers' competence in achieving the work outcomes in the normal range of contexts they are likely to encounter on the job. It is necessary to include range variables in the Red Seal standard because they provide rich contextual information about workplace requirements, the work environment, resources used, and working conditions, all of which are useful for training and assessment purposes. The range variables can also facilitate the assessment of multiple competencies simultaneously.

Evidence of attainment: This element provides the evidence needed to demonstrate competence against agreed standards.

The growing demand for assessing learning acquired informally, non-formally, and in foreign countries calls for occupational standards that support various assessment strategies and tools to evaluate workers' performance. The elements identified above also support these assessment needs.

The NOA has some elements with different labels, which, under close examination, appear to have similarities with critical elements that support assessment. The OPS encapsulates all these elements. The absence of these critical elements in an occupational standard can compromise the transparency and validity of examinations developed using the document. Recommendation 1 integrates all the key elements that support assessment in the new Red Seal standard.

Considering that most countries are adopting a standard structure that includes these key elements, Recommendation 1 will also facilitate the growing need for norm referencing and benchmarking national occupational standards to those of other nations.

Results of this study indicated that key stakeholders use occupational standards for a wide variety of applications (up to 115 Uses of National Occupational Standards in the UK, http://www.euskills.co.uk/download.php?id=1257). Consequently, if the occupational standards are to be useful to all potential users, it is extremely important to select meaningful labels to describe the component parts that are likely to resonate well among users of the standard.

Each Skill component of the occupation is used to formulate a Performance Standard. Although the term "Unit Standard" is more commonly used, research cautions against the use of this term because it causes considerable confusion among key stakeholders. The term "Unit" is commonly used in the modularization of competency and is associated with training (ITA, 2008). It is noteworthy that the Canadian Tourism Human Resource Council (CTHRC) uses the recommended descriptive labels in its occupational standards.

One disadvantage of using the label "Skill Standard" instead of Unit Standard (which, as mentioned above, is more widely used) is that it will make international comparative analysis, norm referencing and benchmarking more difficult.

The International Network of Sector Skills Organizations (INSSO) is considering using the term "Competence Standard". However, the descriptive label "Performance Standard", as used in the OPS, is also desirable.

The UK, the ILO, the European Union, and several other countries are using a common format for all NOS in an effort to facilitate the transferability of skills and the importation of standards across occupations and sectors (Roberts, 2012). A common standard format was also recommended for the new Red Seal standard.

The proposed level of granularity is adequate for all Red Seal occupations and is consistent with the structure of the NOA and OPS, and reflects the structure of exemplary standards reviewed. The structure is based on the hierarchical relationships that exist between Key Competencies, Skills, and Sub-Skills. This hierarchical structure is useful for establishing the link between labour market requirements and training for certification, developed from the standard (ILO, 2011).

Supporting information: See comparative analysis of proposed standard, OPS and NOA (Figure 17); see also (Tien, Chou, & Ven, 2001).

Recommendation 2

That the required knowledge embedded in the Red Seal standard be assigned a taxonomy level reflecting the level of knowledge expectation for a beginning journeyperson in the trade. For ease of application, the six levels of Bloom's Taxonomy can be compressed into the following four levels.

  • Level 1: Recall and understanding
  • Level 2: Application and interpretation of knowledge
  • Level 3: Analysis of information
  • Level 4: Problem-solving, synthesis and evaluation to create new knowledge

Support development of multiple choice exams

Exam questions must target the desired level of performance. Bloom's taxonomy is the most commonly used taxonomy for specifying the level of knowledge. The NOA and the OPS do not rate the knowledge requirements according to taxonomy.

In developing their occupational standards, some organizations, including the CTHRC, are using Bloom's taxonomy to formulate the knowledge requirement at the proper level by selecting the appropriate action verbs to describe exact behaviours. Eventually, these action verbs provide signposts for formulating multiple-choice questions at the desired levels.

The proposed standard specifies the level of the required knowledge according to a modified version of Bloom's Taxonomy. This taxonomy will specify the level of knowledge expectation for a journeyperson in the trade. For ease of application, the six levels of Bloom's Taxonomy can be condensed into four levels, as proposed by the CTHRC (CTHRC, 2011).

Recommendation 3

That the guiding principles of applicable standards related to competency assessment be adopted to inform the development of the Red Seal standards, in order to ensure the validity of the standards for generating assessment instruments.

Support development of multiple choice exams

Support alternative forms of assessment

Although the NOA and the OPS are produced using rigorous development methodology, there is no indication that the methodology conforms to specific competency assessment standards and guidelines. This condition can compromise the validity of assessment tools developed based on the NOA or the OPS.

Occupational and assessment standards must accurately reflect industry requirements and must also be legally defensible. The literature makes several links between the validity of occupational standards and legally acceptable test validity. Test validity is partly ensured by using validated occupational standards for generating test items. Therefore, multiple-choice questions must accurately reflect industry requirements and must be legally defensible.

There are various frameworks that can provide theoretical and practical guidance to ensure that occupational and assessment standards are valid, and that assessment instruments developed from these standards have an audit trail of that validity status, therefore meeting the test of validity.

A review of the relevant literature has shown that the Standards for Educational and Psychological Testing, published by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education (1999), have been the most widely used standards by professional associations to ascertain the validity of occupational standards in connection with credentialing and licensure.

Ensuring the validity of occupational standards and test validity were among the top priorities of the key informants consulted. Data gathered from the consultation revealed several measures that are being taken to ensure the validity of occupational standards and to ensure that they comply with all applicable legal frameworks.

Some of the most widely used legal frameworks are presented in the section Standard Validity: Legally Defensible Frameworks.

Several of the issues regarding the validity of the NOA, identified by the operational review of the Red Seal development process, can be resolved by placing the occupational standard and the Red Seal examination development and administration processes under the operational guidelines of a standard setting agency for personnel certification such as the Standards Council of Canada (SCC).

Supporting information: See section on Standard Validity: Legally Defensible Frameworks in the document; see also section on Standards Council of Canada.

Recommendation 4

That the Red Seal standards be validated by surveying a representative sample of key stakeholders. That the validation incorporate a rating of performance criteria on criticality, difficulty, importance and frequency of use. That a system be designed and developed to combine the rating of these elements into a single testing emphasis measure for each performance criteria. That the testing emphasis measures be used for assigning weight to the competency areas, the skill standards, and the sub-skills, and the weights be used to determine the number of test questions for the Red Seal exam.

Support development of multiple choice exams

At present, the NOA and the OPS are not being validated by surveying a representative sample of key stakeholders; an approach which is supported by best practices (Norton B. , 1997), (CTHRC, 2011) and recommended by the Red Seal Operational Review. This approach is routinely used by several sector councils, including the CTHRC.

Weights are globally assigned to the NOA blocks, tasks and sub-tasks by considering the number of exam questions that should be assigned to each. This decision is not supported by a systematic process. Without a systematic approach for determining testing emphasis, insufficient testing priority may be attributed to critical knowledge and too much emphasis may be given to less essential knowledge components.

The occupational standard should specify testing emphasis based on an analytical examination of the criticality, difficulty, importance and frequency of use of the element of performance (Sage & A., 1985), (CTHRC, 2011), (Norton, 1997). If measures of criticality, difficulty, importance and frequency of performance were built into the standard validation process, the output would allow the jurisdictions to first determine the testing emphasis of each block, task, and sub-task, and use the results as a defensible yardstick for assigning weights for developing test blueprints and tables of specifications. These same weights would be extremely useful for determining training emphasis for curriculum development purposes.

Supporting information: See section on Determining testing emphasis in the document and section on Key Elements Included In The Appendices, specifically Appendix E

Recommendation 5

That the Red Seal standards be integrated in an information management system, designed to facilitate the clustering of performance criteria and other key elements of the standard around complete work outcomes so that multiple competencies can be assessed simultaneously.

Support methods of assessment that evaluate multiple competencies simultaneously

Allow the transferability of skills across occupations

The proposed standard development strategy has a two-pronged approach. During the first stage, Functional Analysis and DACUM are used for developing the occupational standard. All the components of the occupational standard are uploaded to the database of an information management system. From that point in time it would not be necessary to repeat that process for that particular occupation again, because the updating of the standard can be achieved, on a continuous basis and also at specific points in time, with the assistance of one to two experts and an occupational analyst. This digital system will include multiple features specifically designed to streamline the validation of occupational standards in the most efficient, effective and economic way.

There are some apprehensions that the application of digital technology for developing occupational standards will weaken the quality of the standards. The literature review did not support this assumption. WDM has recently developed an International Standard for Food and Beverage Management with the assistance of 153 industry experts from 60 countries using digital communication tools and application software. All the experts were recruited with the assistance of social media networks. The development of the international standard was conducted under the governance of a Global Standard Advisory Committee consisting of senior government policy makers, employers, industry representatives and professional associations from 20 countries. On average, the experts volunteered approximately 15 hours over a period of 3 months to develop the standard.

There are both advantages and disadvantages associated in the application of digital technology for developing occupational standards. Following is a brief review of each:

Advantages:

  • There is an ongoing dilemma as to whether the emphasis of a standard should be on occupational competence, training or assessment. A digital system will allow CCDA to generate various Red Seal products targeted at various audiences and stakeholders from the information stored in the database of the information management system. The user will determine the desirable features to be included in the standard from an interactive menu;
  • There are growing concerns that the development of occupational standard is expensive and protracted. Going digital will contribute to reduce both the time required for developing standards and the associated costs;
  • Once a standard has been developed, complete redevelopment would not be necessary, unless the occupation has undergone sudden and dramatic changes. With the application of digital technology, the standards will be updated on a continuous basis;
  • Larger groups can participate in the development, validation and updating of standards at minimum cost;
  • Digital technology will enhance flexibility;
  • Digital technology will facilitate validation of draft standards;
  • Digital technology will facilitate development of test blueprints;
  • Digital technology will facilitate exam development, pilot testing and administration;
  • Digital technology will facilitate the assessment of multiple competencies simultaneously;
  • Digital technology will facilitate the assessment of standards;
  • In order to achieve a complete work outcome, very often tradespeople need to integrate several key competencies. The workers need to apply the necessary task management skills to be able to plan and integrate these competencies. It is highly desirable to assess these multiple performances simultaneously as the sum is greater than the parts. The difficulty encountered in trying to assess performance criteria simultaneously is to be able to determine which cluster of performance criteria is involved in a particular work outcome. Digital technology can facilitate this process;
  • The NOA and the OPS do not identify the interdependence between technical competencies. This information is useful to determine competencies that should be assessed together and to support methods of assessment that evaluate multiple competencies simultaneously (Australian Government, 2007);
  • Identifying the interdependence of competencies that can be assessed simultaneously is a time-consuming and tedious task. Review of best practices suggested that an information management system can facilitate the process (SkillsNET, 2012). An occupational standard, which facilitates the assessment of multiple competencies simultaneously, must be supported by an information management system, which enables coded performance criteria to be clustered around different work outcomes;
  • The adoption and implementation of a digital platform supporting an information management system will also facilitate the traceability and the transferability of skills across occupations and sectors. Skills can be coded to facilitate their search and retrieval and various clustering patterns.

Disadvantages:

  • Some older workers may not have the necessary level of digital skills to interact with the digital tools and associated software;
  • The facilitators need specialized social network skills (not easy to acquire) to build a virtual community, to stimulate the interest of the experts and to maintain their interest;
  • Periodic updating of the digital system is to be expected to cope with change in technology;

The advantages of going digital exceed by far the anticipated disadvantages.

Supporting information: See the section on Digitization of Standard Development Methodology and Digital Strategy

Recommendation 6

Recommendation beyond the scope of this research project

That the Red Seal training standards be developed in terms of learning outcomes to accommodate learning acquired in informal and non-formal settings. CCDA will need to develop the standard in collaboration with the jurisdictions and the key stakeholders. The standard will describe the skills that apprentices need to learn to do; and the knowledge they need to learn to be able to execute the skills efficiently and effectively.

Support alternative forms of assessment

Support the assessment and recognition of experiential learning (non-formal and informal learning)

In the past, it was common practice to specify learning in terms of input, that is, the content that is taught and what apprentices must learn. Due to the growing need to assess, validate, and recognize learning acquired informally and non-formally, there has been a shift in emphasis from learning input to learning outcome. When learning is specified in terms of learning outcomes, it can be assessed and validated irrespective of the context within which it was acquired.

This recommendation does not fit within the scope of this project.

Recommendation 7

Recommendation beyond the scope of this research project

That a Red Seal assessment standard be developed and implemented to guide the development of policies, instrumentation and practices for assessing apprentices from formal apprenticeship program. CCDA will need to develop the standard in collaboration with the jurisdictions and the key stakeholders. The standard will describe the evidence needed to demonstrate competence, what the apprentice must do and the knowledge that must be tested. Methods of assessment, how assessment instruments are produced and applied (Mansfield, B. and Schmidt, H., 2001, pp. 54, 55)

Support alternative forms of assessment

The Red Seal Operational Review (Atlantic evaluation and research consultants, 2007) has identified some weaknesses in the development of Red Seal examinations. The CCDA initiative to pioneer Multiple Assessment Pathways (MAP) for Red Seal certification raised other issues and concerns about the development and administration of various assessment tools. There is a definite need for a set of standards to guide the development of assessment policies, instrumentation, and practices.

This recommendation does not fit within the scope of this project.

Supporting information: See the Case Study for the European Union, Germany, and Australia.

Recommendation 8

Recommendation beyond the scope of this research project

That a Red Seal validation standard be developed to guide the development of policies, instrumentation and practices for the validation of informal and non-formal learning. This standard will describe the methods of validation, the qualifications of validation practitioners and the types of learning that can be validated. The validation standard can be modeled on the Occupational Performance Standard (OPS), given that it was developed for pilot testing the assessment of informal and non-formal learning. The European guidelines for validating non-formal and informal learning can be used as an example for developing the standards.

Support alternative forms of assessment

While a growing number of Canadians are acquiring a significant amount of occupational skills informally and non-formally, there is at the same time a major learning recognition gap in Canada (ITA, 2008). Skills development agencies will be under increased pressure to recognize and validate learning acquired outside of formal institutions. There is a definite need for a set of standards to guide the development of policies, instrumentation and practices for the validation of informal and non-formal learning so that the Red Seal program can avoid dealing with this issue in a piecemeal way at the provincial/territorial levels. The proposed standard will ensure consistency in the validation of learning.

This recommendation is beyond the scope of this project.

Recommendation 9

That the necessary tool and decision aids be developed to ensure an appropriate and consistent level of granularity in the Red Seal standards.

Represent industry needs

Clearly specify expected performance (transparency)

Be credible within industry

Reflect workplace realities

A sampling analysis of the NOA for the Heavy Duty Equipment Technician trade reveals that there are inconsistencies in the level of granularity of certain elements of some sub-tasks. This inconsistency manifests itself through wide variations in competency deployment and the significance of the work outcome of some sub-tasks. The following examples are illustrative of this issue:

Low competency deployment and low work output

A.3.02 Complete documentation
A.3.03 Communicate with others

High competency deployment and high work output

C.8.01. Repair hydraulic systems

A sampling analysis of the OPS for the Heavy Duty Equipment Technician trade reveals that there are inconsistencies in the level of granularity of certain elements of some sub-tasks. This inconsistency manifests itself through wide variations in competency deployment and the significance of the work outcome of some sub-tasks. The following examples are illustrative of this issue:

Low competency deployment and low work output

A1C Apply environmental skills
A.6E Sample fluids

High competency deployment and high work output
C1C Diagnose and repair hydraulic and pneumatic systems

Guide for Developing National Occupational Analyses (CCDA, 2011) does not provide any tools and decision aids to ensure an appropriate and consistent level of granularity in the Red Seal NOA or OPS analysis. The literature offers few guidelines on appropriate strategies for establishing a consistent level of granularity across the various elements of occupational standards. Yet, having a consistent and appropriate level of granularity in an occupational standard has extremely important implications for training and assessment; an inconsistent and inappropriate level of granularity may compromise the integrity of training and testing emphasis.

Field research conducted by a team of ITA staff in Australia, New Zealand and South Africa revealed some serious difficulties that can be encountered in the assessment of competencies due to occupational standards with a level of granularity that is too fine. The specific concerns reported were that "the tendency to drive competency standards down to ever greater levels of specificity had atomized learning and assessment into somewhat meaningless, disconnected "chunks" of performance unrelated to more integrated performance required in industry for success in an actual workplace. This excessive specificity [...] makes integrated assessment of meaningful workplace performance skills more difficult" (ITA, 2008, p. 39).

The Australian Training Package Development Handbook summarizes the issues and concerns regarding granularity: "It is difficult to generalize about what is an appropriate size for a unit of competency; it must be useful and manageable for the purpose of training, recognition and assessment and it must reflect the complexity of skills and knowledge or the range of activities undertaken" (Australian Government, 2007, p. 16). The document posits that the rating of the importance of the skills and the time required to master those skills are poor indicators of unit size.

Recent exploratory action research conducted by WDM staff suggested that expert workers' ratings of time spent on performing key competencies, skills, and sub-skills can constitute potentially useful information to establish a balanced level of granularity.

This proposed procedure was developed and tested by WDM-Consultants when formulating an international standard for Food and Beverage Management for the CTHRC.

Supporting information:

See the section on the Procedure for balancing key competencies, skills and sub-skills (ITA, 2008)

See the international standard for Food and Beverage Management for the CTHRC

Recommendation 10

That skill standards be formulated in such a way that they support multiple forms of assessment. This can be ensured by inviting specialists in the evaluation of formal and informal learning to attend the standard development workshops as observers so that they can provide input regarding the responsiveness of the standard being developed to the assessment of formal and informal learning

Support alternative forms of assessment

Given the growing number of Canadians who are acquiring a significant amount of occupational skills informally and non-formally (ITA, 2008), skills development agencies will be under increased pressure to validate and award certificates for learning acquired outside of formal institutions. As demonstrated by the CCDA Multiple Assessment Pathways Pilot Project, the traditional multiple-choice exam is not sufficient for assessing the competencies of candidates for Red Seal certification. Other forms of assessment strategies must also be used.

The NOA is specifically designed to support the development of multiple-choice examinations. The increasing demands for assessing and recognizing knowledge and skills acquired through informal and non-formal learning call for occupational standards that support alternative forms of assessment.

To be valid, assessment tools must be aligned to skill standards. A skill standard that supports alternative forms of assessment must have multiple indicators of performance. The specification of performance criteria, required knowledge, range variables, and evidence of attainment can significantly increase an occupational standard's ability to support alternative forms of assessment. (Australian Government, 2007) These components are already embedded in the OPS, and the recommendations to include them in the new Red Seal standard are based on the same rationale.

Recommendation 11

That the Enhanced Red Seal Initiative implement Red Seal Occupational Health and Safety Standards. The standards should be derived from a systematic hazards analysis for each skill standard in relation to environmental conditions and the use of hazardous materials, sources of energy, tools and equipment.

Provide a basis for on-the-job training and in-school training

Represent industry needs

Reflect workplace realities

Allow the identification of critical skills to be assessed

According to the ILO, an average of 6,000 workers die each day as a result of work-related accidents and diseases. This amounts to 2.2 million work-related deaths annually (International Labour Organization, n.d.). Canada has an unacceptably high rate of workplace fatalities, ranking fifth among OECD countries. Research conducted by the Centre for the Study of Living Standards (Sharpe & Hardt, 2006) revealed that the rate of fatality in Canadian workplaces between 1993 and 2005 was five deaths a day. The workplace fatality rate for the construction industry was 20.6 per 100,000 workers or one out of 4,900 over the 1996-2005 period. Results of the study also indicated that older workers were more prone to workplace fatalities than their younger counterparts (Sharpe & Hardt, 2006). The incidence of workplace-related fatalities per 100,000 workers in the 15-19 age group was 1.8, compared to 18.1 for workers in the 60-64 age group (WDM-Consultants, 2009).

The Red Seal program promotes workers' wellbeing and occupational health and safety; however, a rigorous and systematic procedure for assessing hazards associated with Red Seal occupations is lacking. This gap in analysis can compromise training and assessment of competences related to health and safety.

The development of the NOA and the OPS does not include a rigorous and systematic process for identifying the training and assessment requirements in the area of occupational health and safety. This condition is problematic given Canada's high rate of workplace fatalities among OECD countries, and considering that knowledge of occupational health and safety is not assessed by the Red Seal exam.

In a review of literature conducted on behalf of the Workers' Compensation Board of British Columbia, Lisa A. Ronald noted that "considering the staggering costs associated with occupational injuries, including medical costs, lost wage costs, production stoppage and human costs, it is crucial that organizations focus on developing successful safety programs which lead to the decrease, or ultimately, the elimination of workplace accidents'' (Ronald, 1998). The conclusion reached by this review of literature and research is that safety training which is supported by performance feedback can significantly reduce workplace injuries.

Hazard analysis information is used to develop safety training and assessment tools. Given that neither the NOA nor the OPS systematically addresses occupational health and safety, it is recommended that a systematic hazard analysis be incorporated into the Red Seal program.

Recommendation 12

That the Red Seal standards be formulated to contain elements of training and assessment specifications that can be readily used for developing training and assessment standards.

Provide a basis for on-the-job training and in-school training

Occupational standards are focused on employment specifications. They describe what workers need to do, how they will do it, and how well they do it. Occupational standards establish the link between industry and training for skills development (CEDEFOP, 2009a).

The primary focus of training standards is on what workers need to learn, how they will learn it, and how the quality and content of learning will be assessed (formative and summative evaluation). Training standards are usually formulated in terms of curriculum inputs, i.e., subject, syllabus, teaching methods, process and assessment. Training standards that are formulated in terms of inputs can compromise the validity of assessment and the use of alternative assessment approaches.

The literature review and the consultation with key informants performed in the context of this research pointed to several issues and concerns regarding the on-the-job training of apprentices:

  • Inconsistencies in the comprehensiveness, extent and quality of on-the-job training experiences of apprentices;
  • Questionable validity and trustworthiness of on-the-job training assessment and sign-off;
  • Deliberately delaying sign-off to benefit from lower wages over time; and
  • A lack of clarity and inconsistencies in specifying on-the-job training requirements.

The key informants from the Canadian jurisdictions made the following significant comments regarding desirable changes for improving the structure of the standards:

  • A more elaborate NOA that would facilitate the development of training and assessment standards;
  • The apprenticeship training standards are often inflexible towards workplaces. It is very "difficult for one (or many) employers to expose a trainee/apprentice to the entire trade in a specific period of time".

The review of best practices revealed that other countries are already including elements that facilitate the development of training and assessment standards in their occupational standards; for instance, The European Union International Standard Development Model (Mansfield, B. and Schmidt, H., 2001, p. 24), the German organizational framework and standards for apprenticeship occupations (BIBB, 2006), and the Australian Training Packages (Australian-Government, n.d., p. 2). The OPS, which was modeled on the Australian Training Packages, already includes some of these key elements, namely, performance criteria, skills, required knowledge, range variables, and evidence of attainment.

Recommendation 13

Recommendation beyond the scope of this research project

That a Red Seal on-the-job-training standard be developed to provide industry with clear guidelines and specifications for managing, assessing and reporting on on-the-job training of apprentices. CCDA will need to develop the standard in collaboration with the jurisdictions and the key stakeholders. The standard will describe the skills that apprentices need to learn on the job and how they will be assessed.

Provide a basis for on-the-job training and in-school training The same rationale discussed for Recommendation 12 applies to this Recommendation

Recommendation 14

Recommendation beyond the scope of this research project

That in establishing criteria for assessing informal and non-formal learning, consideration be given to the fact that the on-the-job training component of formal apprenticeship is far from being perfect and that several apprentices may not necessarily have the full range of exposure to all the mandatory occupational competencies of their respective trades. Competencies identified through this process will be assessed to determined their relevance to the Red Seal occupational standard.

Provide a basis for on-the-job training and in-school training

The consultation with key informants suggested that the apprenticeship training standards are often inflexible towards workplaces. It is very "difficult for one (or many) employers to expose a trainee/apprentice to the entire trade in a specific period of time". This observation has important implications for establishing standards for trade apprentices. If there is no guarantee that apprentices in the formal apprenticeship program are getting a full range of exposure to all the occupational competencies of their respective trades, it may not be fair to expect candidates to meet that expectation.

The CCDA needs to develop criteria and experience indicators that are fair and equitable for trade apprentices, which at the same time guarantee workers' competence. It can be argued that a standard will not fully address the problem, but it is definitely the first step towards its resolution.

To address this problem, some countries have adopted a policy that enables apprentices to rotate through multiple sites and employers. This strategy requires resources for implementing, managing, and monitoring the activities. Some employers might be unwilling to support a rotating apprenticeship scheme on the grounds that it disrupts continuity.

Recommendation 13 proposes the establishment of an on-the-job training standard for apprentices involved in the formal apprenticeship program.

Recommendation 15

That outputs of the validation of informal and non-formal learning be used to externally benchmark the Red Seal standards, given that candidates may have critical competencies that are not defined in the standards. If there is a justification to include these competencies in the Red Seal standard, they need to be properly validated.

Support the assessment and recognition of experiential learning (non-formal and informal learning)

There is a likelihood that the validation of the informal and non-formal learning of candidates reveals some critical competencies that are not defined in the Red Seal standards. The literature suggests that these competencies should be used to externally benchmark the Red Seal standards (CEDEFOP, 2009a).

Competencies identified through the validation of informal and non-formal learning should be investigated and validated to determine the appropriateness of including them in the Red Seal standard.

Recommendation 16

That the CCDA adopt a policy for developing occupational, training and assessment standards simultaneously. Group leaders of the training and assessment teams should attend the Red Seal occupational standard development workshop. Corrective feedback between the standard, training, and assessment leaders can be used to improve the relevance, responsiveness, and quality of the standard. The ISEC member of the host jurisdiction responsible for developing the Red Seal exam should continue to attend the standard development workshop.

Provide a basis for on-the-job training and in-school training

The key informants from the jurisdictions pointed to a lack of clarity and inconsistencies in specifying on-the-job training requirements. They also suggested that a more elaborate NOA would facilitate the development of training and assessment standards for trade apprentices.

Lessons learned and best practices gleaned from the ITA field research conducted in Australia, New Zealand, and South Africa suggested that this issue can be addressed by developing competency standards and assessment and moderation systems and materials simultaneously (ITA, 2008). Simultaneous development has been successfully implemented in various industrial fields.

The simultaneous development of occupational, training and assessment standards would enable all the parties involved to express their requirements and expectations regarding the occupational standards. Group leaders of the training and assessment teams should attend the Red Seal occupational standard development workshop as observers. Their input and corrective feedback during the standard development process can be used to improve responsiveness of the standard for developing training and assessment. Draft versions of the standard in development should be shared with all parties involved for review and feedback.

Recommendation 17

That employers, workers, regulators and union representatives be involved in the development, validation and updating of standards so that employers can provide input into the current and future skill requirements of occupations.

Represent industry needs

Clearly specify expected performance (transparency)

Be credible with industry

Reflect workplace realities

Occupational standards are focused on employment specifications. They describe what workers need to do, how they will do it, and how well they do it. Occupational standards establish the link between industry and training for skills development (CEDEFOP, 2009a). It is critical to ensure that occupational standards meet the needs of industry. Eighty-eight per cent (88%) of the key informants indicated that it is extremely important to ensure that occupational standards meet the needs of industry. Both the literature review and the consultation with key informants also indicated that occupational standards should not only meet current industry needs, but also anticipate needs in the foreseeable future. However, there is an assumption that workers can only identify skills which they are currently using in the workplace, and that by contrast employers have a better perspective on future skills requirements.

In the UK, NOS Strategy 2010 – 2020 is being implemented under the overarching framework of nine principles, with the first principle being that NOS must be articulated by a representative sample of relevant employers across the UK. A facilitator and representatives of employers (managers, employees), professional bodies, trade bodies and regulatory bodies are involved in the development of the occupational standards. Employers are also involved in the validation of occupational standards (Carroll G. , 2012).

In France, the Organismes Paritaires Collecteurs Agréés, OPCA, (an organisation grouping employers and workers in the sector, working with specialists in training and development) (Lesemann, 2005, p. 25) is responsible for the development of occupational standards, and the CPNE (Commissions paritaires nationales de l'emploi) (Political entity that groups employers and workers in a sector) (Lesemann, 2005, p. 26) is responsible for the validation of occupational standards.

In Germany, employers, trade unions and the federal states are also involved in the development and validation of occupational standards.

In the Asia-Pacific region, employer bodies and associations, representatives from typical enterprises, expert employee representatives, professional associations, regulatory or licensing bodies, and educators and trainers are involved in the development and validation of occupational standards.

In the U.S., the Industry Technical Advisory Committee (ITAC) drives the skill standards development process (guidance, advice, and technical knowledge). It is a requirement that industry have the majority voice in ITAC and that ITAC membership reflect the composition of employers within the industry/occupation by size, region, and business diversity. (TSSB, 2010a, p. 8)

The majority of respondents pointed to the need to include industry in the development process to ensure the validity of occupational standards. They also pointed to the need to ensure the participation of employers, subject matter experts, and an advisory committee.

Although the benefit of involving employers in the development of occupational standards is unquestionable, there may be some difficulties in recruiting employers who are willing and able to commit an extended period of employee time to the standard development workshop. The consultation with key informants indicated that employers are finding it difficult to release their employees for the time required to participate in the workshop. The key informants consulted also noted that the employees being released to participate in the standard development workshop are not always among the best qualified.

Some agencies are resorting to virtual workshops supported by digital technology to overcome the difficulty of finding highly qualified experts to attend a physical workshop. A digital approach has several other advantages including cost and expediency.

Recommendation 18

That various approaches be used for tracking employers' current and emerging skill needs, such as:

  • Using technology and social networks to consult with employers and to track their emerging skill needs;
  • Surveying employers via print and digital media; and
  • Tracking the performance of all graduates over a five- year period to determine their degree of satisfaction with their apprenticeship training.

Represent industry needs

Reflect workplace realities

Same rationale as Recommendation 17.

In some sectors of the economy, there is a growing need to ensure that employees possess cutting-edge skills in their occupation. This new requirement calls for innovative approaches for tracking employers' current and emerging skill needs.

The Australian Training Packages is now supported by a digital system to facilitate the tracking of input from key stakeholders. The main advantage of Australia's occupational standard structure is that the standard can be viewed online in a consistent format and structure, driven in the background by a database. This feature enhances accessibility and navigability; "The increasing flexibility of the latest Training Packages is highly valued." (National-Quality-Council, 2010, p. 3)

The rationale supporting the adoption of this platform is that consultation is critical to the successful development or review of a Training Package, and must be undertaken on a national and individual State and Territory level. Stakeholders' agreement can only be achieved through appropriate consultation with key stakeholder groups.

The UK uses technology and social networks for consultation with employers and individuals. (Roberts, 2012) The UK has also implemented several online strategies for developing and updating occupational standards.

Training regulations in Germany are reviewed periodically by employers' associations and unions to ensure relevance and currency. Employers' associations and unions monitor changes in the workplace and assess the need for revising the training regulations.

Strategies identified by this study for tracking emerging skill needs include:

  • Using technology and social networks to consult with employers and to track their emerging skill needs.
  • Surveying employers via print and digital mediums.
  • Tracking the performance of all graduates for five years, and finding out what additional training they require and what their needs were, and reflecting this information in the curriculum.

Recommendation 19

That the composition of standard development, validation, and updating committees reflect a higher employer to employee ratio.

Represent industry needs

Reflect workplace

realities

Both the literature review and the consultation with key informants also indicated that industry should be involved in standard development, validation, and updating processes to ensure that the standards meet not only current industry needs, but also anticipated needs in the foreseeable future.

There is emerging agreement that, while employees can accurately describe current competence requirements in their occupation, they are less likely to be able to identify and anticipate future skills needs. On the other hand, because employers are involved in short-, medium- and long-term strategic planning they can identify both current and future skills needs.

The literature review and the consultation with key informants revealed several best practices for the development, validation, and updating of occupational standards to ensure their relevance to industry's needs. A key strategic suggestion was to have more employers and fewer employees on standard development, validation, and updating committees.

In the UK, NOS Strategy 2010 – 2020 is being implemented under the overarching framework of nine principles, with the first principle being that NOS must be articulated by a representative sample of relevant employers across the UK. A facilitator and representatives of employers (managers, employees), professional bodies, trade bodies, and regulatory bodies are involved in the development of the occupational standards. Employers are also involved in the validation of occupational standards. (Carroll G. , 2012)

In France, the Organismes Paritaires Collecteurs Agréés, OPCA, (an organisation grouping employers and workers in the sector, working with specialists in training and development) (Lesemann, 2005, p. 25) is responsible for the development of occupational standards, and the CPNE (Commissions paritaires nationales de l'emploi) (Political entity that regroups employers and workers of a sector) (Lesemann, 2005, p. 26) is responsible for the validation of occupational standards.

In Germany, employers, trade unions, and the federal states are also involved in the development and validation of occupational standards.

In the Asia-Pacific region, employer bodies and associations, representatives from typical enterprises, expert employee representatives, professional associations, regulatory or licensing bodies, and educators and trainers are involved in the development and validation of occupational standards.

In the U.S., the Industry Technical Advisory Committee (ITAC) drives the skill standards development process (guidance, advice, and technical knowledge). It is a requirement that industry have the majority voice in ITAC and that ITAC membership reflect the composition of employers within the industry/occupation by size, region, and business diversity. (TSSB, 2010a, p. 8)

The majority of respondents pointed to the need to include industry in the development process to ensure the validity of occupational standards. They also pointed to the need to ensure the participation of employers, subject matter experts, and an advisory committee.

Although the benefit of involving employers in the development of occupational standards is unquestionable, there may be some difficulties in recruiting employers who are willing and able to commit an extended period of employee time to the standard development workshop. The consultation with key informants indicated that employers are finding it difficult to release their employees for the time required to participate in the workshop. The key informants consulted also noted that the employees being released to participate in the standard development workshop are not always among the best qualified.

Some agencies are resorting to virtual workshops supported by digital technology to overcome the difficulty of finding highly qualified experts to attend the face-to-face meeting. A digital approach has several other advantages including cost and expediency.

Recommendation 20

That the updating of Red Seal standards be conducted as part of a continuous improvement process rather than at fixed intervals varying between 3 to 5 years. An information management system should be used to facilitate the continuous updating of occupational standards. The same system should also be used for developing and validating standards

Represent industry needs

Reflect workplace realities

In some sectors of the economy, there is a growing need to ensure that employees possess cutting-edge skills in their occupation. This need is often exacerbated by the diminishing shelf life of technical knowledge. This new requirement calls for innovative approaches for tracking employers' current and emerging skill needs.

The key informants indicated unanimously that each trade is unique and, as such, occupational standards are updated on a need basis, depending on new developments.

All respondents included the key word "change" in their responses when asked to identify the criteria used to assess the need for updating standards. The most common changes identified by respondents include technology, scope, trends, usage, skill set, best practices, legislation, working practices, environmental/green issues, and learning outcomes. The great majority of respondents also mentioned industry and/or user requests as a criterion used to assess the need for updating standards.

It is worth noting that Australia has moved into a continuous improvement cycle for updating its occupational standards (Training Packages). (Australian-Government, 2010) Australia uses websites to publicize, seek feedback on, and to validate the standards.

The UK uses technology and social networks for consultation with employers and individuals. (Roberts, 2012) The UK has also implemented several online strategies for developing and updating occupational standards. When asked to identify desirable improvements to the existing system, Roberts, Assistant Director at the UK Commission for Employment and Skills, indicated the increased use of technology and social networks for consultation with employers and individuals. (Roberts, 2012)

The NOA and the OPS are updated at fixed periods or in response to industry requests. The review of exemplary standards indicated an emerging trend calling for the constant monitoring of changes in skill requirements and the continual updating of occupational standards to ensure that they are relevant to industry needs and that they reflect workplace realities.

Recommendation 21

That the draft Red Seal standards be subjected to developmental testing procedures (one-to-one, small group and field tests) to assess their usability and ensure that they meet the needs of the intended users. The standards should also be reviewed to ensure that it is free from discrimination against individuals or group of individuals. Additionally, that draft Red Seal standards should be released for review and comment by the general public.

Clearly specify expected performance (transparency)

The jurisdictions expressed some concerns about the NOA, two of which were related to the need for the NOA to be transparent in terms of expected performance:

  1. The lack of clarity in specifying expected performance.
  2. The absence of indications regarding critical skills to be assessed.

The key informants identified two desirable changes for improving the standards:

  1. The need to provide clearly articulated performance expectations and criteria.
  2. The need to identify critical skills to be assessed.

The key informants also identified two disadvantages of the current standard structure:

  1. Difficulty in balancing knowledge requirements with performance requirements.
  2. Performance criteria or a performance measure not specified.

The key informants indicated that standards must help to reduce ambiguity and improve understanding of skills expectations in specific work contexts. Similarly, Stewart and Hamlin indicated that there are hundreds, perhaps thousands of users who struggle and become frustrated in applying occupational standard for development purposes. A developmental testing of the prototype standard, involving a user verification and revision can improve the usability of document. (Stewart & Hamlin, 1992, p. 14)

The development of occupational standards is not a science. The process used is based on tacit knowledge, heuristics, expert opinion, and conventional wisdom. If the process is weak, the product is more likely to be ineffective. Instructional designers have developed the User Verification and Revision process to identify and eliminate flaws, errors, and ambiguities in instructional material. Research conducted by Chinien indicated that even material verified and revised using input from a single user is significantly more effective than the unrevised version of the material. (Chinien, 1990) The consultation with key informants suggested that the draft occupational standard should be user verified and revised to assess its usability and ensure that it meets the needs of the intended users.

The recommended procedures for conducting the User Verification and Revision process include one-to-one sessions, small group evaluation sessions, and field tests. The consultation with key informants suggested that the participants in the User Verification and Revision process should include apprentices, employers, expert workers, and instructors.

The standards should also be reviewed for gender and culture biases, and to determine whether reasonable consideration has been given to people with disabilities.

The NOA is validated by each jurisdiction using a purposive sample of experts. Some countries seek input from the general public as part of the process for validating occupational standards. This additional step contributes towards increasing the credibility of occupational standards.

Recommendation 22

That the standard template and all its component parts be used for the specification of the Red Seal Occupational Standards. All the components should be precisely aligned to ensure the highest level of transparency of expected performances.

Represent industry needs

Clearly specify expected performance (transparency)

Be credible within industry

Reflect workplace realities

It is important to use a unique template for developing national occupational standards in order to avoid confusion and ambiguity, and to ensure consistency within and between standards. The CCDA has prescribed a common format for the NOA and OPS, and both have a policy manual that guides their development processes.

Several agencies have developed a common template for developing their NOS. In the UK, a common template is now being used to structure all occupational standards in a common format in order to facilitate ease of access by employers. China has a prescribed format for developing standards. (Qualifications Framework Secretariat, 2008) Australia has developed a unique template for all of its Training Packages. The ILO has developed a model and a template for transnational occupational standards. Several of the Canadian sector councils have also developed their own template to standardize the format of the occupational standards.

A review of international case studies of innovative and exemplary occupational standards has revealed that a consensus is emerging regarding the most desirable features to include in national occupational standards, as well as the most appropriate methodology to use for developing standards. Insights gained from the analysis of the study findings were crossed with the current and future needs of the Red Seal program, and the results were used to generate recommendations for a new occupational standard structure and development process for the Red Seal program. Recommendation 22 proposes a template for the new Red Seal standard. This template encapsulates the distinctive features to be included in the standard. All the components included in the standard are precisely aligned in the template to show their linkages and relationships, in an effort to bring the highest level of transparency to the expected performance.

Recommendation 23

Recommendation beyond the scope of this research project

That due consideration be given to worker's attitude in the Red Seal standards, since proper attitude is essential for effective and efficient performance.

Be credible within industry

Due to the effects of the dramatic economic, technological, and socio-technical changes in the workplace, the previously narrow view of the goal of TVET (i.e., the development of technical skills) has become broader and more comprehensive. The New Zealand Treasury Board has argued that "specialized technical skills are important, but so too are the many "soft" skills such as communication, teamwork, creativity and problem solving. Attitudes and values matter as much as knowledge and technical skill." (New Zealand Government, 2008, p. 2) TVET is now also expected to prepare people to become effective members of a flexible, adaptable, and competitive workforce (New Zealand Government, 2008, p. 2).

This recommendation to integrate the attitude dimension in the Red Seal standard is in line with the new conception of TVET reflected in the competency framework (Figure 25) developed by the ILO (International Labour Organization, 2008b, p. 12). Studies have demonstrated that employers perceive worker attitudes and values to be just as important as occupational skills. Additionally, the shift to a development paradigm that is more sustainable also calls for workers with strong values, ethical behaviours, and decision-making skills. Both values and attitudes influence workers' performance. (Australian Government, 2007)

This recommendation is also based on another new definition of TVET, outlined in the UNESCO and ILO Recommendations on Technical and Vocational Education and Training for the Twenty-first Century, adopted by UNESCO's General Conference in 2001. The definition is also broad and inclusive: "TVET refers to those aspects of the educational process involving, in addition to general education, the study of technologies and related sciences, and the acquisition of practical skills, attitudes, understanding and knowledge relating to occupations in various sectors of economic and social life." (UNESCO-ILO, 2002)

Effective performance in TVET consists of the successful interactive effects of skills, knowledge, and attitude. All TVET programmes aim to develop these domains of learning. While there is a considerable amount of literature dealing with the development of cognitive skills, there is a paucity of publications dealing with the affective domain in TVET (Chinien, C., 2003).

According to the limited studies available, five approaches show some potential for attitude development in TVET. These are a democratic approach, indoctrination approach, group discussion, dramatic involvement, and role modelling (Chinien, C., 2003).

Testing of attitudes can be performed using self-rating, direct observations, and instructor rating. Irrespective of whether the attitude is to be tested by the Red Seal exam, it is still beneficial to delineate its dimensions in the standard to inform the development of on-the-job and in-school training.

Recommendation 24

That the guiding principles developed by the Alliance of Sector Councils be adopted to frame the planning, development, validation, implementation, and maintenance of occupational standards for the Red Seal program.

Clearly specify expected performance (transparency)

Be credible within industry

The great majority of the national and international key informants (88%) and 100% of respondents from the jurisdictions rated the need 'Be credible within industry' as either very important or extremely important. All respondents from the jurisdictions rated this need as the number one priority.

The Alliance of Sector Councils (TASC), in close collaboration with all the Canadian Sector Councils, has developed 11 guiding principles that should be applied consistently at all stages of planning, development, implementation, and maintenance of occupational standards. These principles are:

  • Accessible, equitable, and fair;
  • Current, relevant, and valid;
  • Coherence and rigour;
  • Confidentiality;
  • Consensus;
  • Harmonization;
  • Impartiality and independence;
  • Openness and transparency;
  • Representative;
  • Sustainability; and
  • Voluntary. (The Alliance of Sector Councils, no date)

The key informant from the CTHRC indicated that the sector council has fully adopted these principles, as mandatory requirements, for all their occupational standards. The International Network of Sector Skills Organizations is also considering the adoption of these guiding principles.

The CCDA and the jurisdictions have already established some rigorous explicit and implicit guidelines for developing the NOA and the OPS. The adoption of these TASC principles will provide a comprehensive framework to operationalize the planning, development, validation, implementation, and maintenance of occupational standards for the Red Seal program. The implementation of this recommendation should increase the credibility of the Red Seal standards both nationally and internationally.

Recommendation 25

Action research be conducted to test the trustworthiness of Red Seal standards for developing assessment tools with an acceptable level of consistency regardless of method of assessment.

Support multiple assessment strategies to assess performance

Considerable efforts are made to ensure that an occupational standard is a valid and defensible source of information for developing assessment tools.

Best practices in assessment and evaluation indicate that if several individuals were to use an occupational standard for developing assessment tools, there must be a level of internal consistency among the tools, regardless of the method of assessment. An assumption is made that all the elements embedded in the occupational standard will contribute to ensuring a level of consistency regardless of method of assessment. There was no indication in the literature that this assumption has previously been tested. Consequently, it is recommended that action research be conducted to test the trustworthiness of using Red Seal standards for developing assessment tools with an acceptable level of consistency regardless of method of assessment.

Recommendation 26

That during the development and validation of Red Seal occupational standards, each skill standard be rated as "mandatory" if it represents core competences required nationally or as "elective" if it reflects specific regional requirements in competence requirements.

Reflect workplace realities

The great majority (84%) of national and international key informants and key informants from all jurisdictions rated this need ('Reflect workplace realities') as either extremely important or very important. Respondents from the jurisdictions also rated this need as their number two priority.

The key informants consulted believed that national occupational standards (NOS) should reflect workplace realities and regional differences in workplace requirements because of the implications for exam development and certification. It was suggested that a distinction be made between mandatory and elective standards. Some international NOS reviewed also classify Skill Standards as mandatory or elective. Mandatory standards are competences required to be considered competent nationally. Elective standards are competences required in a special context or work environment. Key informants also pointed to the need to allow apprentices to meet their elective requirements in occupational fields outside their trade specialization to make them more employable and flexible.

The OPS uses the 'mandatory' and 'elective' concept, while the NOA uses the 'common core' and 'non-common core' concept to address the same issue. Both concepts and their descriptive labels are equally acceptable. However, if the CCDA should embrace the suggestion to allow apprentices to meet their elective requirements in occupational fields outside their trade specialization, the mandatory/core concept would be more appropriate.

Based on the discussion, it is recommended that the Red Seal Skill Standards be designated as 'mandatory' if they represent core competences required nationally, or as 'elective' if they reflect regional differences in competence requirements.

This course of action has already been adopted by Australia and South Africa. The Australian Training Packages are made up of mandatory and optional components endorsed by the NQC and optional support materials. (National-Quality-Council, 2010, p. 24) The South African Unit Standards also includes core and elective units.

Recommendation 27

That the validation of Red Seal standards include a peer review of associated trades in order to identify and resolve trade infringements.

Be credible within industry

Reflect workplace realities

One of the suggestions from the consultation with key informants was to conduct a peer review involving associated trades to identify and resolve any trade infringements. Any identified trade infringements will need to be resolved with the collaboration of all parties concerned. The review of literature did not provide any indications that other countries are using a similar process to address potential trade infringements.

This peer review will eliminate trade infringement issues when the standard is still in the developmental stage, eliminating costly and time consuming adaptation at a later stage.

Recommendation 28

That CCDA conduct a proof of concept to explore the feasibility and potential benefits of developing a hybrid standard development methodology, which combines strengths of Functional Analysis (FA) and DACUM. This hybrid methodology should be tested by developing one Block of an occupation during a few DACUM workshops, so that the outputs of the hybrid approach can be compared to those of DACUM. Embedding the proof of concept within a regular DACUM workshop eliminate the need for costly investment in research and development. Additionally, if the hybrid methodology fails to function as expected, the facilitator can revert to DACUM to salvage the block being used as test bed. DACUM should remain the methodology of choice for developing the Red Seal standard until convincing evidence can be drawn from the proof of concept to justify the adoption of the hybrid methodology.

It is highly recommended that CCDA commissioned a study designed to make a detailed comparative analysis of occupational standard developed using DACUM and FA in order to validate results of experimental research and experts' opinion, which suggest that occupational standards developed using FA is qualitatively better that those produced using DACUM.

A second option is to reengineer and retool DACUM so that it meets and exceeds the current and future needs of the Red Seal program as an occupational development methodology. This process will involve the survey of DACUM leaders and practitioners world-wide to determine areas needing improvement and input to guide improvement; and reengineer, retool, and pilot test the new DACUM with the assistance of the international community of DACUM experts. The reengineered DACUM will take advantage of digital technology and appropriate application software. This initiative will build on Canada's leadership, reputation, and pride for having designed the simplest, most objective and most widely used occupational development methodology in the world, which is highly accessible and affordable even to least developed countries.

Represent industry needs

Clearly specify expected performance (transparency)

Be credible within industry

Reflect workplace realities

DACUM was developed in Canada by the Department of National Defence for analyzing occupations. It is a national pride of the Canadian ingenuity in developing an extremely simple solution to resolve a highly complex problem in the field of standard development. DACUM is the most widely use methodology for developing occupational standards internationally.

Functional Analysis (FA) is the second most widely use methodology for developing occupational standards. Some of the countries that are considered as world leaders in occupational development are using FA. The European Union uses FA for the development of their standards as well as for maintaining relevance and currency of their standards (Mansfield, B. and Schmidt, H., 2001). Australia uses functional analysis and the modified DACUM technique to identify tasks and roles. (Australian-Government, 2010, p. 27). The UK has been using FA for more than 25 years for developing its national occupational standards. The Asia-Pacific Region uses a combination of a Modified DACUM and Functional Analysis. Delphi.

The main strengths of DACUM lie in its simplicity and the systematic process it offers for rapidly generating the content of occupational standards with the collaboration of a small group of expert workers. In spite of these strong positive features, DACUM was perceived by a key informant to oversimplify skill sets, to be too narrow in focus, and to discredit the skill. On the other hand, the literature and the key informants consulted suggest that the FA methodology provides more depth and richness to the skills. However, the international experts believed that while FA can provide a better conceptual lens to analyze occupations, the methodology lacks a systematic approach for subdividing an occupation into its component parts. Tien, Chou, & Ven have also praised the effectiveness of FA methodology:

  • It is a holistic system with a structured analysis approach used to analyze whole occupations in terms of outcomes rather than specific tasks;
  • The granularity of the analysis can be of several levels to accommodate the scope of the occupational field;
  • It can present a clear link between what is needed for employment and what is taught in training;
  • The methodology not only allows the identification of the best current (existing) practice, but also competences that may be required in the future (Tien, Chou, & Ven, 2001).

A key informant has argued that:" FA is the most rigorous and robust approach to NOS development. In the hands of skilled analysts and expert occupational practitioners, it inevitably establishes a structure which has logical internal relationships and an elegant description of good practice in the occupation" (Geoff Carroll, Key Informant).

An experimental study conducted in Taiwan to assess occupational standard development methodologies (Functional Job Analysis, DACUM, Position Analysis Questionnaire, and CODAP), revealed that Functional Analysis was more suitable than other methodologies however, researchers found considerable variations between standards developed through different methodologies (Tien, Chou, & Ven, 2001).

Meriot has argued that because of the changing skill requirement, the U.S. traditional behavioural strategies and the British functional approach for developing standards must be replaced with a more multidimensional approach. The author further argued that the best occupational standard development methodology should be useful, scientifically sound and fit-for purposes, such as defining occupational and training standards and creating assessment and certification tools (Meriot, 2005). Meriot has praised the quality of the a standard development methodology developed in France, Emploie-Type Etudié dans sa Dynamique (ETED). In spite of a major deployment of effort and a considerable amount of time spent to research the ETED, project staff were unable to obtain sufficient information on the methodology to be able to compare it with DACUM and FA.

A key informant who has been using DACUM and FA suggested the use of a hybridized methodology which combines the ability of DACUM to quickly elicit details with the robust structure produced by Functional Analysis.

While FA provides the best conceptual lens to analyze an occupation, the process can be significantly enhanced by combining it with a more systematic approach to drive and facilitate the analysis process. The WDM team therefore recommends that FA should be used in conjunction with DACUM for developing the new Red Seal standards.

The proposed standard development strategy has a two-pronged approach. During the first stage, Functional Analysis and DACUM are used for developing the occupational standard. All the components of the occupational standard are then uploaded to the database of an information management system. From that point in time it would not be necessary to repeat that process for that particular occupation, because the updating of the standard can be achieved, on a continuous basis and also at specific points in time, with the assistance of one to two experts and an occupational analyst. This digital system should include multiple features specifically designed to streamline the validation of occupational standard in the most efficient, effective and economic way.

Supporting information: For more detailed definitions of FA and more information on the procedure followed to conduct FA see also (ILO, 2011, p. 24) and (Carroll, G. & Boutall, T., 2011).

Recommendation 29

That Red Seal skill standards be formulated so that they are broad enough to be used across a range of settings, but flexible enough to be useful in any specific context so that they can support the development of multiple assessments.

Provide flexibility for adapting standards to labour market demands

Flexibility and transferability are often used interchangeably in the occupational standard literature, as illustrated in the following example. The CTHRC defines flexibility as the ability to apply the skills embedded in a standard in various contexts. This aspect of flexibility is facilitated by Recommendation 35, which deals with the issue of transferability.

Flexibility can also be defined as an enabling structure that facilitates rapid modification of the standard to accommodate changes in technology and methods (Mansfield, B.; Andersson, L. G., 2004). The Range is the element that facilitates the updating of occupational standards (Australian Government, 2007).

The review of literature identified 115 uses of occupational standards. This finding suggests that occupational standards must be flexible enough to accommodate these various uses. Recommendation 5, which calls for the integration of the standard into an appropriate database housed in an information management system supported by digital technology, will address this aspect of flexibility.

Recommendation 30

That an information management system, which is embedded in a customized digital platform, be used to facilitate the traceability of transferable skills and knowledge across occupations and across sectors.

Recommendation 31

That the CCDA, HRSDC and the jurisdictions promote to all their key stakeholders the benefits of embedding enabling elements to facilitate the traceability of transferable skills in Red Seal standards.

Allow the transferability of skills across occupations

The Australian Training Package defines transferability as the need "for skills described within units of competency to be used in a range of different contexts within an industry or across industries" (Australian-Government, 2010, p. 17). The transferability of skills is considered to be one of the most important aspects of the Australian standard.

Canada was once a world leader in transferable skills research. The first generic skills study was pioneered in 1979 by Arthur Smith, an employee of Employment and Immigration Canada (Smith, 1979). Smith is the father of the transferable skills concept and its field of research. The International Labour Organization later adapted the concept to develop the Module of Employable Skills, which provides the infrastructure for developing transferable skills standards and transferable learning modules.

The Generic Skills study conducted by the Government of Canada (Smith, 1979) determined a threshold of 30%, whereby "if 30% or more of the job holders responded to a particular skill, it was recorded as an occupational requirement" (Smith, 1979, p. 2). The reusability of these skills was determined as each pair of occupations was "compared to determine the percentages of skills held in common (COMMUNALITY FACTOR) and the percentage of skills which each occupation used of the total skills held by the other occupation (TRANSFERABILITY FACTOR)" (Smith, 1979, p. 2).

For example, Aircraft Mechanics use 93% of the tool skills of a Body Repairer and Body Repairers use 72% of the tool skills of an Aircraft Mechanic (transferability factors). The communality factor between the two is 68% (Smith, 1979).

Seventy per cent of the key informants (more than two-thirds of whom were from the jurisdictions) believed that the transferability of skills across occupations and across sectors is an important feature of an occupational standard. The case study of best practices revealed two examples of standard structures that facilitate the transferability of skills:

  1. the Australian Training Packages and
  2. SkillsNET.

Knowledge of the transferability of skills has several key advantages:

  • Facilitates the retooling of workers' skills for new occupations;
  • Reduces time necessary for retooling;
  • Facilitates the management of the workforce when there are considerable variations in labour market requirements;
  • Enables savings in both time and resources for the design of training materials; and
  • Enables apprentices who want to change their apprenticeable occupation to transfer competencies already acquired.

The Unit Competency of the Australian Training Packages is coded to facilitate transferability. As for SkillsNET, the SkillObject-centric competencies are clustered using a taxonomy that identifies and tags competencies in various configurations (e.g. parent–child relationship, co-dependent, and so on). Transferability of skills can easily be tracked across occupations and sectors.

Further research in this area was undertaken by the International Labour Organization under the theme Module of Employable Skills (ILO, 1992). So far, apprenticeship training has paid very little attention to the transferability of skills. The threat of trade infringement is often seen as a major barrier to the adoption of the concept.

Complete transferability of skills standards must be facilitated with an appropriate database housed in an information management system that is supported by digital technology. Recommendation 5 fully addresses the digitalization of the standard development, validation, and updating processes.

Recommendation 32

That the proposed template for the Red Seal standards be adopted to increase flexibility and thereby allow for the continuous updating of standards. The proposed standard structure and the standard development processes, along with the adoption of a digital platform, will make the standard extremely flexible and customizable to the needs of various key stakeholders.

Provide flexibility for adapting standards to labour market demands

A review of international case studies of innovative and exemplary occupational standards has revealed that a consensus is emerging regarding the most desirable features to include in national occupational standards. It is also becoming common practice to develop a template to ensure the uniformity of occupational standard. The ILO had developed a template for developing transnational standards (International Labour Organization, 2006). The UK has recently developed a new template for all its NOS (UK Commission for Employment and Skills and the Alliance of Sector Skills Councils, 2010b). Australia has also developed a template for its National Training Packages (Australian-Government, 2010).

Insights gained from the analysis of the study findings were compared with the current and future needs of the Red Seal program, and the results were used to generate recommendations for a new occupational standard. The recommended structure of the proposed Canadian National Skills Standards is captured in the recommended template.

The template should be validated by the jurisdictions' users of Red Seal standards. Results of the validation should be used to revise the template.

Supporting information: See also:
Rationale for Recommendation 20;
ILO (International Labour Organization, 2011);
(UK Commission for employment and skills and the Alliance of Sector Skills Council, 2010a, p. 7);
Case study Australian Training Packages.

Recommendation 33

That the name "National Occupational Analysis" be changed to one that is more descriptive and consistent with the branding of the Red Seal program. The suggested new name is "Canadian National Skills Standards" (CNSS).

Represent industry needs

The CCDA initiative to enhance the Red Seal standard should be supported by rebranding the product. Rebranding the NOA requires a name change. The review of international best practice for branding national standards revealed the following defining attributes of name selection:

  1. Country of origin;
  2. Geographical scope (national, regional, transnational);
  3. Emphasis on skills, competence, performance; and
  4. Standard

On the basis of these attributes, the recommended name for the Red Seal program is "Canadian National Skills Standards".

Recommendation 34

That a National Occupational Competency Framework be adopted to inform and facilitate the joint CCDA and HRSDC efforts to enhance the Red Seal program.

Allow the identification of critical skills to be assessed

Support the assessment and recognition of experiential learning (non-formal and informal learning)

A general conclusion reached from the review of international best practices for defining occupational competence was that a framework is essential to establish a holistic, comprehensive and systematic approach for the development of occupational, training and assessment standards. Given that such a framework has not yet been developed for the Red Seal program, insight gained from the study findings, coupled with information gleaned from some of the world's most knowledgeable experts, were used to develop a prototypical framework. The recent expertise acquired by WDM project staff in developing a digital skills framework for HRSDC (Chinien & Boutin, 2011) also contributed in this model building process. Additionally, some lessons learned from the review of the Chinese vocational qualification and certification framework were used in the design and development of the proposed framework.

Due to the effects of the dramatic economic, technological and socio-technical changes in the workplace, the previously narrow view of the goal of TVET (i.e., the development of technical skills) has become broader and more comprehensive. The New Zealand Treasury Board has argued that "specialized technical skills are important, but so too are the many "soft" skills such as communication, teamwork, creativity and problem solving. Attitudes and values matter as much as knowledge and technical skill" (New Zealand Government, 2008, p. 2).

TVET is now also expected to prepare people to become effective members of a flexible, adaptable and competitive workforce. This new conception of TVET is reflected in the competency framework (Figure 25) developed by the ILO (International Labour Organization, 2008b, p. 12).

Due to the evolution in the conceptualization of workers' competence, the array of skill requirements has also expanded to include much more than technical skills. The framework also includes other competence domains such as social, communication, cognitive/problem solving, and learning to learn, as well as attitudes and values.

The CCDA effort to reengineer the Red Seal program could benefit from a sound conceptualization of competence to provide the underpinning foundation for developing the architecture of its standard and for engineering its contents. It is worth noting that the OPS has a definition of competence: "Ability to perform tasks and duties to the standard expected in the workplace" (Canada, no date). This definition is not part of any framework, but is simply included in the definition of terms. This definition of competence is very narrow in comparison to international trends. It should be noted, however, that in defining the dimensions of competence, the OPS suggests that competence include all aspects of work performance, not just the narrow task skills, and lists four dimensions of competence:

  1. task skills;
  2. task management skills;
  3. contingency management skills; and
  4. job/role environment skills (Canada, no date, p. 5).

To prevent Canadian jobs from being outsourced, Canadian workers will need to become more productive. There is little room for improving productivity by working harder. The true potential for improving the productivity of the Canadian workforce is by working smarter. A recent review of labour productivity conducted for the New Zealand Treasury suggested that "higher skills increase individuals' productivity and the productivity of others they work with. Skills have a dynamic effect on productivity growth by increasing the capacity to innovate and apply new ideas. Skills can enhance the returns to capital investment, and increase firms' ability to adapt to new markets and competitive challenges" (New Zealand Government, 2008, p. 2).

The ILO Labour Conference of 2008 also concluded that a highly skilled workforce "fuels innovation, productivity, increase in enterprise development, technological change, investment, diversification of the economy, and competitiveness that are needed to sustain and accelerate the creation of more and better jobs in the context of Decent Work Agenda, and improve social cohesion" (International Labour Organisation, 2008a, p. 2).

This framework was designed to assist the CCDA in enhancing the Red Seal program to make it more credible with industry, to better reflect workplace realities and to cater to industry needs.

Recommendation 35

That a Steering Committee composed of employers, workers, regulators and union representatives be established to provide intellectual guidance and practical direction to the development of Red Seal standards.

Be credible within industry

Reflect workplace realities

Represent industry needs

In the U.S., the Industry Technical Advisory Committee (ITAC) drives the skill standards development process (guidance, advice and technical knowledge). It is a requirement that industry have the majority voice in ITAC and that ITAC membership reflect the composition of employers within the industry/occupation by size, region and business diversity (TSSB, 2010a, p. 8).

In Australia, a Steering Committee oversees the development and formal review of a Training Package. The Steering Committee also advises on project expenditure. The Steering Committee includes representatives from:

  • Small, medium and large enterprises from industry sectors covered by the Training Package;
  • Relevant industry associations, unions and groups;
  • At least one RTO (public or private, or both);
  • At least one STA; and
  • DEEWR (observer).

From a quality perspective and as part of industry validation, the Steering Committee usually would agree on the final version of the Training Package or review the report prior to submission to the National Quality Council (Australian-Government, 2010, pp. 24-26).

The CTHRC also has a Steering Committee that provides intellectual and practical guidance to the standard development process. The following are the specific responsibilities of a steering committee as described by the CTHRC:

  • Assist in defining the scope of the occupation;
  • Select consultants;
  • Approve communication plan;
  • Monitor project budget and schedules;
  • Review and approve deliverables; and
  • Ensure that the standard meets industry needs (Mondor, 2011).

It is highly desirable to establish a steering committee for developing the new Red Seal standard. This committee will ensure that the standard is developed according to agreed guiding principles, and that it is credible within industry, reflects workplace realities and represents industry needs. The committee will also oversee the quality control and quality assurance measures and will ratify the standard after its validation. Key informants suggested that the committee should be composed of the following key stakeholders: regulators, employers, unions and training providers.

Recommendation 36

That a norm referencing exercise be conducted at the front-end of the standard development process to benchmark the Canadian standards against that of other countries.

Reflect workplace realities

A well-educated workforce is a key ingredient for becoming and remaining competitive in the globalized, knowledge-based economy. This need has been the driving force of several policy responses to achieve a world-class workforce. Most countries are now norm referencing and benchmarking their occupational standards to those of other countries.

The HRSDC provides a considerable amount of funding to the sector councils for developing occupational standards. Whenever possible, the occupational standards developed by the sector councils should be consulted when developing standards for the Red Seal program. This can be achieved by norm referencing the relevant standards developed by the sector councils.

According to the CTHRC, norm referencing "refers to the process of comparing programs to inform decisions on the program scope and to establish common benchmarks. [...] Norm referencing activities should be undertaken to benchmark the standard with existing relevant international, regional, or national standards and policies and procedures" (Mondor, 2011, p. 7).

The norm referencing exercise will facilitate the importation and tailoring (Carroll & Boutall, 2011) of standards from the Canadian sector councils or other national and international organizations for examination and consideration when planning the development of standards for the Red Seal program.

Recent norm referencing exercises conducted by WDM-Consultants showed some gross deviations in how occupational skills are classified in terms of Key Competencies, Skills, and Sub-Skills across nations. Some skills that were classified as Key Competencies in one NOS were classified as Sub-Skills in others. This issue significantly affects the validity and reliability of using NOS to inform foreign credential recognition and the mobility of foreign workers.

Recommendation 37 – Recommendation for further research

That the required knowledge embedded in the Red Seal standard be classified according to a modified version of Gagne's learning hierarchy. This hierarchy will specify the level of knowledge expectation for a journeyperson in the trade. For ease of application, the six levels of Gagne's learning hierarchy have been condensed into the following five levels:

  • Level 1: Recall and recognition
  • Level 2: Discrimination
  • Level 3: Concepts
  • Level 4: Rules and principles
  • Level 5: Problem solving
Provide a basis for on-the-job training and in-school training

The NOA provides critical information that can be used as a basis for developing on-the-job training and in-school training. The value of that information for curriculum development and training could be enhanced if the knowledge requirements were classified according to Gagne's learning hierarchy (Gagne, 1985).

Bloom's Taxonomy is very often inappropriately used for developing training. The levels of the taxonomy are independent from one another. Test questions can be written at any specific level. In contrast, the levels of the learning hierarchy are interrelated. Learning builds on the achievement of successive levels.

When the level of learning hierarchy is not specified, curriculum development for knowledge acquisition is reduced to simple guesswork.

Recommendation 38 –

Recommendation for further research

That the CCDA conduct a proof of concept study to examine how the taxonomy levels for the psychomotor domain can be applied to establish expected levels of performance for psychomotor skills. The CCDA should use a modified version of Ferris and Aziz's classification (Ferris & Aziz, 2005) to assign levels to the psychomotor skills according to the following levels:

Level 1: Plan work operations: Ability to use technical specifications to plan the execution of a work outcome.

Level 2: Handle materials: Ability to handle materials appropriately and sustainably.

Level 3: Execute work outcomes competently using tools and equipment: Ability to use tools and equipment for performing a range of tasks rapidly, efficiently, effectively and safely.

Level 4: Control and assess work outcomes: Ability to control and assess quality of work outcomes by identifying and correcting particular deficiencies.

Clearly specify expected performance (transparency)

Provide a basis for on-the-job training and in-school training

Support alternative forms of assessment

Support multiple assessment strategies to assess performance

Support methods of assessment that evaluate multiple competencies simultaneously

While there is widespread effort to assign levels to the knowledge components embedded in an occupational standard, little effort is currently being made to assign levels to the psychomotor skills according to a known taxonomy. Yet most trade workers spend two-thirds or more of their time at the workplace performing psychomotor skills. Specifying the level of psychomotor skills helps to pitch training and assessment at the required level of performance.

Supporting information: See (Ferris & Aziz, 2005) for classification.

Recommendation 39 –

Recommendation for further research

That the CCDA undertake a study to examine how levels of workers' autonomy can be embedded in the Red Seal standards, and assess the benefits of implementing such a levelling structure.

Support multiple assessment strategies to assess performance

The review of exemplary standards indicated that several countries (e.g. Hong Kong and Malaysia) specify workers' level of autonomy in their occupational standards. While including this dimension in occupational standards appears to have merit, little is known about the stratification of the levels and their immediate benefits. Specifying the level of workers' autonomy may help to pitch training and assessment at the required level of performance.

Supporting information: (ASIA-Pacific Economic Cooperation, 2009)

Date modified: