NiveauExamens interprovinciaux Sceau rouge / Sceau Rouge

Analysis of Red Seal Exam Questions for Construction Electrician & Motor Vehicle Body Repairer

Prepared by the Centre for Canadian Language Benchmarks
For Human Resources and Skills Development Canada
April 2011

Executive Summary

This project was commissioned by Human Resources and Skills Development Canada (HRSDC) on behalf of the Canadian Council of Directors of Apprenticeship (CCDA) to conduct an analysis of the reading demands of the Red Seal exam questions for the Construction Electrician and Motor Vehicle Body Repairer trades. The objective of the project was to determine the current reading competency levels of English and French Red Seal examination questions, as selected by HRSDC, to ensure consistency in the development of Red Seal examinations over time.
The results of the analysis in terms of Canadian Language Benchmarks (CLB) and Niveaux de compétence linguistique canadiens (NCLC) show that the test questions, for both Construction Electrician and Motor Vehicle Body Repairer trades, generally fall in the CLB 6-9 range. 

The following is a summary of observations and recommendations:

Construction Electrician

Observations:
Based on the analysis of the Red Seal exam questions in both official languages, the following points can be noted:

• A majority of questions (71%) included content taken from the Canadian Electrical Code (CEC).
• There were 10 figures provided to help examinees in answering the questions. They varied from simple to complex.
• The average number of simple sentences per stem was six. The average number of complex sentences per stem was two.
• The average number of words per stem ranged between 12.5 and 27.4 words for the English questions, and between 12.5 and 40 for the French version. A large number of questions included over 30 words per stem in English and over 40 words per stem in French.
• The use of passive voice constructions was fairly common, which could make the questions more complicated to read.
• The vocabulary was complex and sometimes covered a large area, especially in the French questions. Many terms were technical and would possibly be known by most apprentices to the electrical trade. Several acronyms were not spelled out in both French and English questions, which goes against the HRSDC guidelines.  
• As mentioned, the CLB and the NCLC level range for the questions reviewed was 6 to 9. Many of the questions received a higher benchmark rating due to the following factors:
• use of complex and technical vocabulary, and of unexplained acronyms;
• use of complex vocabulary taken from the CEC;
• use of too many words (up to 40) in stems and responses, especially in the French questions.

Recommendations:

• When editing Red Seal questions that are authored, it is recommended to follow plain (clear and simple) language principles and HRSDC’s guidelines for writing multiple-choice questions (Style Guide), especially as it relates to Rule #3 as well as other guidelines for this purpose. This involves constructing shorter sentences and using simpler language and everyday wording.
• It is recommended that all new terms and acronyms that may not be universally known follow Rule #2 of HRSDC’s Style Guide. It may also be useful to include in the National Occupational Analysis (NOA) a list of the common vocabulary in both official languages used in Canada so that examinees can study the terms before taking the examination.
• For questions that attempt to determine if an examinee understands the correct process to follow, it is recommended to provide an appropriate diagram or flow chart to aid comprehension of the question. 
• In some questions that are high on words or use complex vocabulary/jargon, the question writers should think of alternate ways (using examples or case studies) to determine whether a candidate knows a specific fact or learning outcome.
• In order to simplify and bring the questions closer to an examinee’s experience, it is recommended to use the active voice as much as possible.
• It is recommended to use the same vocabulary in the stem and in the responses; avoid using synonyms within the same item otherwise it becomes a test of reading ability.
• It is recommended to list the responses in order of increasing wording (otherwise it increases the reading level).
• In order to simplify lengthy stems, it is recommended to use scenarios at the beginning of the stem, followed by the question.
• Selection of final questions should take into consideration the use of acronyms and whether they are identified appropriately in the question as per HRSDC’s Style Guide.
• When writing Red Seal exam questions, it is recommended to keep in mind ways of simplifying language and terms used so that the examination questions are at a college level readability (CLB 7) for the targeted clientele. 
• It is recommended, when making changes to questions based on what is recommended in this document, to modify the CLB-NCLC levels identified in red in this report.

Motor Vehicle Body Repairer

Observations:

Based on the analysis of the Red Seal exam questions in both official languages, the following points can be noted:

• The exam questions for MVBR tended to be clear and fairly easy to follow.
• There were a large number of responses that described procedures related to a process.
• There was only one diagram/figure provided as an aid to answer the questions. The graphic was very simple and easy to understand.
• There was sometimes a discrepancy between the English and French texts in terms of terminology, including verb choices.
• The average number of words per stem was 14 in English and 17 in French. Some questions included over 30 words per stem in English and over 40 in French.
• The CLB-NCLC level range for the questions reviewed is 5 to 9. There was one example of a CLB-NCLC 9 question, but this did not represent the levels noted in the majority of the questions.

Recommendations:

• When editing Red Seal questions that are authored, it is recommended to follow plain language principles and HRSDC’s Style Guide, especially as it relates to Rule #3.
• It is recommended that all new terms that may not be universally known follow Rule #2 of HRSDC’s Style Guide. It may also be useful to include in the NOA a list of the common vocabulary used in Canada so that examinees can study the terms before taking the examination.
• Where questions attempt to determine whether an examinee understands the correct process to follow, it is recommended that a process diagram or flow chart be provided to aid comprehension of the question as per Rule #4 of HRSDC’s Style Guide.