2002 NSDUH Field Interviewer Manual

10. QUALITY CONTROL

10.1 Introduction

Quality control is important at all stages of the data collection process, but nothing affects the quality of the data as much as you do. Your ability, preparedness, and willingness to properly perform your data collection tasks following all project protocols are the most important components of the overall quality control process. This chapter discusses the quality control procedures in place to help you do your best job.

10.2 Staff Training and Assessment

New-to-Project Training

During the new-to-project FI training session, it is the project staff's job to present you with all of the information necessary to properly perform your job as an interviewer on this study. This includes detailed training on conducting the screening and the interviews, as well as training on all administrative procedures. An essential part of this training involves your hands-on practice—practice using the materials and equipment, completing the screening, conducting the interview, and preparing your PT&E reports. By the end of the training session, you will have had the opportunity to actively practice every step involved in being an effective interviewer for the NSDUH. You will also participate in a certification process at the end of training (see Section 10.2.3).

To make sure you are learning what you need during the training program, project staff and other RTI personnel will monitor your performance throughout the session. They will watch while you conduct your practice cases, observe while you fill out your time sheet, and so forth. During this time, you should take advantage of the opportunity to ask any questions you might have. Therefore, the purpose of monitoring is two-fold: 1) project staff can evaluate your preparedness, and 2) you have the chance to make sure you are doing everything properly.

Once you have the opportunity to practice and ask questions about the process, the training staff will monitor your performance during the practice exercises and give you prompt feedback. Specific items to be evaluated include:

• explaining the survey and its purpose

• familiarity with the informational materials

• completing the screening process using the Newton

• explaining and obtaining informed consent

• setting up the computer equipment

• administering the NSDUH interview using CAI

• reading the questions, maintaining eye contact, pacing the questions, probing, and other general presentation skills

• following instructions on the computer screen

• resolving inconsistent information

• using function keys (entering comments, keying "don't know," etc.)

• proper completion of the end of the interview tasks

• recording administrative data

• transmitting information to RTI.

This evaluation helps the staff determine how well they have done their job of training you, and to see how well you have done your job of listening and practicing. Evening study halls are available for those who feel they need additional help during training. Trainers and supervisors may request that certain trainees attend these study halls for one-on-one work. Any interviewer who cannot demonstrate the abilities listed above will receive additional training and will be required to participate in additional practice sessions with his/her supervisor.

10.2.2 Veteran Sessions/Ongoing Training

All interviewers who have worked on the study the previous study year are invited to a special Veteran Training Session during January. While computer programs are updated for the new survey year, trainees learn results of the past year, changes for the coming year, and ways to continually improve their performance. Before receiving an assignment for the new year, veteran interviewers must also successfully complete a certification process at training (see Section 10.2.3).

Learning continues throughout the year as supervisors use specially developed materials to review procedures with staff to continually enhance their understanding of the project procedures. All interviewers have a copy of "Steps to Maximize Data Quality" which reviews the major data collection quality issues (see Exhibit 10.1). This document is always readily available in the Job Aids portion of the Showcard Booklet.

In addition, ongoing training includes field observations. You may be accompanied by a project supervisor as you work. The purpose of such field observations is to ensure the collection of high quality data for our client. These observations are not cause for concern but should be viewed as an opportunity to be sure you are following important study protocols. In order to help you perform your job better, you will receive constructive feedback based on the observer's findings and possibly re-training if needed.

10.2.3 Certifications

Before receiving an assignment, all trainees (both new and veteran) must read and sign an agreement to conduct all field work in accordance with established project protocols.

In addition, periodic assessments of all field staff will determine the level of understanding of project procedures. Staff not achieving an adequate score on the first assessment will participate in additional study before completing another assessment. Any staff not able to pass the second assessment will be released from the project. Staff completing the assessment in good standing are then certified to conduct NSDUH interviews.

10.3 Evaluation of Training Programs

At the close of each interviewer training session, you are asked to complete an evaluation form to assess the training program and materials, the trainers, and the training facilities. Your feedback on the effectiveness of the NSDUH interviewer training program is an important part of letting us know whether or not the training program was thorough and effective. Your evaluation is also used to improve preparations for future NSDUH training sessions.

For graduates of New-to-Project training, an additional evaluation is completed after several months of actual work experience to see how well they feel the training program prepared them for their work on NSDUH.

Exhibit 10.1 Steps to Maximize Data Quality

Exhibit 10.1 Steps to Maximize Data Quality (Continued)

10.4 Screening Edits

Due to the computerized and immediate nature of electronic screening, be extra cautious as you enter the information to be sure you are entering the correct data. When you complete each roster, review the entries with the screening respondent, checking that everything is entered correctly and completely. If there are any omissions or incomplete items, ask the respondent for the missing information.

Once received at RTI, all screening data are edited by machine (computer). Complex programs evaluate the data and check to be sure you are conducting the screening correctly. Feedback on any problems noted during the process is given to your FS to share with you.

10.5 Interviewing Edits

Since the majority of the information you collect for NSDUH is entered directly into your laptop computer, the traditional editing of completed hardcopy materials is eliminated. Edits such as checking skip patterns for missing data are performed by the computer software during the interview.

All CAI interview files transmitted to RTI are reviewed by computer. This computer edit of completed interviews is more complex than the edits performed on your laptop in the field. As with screening edits, complex programs evaluate the interview data to ensure you are administering the interview correctly. Again, FSs will share the feedback they receive with you.

10.6 Screening/Interview Observations

Your FS and/or your RS may want to observe your work, so you may be accompanied by an observer as you conduct screenings and/or interviews at some time. If so, your supervisor will notify you. When you arrive at an SDU, introduce the observer and explain that he/she wants to determine how well the data collection process works. If the respondent refuses to allow the observation, you should proceed with the screening and/or interviewing process without the observer.

10.7 Verification

In order to provide continuing feedback to you and to our client, SAMHSA, on the quality and accuracy of cases, your work is subject to an intense verification process. RTI is contractually bound to verify the quality and accuracy of all field work conducted on the NSDUH. We must be certain that all procedures are being implemented properly. Inaccurate or substandard work is very costly to the project, so we must determine if and why it is occurring and take corrective actions. Falsified work is unacceptable under any circumstances, and the discovery of any falsified work will result in immediate termination from the project and from any future work with RTI.

All finalized cases are subject to verification.

10.7.1 Screening Verifications

Verification contacts are made with a random sample of all completed screenings reported as:

• vacant

• not a primary residence

• not a dwelling unit

• residents there less than ½ of the quarter

• no eligible SDU members

• HU/GQU contains ONLY military personnel

• no one selected for interview.

As discussed previously, enter the contact person's first name and telephone number at the conclusion of these types of screenings. For the first three types listed, the contact person is typically a neighbor; for the others, it is usually the screening respondent. When asking for this information, emphasize it is for verification purposes only; we want to verify the quality of our work, not their information.

10.7.2 Interview Verifications

As discussed earlier, the respondent completes the top portion of a prepared Verification Form at the close of each interview. For interviews with a youth respondent, the parent or guardian should complete the form, unless he/she is not available to do so. The completed forms are mailed to RTI.

A random sample of each interviewer's finalized cases are verified with the respondent or, for youth respondents aged 12-17, permission is first obtained from a parent prior to contacting the youth for the actual verification information.

10.7.3 The Verification Process

Screening and interview verifications are conducted by RTI's Telephone Survey Unit (TSU). Respondents without telephones are contacted by mail. As needed, supervisors may also conduct in-person field verifications. During the verification contacts it is inevitable that some discrepancies will arise between a respondent's and the interviewer's account of the same situation. Resolution procedures are then implemented to attempt to settle the issue. Information on any unresolved discrepancies will be given to your FS and your RS to determine the next steps to take.

The verification process is a tool for the NSDUH project staff to monitor not only your performance as an interviewer but also their performance as project trainers and managers. The information gathered helps the NSDUH staff ensure that the data being collected are of the highest possible quality. You will receive prompt feedback on your performance and will be notified immediately should any problems arise.

REVIEW OF CHAPTER 10
Quality Control

To assist your learning process, read the following summary of key points in this chapter. Following the summary are several questions to ask yourself. These questions will help you identify those areas you understand, and pinpoint the areas where you would benefit from re-reading a particular section.

SUMMARY

• Quality is the backbone of this study and we have strict quality control measures in place at every stage of the data collection process.

• Screenings need to be edited as you are completing them—every piece of information which you collect is used in our analyses. Therefore, every roster should be complete and accurate.

• While administering interviews, the CAI program is continually editing your work for missing data and proper skip patterns and provides immediate feedback if mistakes are found.

• RTI is contractually bound to verify the quality of our field work. All finalized cases, be they screenings or interviews, are subject to verification.

QUESTIONS TO ASK YOURSELF

1. If falsified work is discovered, what happens to the FI?

2. What verification information from the respondent is entered into the Newton for screenings where no one is selected for an interview?

3. When interviewing a youth respondent, who should complete the Verification Form?

This page was last updated on May 16, 2008.

SAMHSA, an agency in the Department of Health and Human Services, is the Federal Government's lead agency for improving the quality and availability of substance abuse prevention, addiction treatment, and mental health services in the United States.