1999 National Household Survey on Drug Abuse Data Collection Report

8. QUALITY CONTROL

While every step was designed to help collect the highest quality data possible, the 1999 NHSDA included several quality control programs which are described in this chapter.

Each field interviewer had at least one regularly scheduled weekly telephone conference with his/her Field Supervisor. During this call, the FI reported progress made toward completing the work; reviewed Production, Time, and Expense (PT&E) information for the week; discussed field problems; and asked any questions that had emerged during the week. The FS then provided feedback on the progress and quality of work and offered solutions to problems or questions encountered.

Regular weekly telephone conferences also were held between the Regional Supervisor and each of the FSs in his/her territory. FI production and performance were discussed during these conferences, as were budget considerations and any problems that were occurring.

Beginning at training, FI and FS performance was monitored closely and consistently throughout the field period. Training classrooms were small enough to observe and evaluate each FI's individual performance and comprehension. The classroom trainers worked together to evaluate FIs on a daily basis. Reports of struggling FIs were given to the site leader daily to help identify problems and develop resolution plans. These evaluations also ensured that those FIs who were struggling but willing and capable of doing the work would receive the necessary help after training to interview successfully on the NHSDA. FIs needing extra help at the close of the training session were identified, and further training was provided by the FS. These FIs also were monitored more closely as they began their assignments.

At the end of every quarter of data collection, each FS evaluated the FIs in his/her region to decide how to allocate bonus funds and whether to recommend any merit-based pay raises. FSs considered all the facets of being a "good FI," including production, response rates, costs, timeliness, attitude, commitment, attention to details, data quality errors, and willingness to take on additional work (particularly to work on hard refusals). To decide how to divide bonus funds, the FS ranked each FI. Additionally, pay raises were not necessarily related to bonus money; an FI might not receive a bonus but could still be eligible for a raise. For both bonuses and pay raises, RSs and RDs reviewed the FS's decisions.

At the end of the calendar year, each FS used a standard RTI multiple-choice form to generate an interim evaluation of FIs who were active on the NHSDA. The FS used this same form to provide a final evaluation of FIs who "attrited." Completed evaluations were added to the interviewer's personal data file at RTI. The FS generally completed this form without RS or RD input.

Every month NHSDA management personnel received from RTI a listing of those field interviewers who had voluntarily chosen to leave the project (those terminated did not appear on this list). The listed FIs were contacted and a short questionnaire was administered to determine the main and secondary reasons they left the project. These data were then keyed and used to produce a quarterly report summarizing the reasons.

In an effort to evaluate FS performance from the perspective of the field interviewers being supervised, a telephone survey was designed using a Touchtone Data Entry (TDE) system. During the period from November 15 to December 12, 1999, FIs called a dedicated toll-free RTI line and entered their responses using the touchpad on their telephones. They were asked to rate their FSs in categories such as availability, timeliness, consistent schedule, ability to resolve field problems, experience, efforts to create a supportive work environment, skill at scheduling assignments, and administrative and organizational skills. This system allowed FIs to evaluate their FSs' strengths and weaknesses quickly and confidentially. The results were then aggregated and reported to management. Approximately 79 percent of the FIs working during November, 1999 responded to the TDE questionnaire.

The FIs did not review or edit any of the respondent's answer sheets. The FIs were only responsible for checking through the questionnaire booklet at the time of the interview to ensure that they—as the interviewer—had completed it correctly and completely. Once thisedit was finished, they placed these materials, along with the respondent's answer sheets, in the RTI business reply envelope for mailing to RTI (or to their FS, if required).

Either special editors at RTI or the FS did an in-depth edit of the first few completed cases for each FI, checking for correct skip patterns, missing data, and legibility. The FI was given feedback and re-trained if necessary.

Each FS was equipped with a laptop computer and given access to the NHSDA Web-based Case Management System. FIs transmitted screening data daily from the Newton, including record of calls data, verification information for non-interview cases, added DUs, address updates, and cost data. When the Newton screening data was transmitted to RTI, it was checked by the control system's defined consistency checks, then was posted to the CMS for monitoring purposes. The completed interview data was transmitted to RTI by FIs from their laptop computers and checked against screening data to ensure each completed case was received and that the correct respondent was interviewed.

The FS System on the CMS included the following data quality functions:

• Daily and Weekly Reports with access to archived reports (for comparison data).

• An interactive data information page for monitoring production.

• An interactive record of calls page for monitoring FI work patterns.

• Verification data.

The Data Quality Report displayed various data quality issues and allowed the FS to provide specific feedback to FIs who were experiencing problems. The report included missing data items on Verification Forms and procedural errors such as Case ID or Verification ID problems. The report also included a list of cases that could not be used due to the FI interviewing the wrong household member or administering the wrong questionnaire version (CAI vs. PAPI).

The Missing Screening Data Report displayed the screening data, listed by FI, that was missing for specific Case IDs. FSs used this report to monitor the quality of the screening data that each FI collected. The data on this report represented information that the respondent refused to provide or indicated areas where the FI either made errors or may have been taking short-cuts. FSs monitored specific problems and trends and were able to provide immediate feedback and re-train FIs as necessary.

FSs used the Overdue Case Report to account for completed interviews that should have arrived via mail (PAPI) or transmission (CAI) over a period of time. PAPI interviews were considered overdue if not received within 10 days of the date of interview (as reported by the Newton record of calls data). CAI interviews were considered overdue if not transmitted within three days of the date of interview.

Cases displayed on this report were investigated to ensure the completed interview was mailed or transmitted or that the correct Case ID was used and reported as a completed interview. FSs and programming staff worked to resolve any pending issues with overdue cases.

The Length of Interview Report listed the completed interviews (CAI only) that were either finished in a relatively short or extremely long amount of time. The times were derived from the CAI interview file (total time and timing of specific sections) so that FSs could monitor possible problem situations (such as short-cutting or problems with the laptop that might cause the time-frame to be strange).

The Case Data Information portion of the FS system provided all FI production data and allowed the FS to interact with the data and view it in special ways. The type of cases the FS viewed was determined by the drop-down items selected. Each of the following items were available to select (single or multiple items), after which a data table containing all of these items (for the subset of cases) displayed:

• Case ID

• Type of case (Screening, Interview A, or Interview B)

• Status and Result Code (record of calls event codes)

• Result Code Date (date of the record of calls code)

• # Calls (total number of contacts at the household)

• FS Note (any notation the FS attaches to the case)

• Questionnaire Rec'd (date the PAPI case was received or the CAI case was transmitted)

• Verification Status

• FI ID (FI assigned to the case)

• Address of the SDU.

There were special features within this function that displayed additional data:

• Overdue cases (highlighted in yellow)

• Added DUs (highlighted in green)

• Click on CaseID to view entire record of calls

• Click on Refusal Code to view entire refusal report

• Click on Verification Status to view verification history of case

• Click on FI ID for Production, Time and Expense data

• Click on address to view map of the area.

The data provided in this table allowed the FS to evaluate many aspects of the FI's work.

The Filter Record of Calls allowed the FS to view the FI's record of calls events by filtering on the following items:

• Case ID

• Data Type (Screening, Interview A, or Interview B)

• Result Code

• Day of week (All days, Mon-Sun)

• Time periods of day (6am-Noon, Noon-4pm, 4pm-12am, 12am-6am)

• Date (before a date, after a date, a specific date or between two dates)

• FI.

The FS could analyze the FI's work pattern and spot instances where an FI might enter "false" results.

A Data Quality Team was formed to centralize the identification, resolution, and distribution of information to field staff concerning data quality and verification issues. The Data Quality Team was made up of one Coordinator and three Data Quality Specialists. The Data Quality Coordinator supervised the three Data Quality Specialists as they monitored the data quality of specific regional areas. The Data Quality Coordinator also interacted with supervisors in RTI's Telephone Survey Unit (for verification issues), and data receipt and data preparation units to oversee data quality issues.

In order to verify the quality and accuracy of the FIs' work, a complex verification procedure was implemented. This involved the selection and verification, predominately by telephone but sometimes by mail, of at least 15 percent of final interview cases, and at least 5 percent of final non-interview screening cases. Additional work could also be selected for the verification process—field management staff could elect to have 100 percent of the FI's work verified, or they could select an individual case or a group of specific cases to be verified beyond what was randomly selected. These verification contacts were made with the actual interview respondent.

Verification information for completed interviews was obtained from the Verification Form completed by each interview respondent (see Exhibit 8.1). For the final non-interview screening codes of 10 (vacant), 13 (not primary residence), 18 (not a dwelling unit), 22 (dwelling unit contains only military personnel), 26 (not eligible for the quarter), and 30 (no one selected for interview), the contact information was recorded immediately in the Newton at the time the case was finalized. For codes 10, 13 and 18, the contact was made with a knowledgeable person, such as a real estate agent, property manager, or neighbor. For codes 22, 26, and 30, the verification was completed most often with the screening respondent.

The telephone verification was conducted by specially trained staff in RTI's Telephone Survey Unit (TSU). Spanish translations of all materials were available for verifications with Spanish-speaking respondents. Again, most of the selected code 70s, and all of the selected codes 10, 13, 18, 22, 26, and 30, were verified by TSU. The NHSDA telephone verification script used depended on the final status code of the case (see Appendix E).

For those selected code 70s that did not have a telephone number on the Verification Form but did have an address, verification by mail was attempted. The mail verification letter (see Exhibit 8.2) was sent (CAI or PAPI version) to the respondent to complete and return by mail to RTI. The completed verification letters were keyed, and the results were displayed in the CMS and on the Verification Reports.

TSU Verification had two stages. For most problems found during the initial call, a follow-up call was made to confirm the problem. That follow-up call was made the following week by the Verification Team, an elite group of telephone interviewers who were trained on all project procedures and protocols.

During the follow-up call, the Verification Team member conversed with the respondent (as opposed to following a script) in an attempt to confirm or resolve the identified problem(s). The Verification Team member had the problem (or list of problems) identified for that case during the TSU verification interview as well as a bulleted list of items (a subset of items from the TSU questionnaire) that must be covered during their conversation with the respondent. The result of the call was either a confirmation that the problem (or additional procedural problems) occurred during the screening or interview or a resolution of the problem by clarifying the issues with the respondent. If any other important information was obtained during the follow-up call, the Data Quality Team distributed this information to the FS and RS.

Table 8.1 provides a summary of the results of the in-house TSU verifications for CAI interviews, while Table 8.2 provides the same results for PAPI cases. Corresponding information for the TSU verification of non-interview codes 10, 13, 18, 22, 26, and 30 is also displayed in Table 8.1 for CAI and Table 8.2 for PAPI.

In addition to the verification procedures conducted on completed work received in-house, additional steps were taken in the field to ensure complete and accurate collection of data. This field verification was generally initiated after one of three circumstances occurred:

1. an FI had an unusually large number of in-house verifications "fail";

2. the FI exhibited unusual or suspicious behavior; or

3. an FI reported numerous cases as being completed that failed to arrive at RTI within a reasonable length of time.

The Data Quality Team worked with the FS and RS to select the cases to be field verified. These finalized cases were transmitted to the Field Verifier's Newton (either the FS or another FI conducting the field verification) so that the screening data could be verified. The Field Verifier returned to the SDUs that were assigned and queried the respondent in an effort to determine whether or not proper contact had been made by the FI in question. The Field Verifier also verified the screening information. If an interview had been completed, the Field Verifier confirmed some of the demographic data from the interview with the respondent. The Field Verifier also reviewed some protocol issues with the respondent to ensure the FI had followed protocol and acted in a professional manner. Results of the field verification were reported to the Data Quality Team and the FS. If the Field Verifier found the work to be invalid, he or she reworked the case.

In general, the need for such in-field verification was limited, but it did occur. In the 1999 NHSDA, a total of 2,497 cases were selected for Field Verification. This process led to the identification and termination of FIs who were determined to have submitted fraudulent work. All their work was verified and reworked as necessary. A total of 360 invalid interviews and 650 invalid screenings involving 34 FIs were identified via in-person field verification. All 34 FIs were terminated.

The Verification Status on the Case Data Information link on the CMS allowed project staff to view the verification status of each case and monitor trends across status codes or areas. The following Verification Status codes were used to monitor the verification at the case level:

NF: No Form (Code 70s)

NP: No Phone

RE: Refusal—not selected

NS: Eligible, but not randomly selected for verification

ST: Selected for TSU (Telephone) Verification

SF: Selected for Field Verification

SM: Selected for Mail Verification (Code 70s without phone numbers)

OK: Completed Okay

UC: Finalized—Unable to Contact

UN: Finalized—Unresolveable

SS: Completed—Some shortcuts

IR: Completed—Invalid, then reworked

IW: Completed—Invalid, not reworked

Since verification selection was random, it helped to see which cases had been selected. If project staff wanted additional cases to be selected for verification, they used this link on the CMS.

The Verification Short FI Level Report provided a snapshot of the problems identified during TSU Verification and Mail Verification. Page one provided a summary of data for a subset of codes: 10, 13, 18, 26, and 30. Displayed were the number of cases of these status codes that had no form (code 70 only), no phone, refused, percent of cases with no form/phone (once greater than or equal to 30 percent), percent of cases refused (once greater than or equal to 30 percent), count of other ineligible, count of eligible, count of selected for TSU, and count of selected for mail. From this data, supervisors could see if an FI had a high percentage of cases with no phones, no forms, refused, and how many have been sent to Mail Verification (which is not as successful as Telephone Verification in obtaining a response).

More specific details of the problems displayed on page one were contained on page two of the report. The second page displayed each problem identified during TSU and Mail Verification. A case could have multiple problems, so all problems for all cases were displayed here to track trends related to possible shortcutting. There were 45 Problem Codes divided into four tables by Screening and Interview Result Code (Exhibit 8.3).

All editing and coding work was subjected to a quality control check by the editing supervisor and a senior editor. Initially, for each task performed by an editor, a 100 percent quality control check was done on his/her work. Based on the number of errors detected and the number of steps involved in the process being checked, an error rate was calculated. Any detected errors were reviewed with the editor and retraining was conducted as needed. When an editor's error rate fell below 1 percent, all further work done by that editor on that procedure was sampled for a quality control check.

Once an editor passed the 100 percent verification phase, a sample of 10 percent of the editor's work on a given task was checked. For non-interview work processed in batches, two forms were pulled at random from the batches. For interview batches, one questionnaire and answer sheet set was pulled at random. The cases selected were re-edited, error rates were calculated, and all errors were reviewed with the editor. If an editor's error rate ever rose to one percent or above, the entire batch was checked.

The CAI data was in electronic form so no manual editing was needed. The CAI data was run through computerized consistency checks similar to those run on the PAPI data once the PAPI data were in electronic form.

Verification Results for CAI Interviews
1999 NHSDA

Code 70s					Non-interview Screening Codes
	Overall Result	Verified by TSU	Verified by Mail	Code 50 (no problem)	Verified by TSU	Code 50 (no problem)
Q1	2,052	1,897	155	1,665	4,054	3,271
Q2	4,433	3,968	465	1,928	6,940	3,323
Q3	6,806	6,006	800	3,631	8,602	5,830
Q4	6,436	5,956	480	3,851	6,711	4,944
Total	19,727	17,827	1,900	11,075	26,307	17,368

Table 8.2

Verification Results for PAPI Interviews
1999 NHSDA

Code 70s					Non-interview Screening Codes
	Overall Result	Verified by TSU	Verified by Mail	Code 50 (no problem)	Verified by TSU	Code 50 (no problem)
Q1	408	378	30	277	1,581	1,250
Q2	1,330	1,161	169	528	2,058	974
Q3	706	563	143	305	679	450
Q4	663	598	65	356	707	505
Total	3,107	2,700	407	1,466	5,025	3,179

Exhibit 8.1

Verification Form

VERSION EN ESPANOL AL OTRO LADO

Exhibit 8.1 (Continued)

Verification Form

ENGLISH VERSION ON OTHER SIDE

CAI and PAPI Mail Verification Letters

CAI

Exhibit 8.2 (Continued)

CAI and PAPI Mail Verification Letters

PAPI

Short FI Level Verification Report Problem Codes

Problems included both CAI and PAPI unless otherwise specified.

Code 70 Problems

1 Incorrect phone number for address

2 Correct address/phone but R unknown

3 Roster Incorrect

4 Correct address/phone but no adult to give permission to speak with teen R

5 Not contacted by FI

6 Did not complete interview

7 Interview completed some other way (not in person or by phone)

8 Interview completed by phone

9 Option not offered to enter answers in computer (CAI)

10 Tutorial not completed (CAI)

11 No headphone option (CAI)

12 FI unable to assist when R had difficulties with computer (CAI)

13 Option not offered to complete on answer sheets (PAPI)

14 1-5 Answer sheets reported (PAPI)

15 Answer sheets handed in a stack (PAPI)

16 No confidential red tape (PAPI)

17 Less than 30 minutes (CAI or PAPI)

18 Less than 30 minutes and No option given (see #9 or #13 above)

19 Less than 30 minutes and answer sheets handed in a stack (PAPI)

20 Less than 30 minutes and only 1-5 answer sheets reported (PAPI)

21 FI Not Professional (CAI or PAPI)

Code 30 Problems

22 Incorrect phone number for the SDU

23 Correct Roster/Address, but SR Unknown

24 Does not remember FI - Correct Address but Roster Incorrect

25 Does not remember FI - Wrong Address but Correct Roster

26 Does not remember FI - Wrong Address and Incorrect/Refused Roster

27 Remembers FI - Correct Address but Roster Incorrect

28 Remembers FI - Wrong Address but Correct Roster

29 Remembers FI - Wrong Address and Incorrect Roster

30 Telephone Screening or some other way (not in person)

31 FI Not Professional

Exhibit 8.3 (Continued)

Short FI Level Verification Report Problem Codes

Code 22 Problems

32 No known contact with FI

33 Speaking to SR, not familiar with address

34 All HH members not on active military duty

35 Contact some other way (not in person or telephone)

36 Telephone screening

37 FI Not Professional

Code 10, 13, 18, 26 Problems

38 No one familiar with the address

39 Speaking to SR and no FI contact

40 Code 10 - reported as not vacant at time of screening

41 Code 13 - reported as primary place of residence for the quarter

42 Code 18 - reported as a DU

43 Code 26 - reported by resident that someone did live there for most of the quarter

44 Code 26 - reported by non-resident that someone did live there for most of the quarter

45 FI Not Professional

This page was last updated on June 03, 2008.

SAMHSA, an agency in the Department of Health and Human Services, is the Federal Government's lead agency for improving the quality and availability of substance abuse prevention, addiction treatment, and mental health services in the United States.