1999 National Household Survey on Drug Abuse Data Collection Report

4. PREPARATION OF SURVEY MATERIALS

RTI staff preparing survey materials for 1999 worked hard to test and perfect both the CAI interview program and the Newton electronic screening program. In addition to the electronic materials, the paper version of the 1998 questionnaire also was updated for use in the 1999 PAPI sample. All other manuals and interview materials were re-examined and updated for the conversion to electronic data collection. With the tremendous staffing increase, the preparation for training required meticulous planning.

4.1 Electronic Screening

Exploring the possibilities and developing the Newton program for electronic screening began with Field Test No. 2 (FT-2) during the fall of 1997. The FT-2 screening program started with a brief introduction script, moved directly to the first question to be asked of the screening respondent, then continued with screening up through the automated respondent selection process. The August Field Test, conducted in August of 1998, expanded beyond the FT-2 screening basics to include more detailed introductory scripts and allow for complete capturing of Record of Calls data. Another added feature allowed RTI-Certified bilingual interviewers to switch the displayed question text to Spanish for screenings conducted with Spanish-speaking respondents. Both FT-2 and the August Field Test excluded group quarters from the sample and did not ask the interviewers to record missed dwelling units.

Using August Field Test staff reactions and test results, the Newton program was modified and further enhanced for the 1999 NHSDA. Additions included programming to allow field staff to conduct screenings at Group Quarters Units (which had been done in prior years on blue screening forms) with minor introduction alterations and question wording differences between the housing unit and group quarters unit versions appearing on the Newton automatically. Also included was the ability to add missed housing units and to conduct screenings at those units added to the sample (as interviewers in prior years had done on yellow screening forms).

Several screening questions on the 1998 paper screening form were modified slightly for the 1999 electronic screener:

• The enumeration questions (which establish the number of household residents during the quarter and for housing units, identify the householder) had minor wording changes, such as altering the verb tense from "Did you" to "Have you."



• The term "Latino" was included in the Hispanic origin question.



• The Race question had a number of substantial changes. Most significant was the ability of respondents to choose multiple race categories. Also, the defined answer choices were reordered and included additional terms. The texts of the two race questions (asked of a screening respondent) are shown below:

1998
"Are you black, white, American Indian, Alaskan Native, or Asian or Pacific Islander?" (CIRCLE ONE RESPONSE)



1999
"Are you White, Black or African-American, American Indian or Alaskan Native, Native Hawaiian or other Pacific Islander, or Asian?" (MARK ALL THAT APPLY)



• Other minor wording differences were made as required by or made possible by the use of the handheld computer for electronic screening.

Management tools were also added to the 1999 Newton screening program to enhance the ability of supervisors to check the work and accuracy of the interviewers. For example, after FIs transmitted and a daily system update was conducted, all updated or new call records were available to supervisors on the Web-based Case Management System (CMS). The following day when speaking with an FI, the FS could view all the call records and know exactly when the FI had visited the unit. Verification data was also entered in the Newton screening program for quality control checks (see Chapter 8).

4.2 Questionnaire Development

4.2.1 CAI Instrument

Alterations to the August Field Test program provided the programming for the 1999 computer instrument, which included a straight-forward CAI Manager system as well. The following changes were made to the content of the CAI instrument from the previous year's paper questionnaire:

• Included new Tobacco module;



• Included additional categories for race/ethnicity questions and allowed for multiple responses;



• Included newly available medications in the "pills" sections and modified the list of substances known to be misused;



• Moved several questions about treatment and counseling from the Special Topics module to the Treatment module;



• Deleted the Drug Experiences module;



• Deleted a number of questions from the Youth Experiences and Social Environment modules;



• Included questions to determine national origin and length of residence more accurately;



• Modified education and employment questions; and



• Deleted the "Access to Care" questions and a few other questions in the demographic section.

For the audio computer-assisted self-interviewing (ACASI) portion of the interview, the computer program required digital recordings (WAV files) of someone reading all the questions and answer choices, including question fills. These WAV files were prepared and tested with the CAI instrument.

Materials used during the actual interview, including the Reference Date Calendar and the Showcard Booklet, also were updated. The Pill Cards were revised to reflect the changes in the pills sections.

4.2.2 PAPI Instrument

With the exception of the Tobacco module, the changes between the 1998 and 1999 instruments detailed above were also made to the hardcopy questionnaires prepared for the 1999 PAPI sample. Dates and other year-specific items also were updated. Other materials specifically for PAPI interviews, such as the PAPI Showcard Booklet and a separate version of the Question and Answer Brochure, were also updated.

4.2.3 Spanish Translations

Both the CAI instrument and the PAPI questionnaire also were prepared in Spanish. Based on the 1998 translation, all changes for 1999 were translated and incorporated. While seemingly straight forward, tremendous effort was needed for the conversion to CAI. All Spanish computer programming had to be prepared. The recording of the Spanish WAV files (to allow respondents to listen to the ACASI sections in Spanish if necessary) was time consuming; 'proper' pronunciations that could be understood by all Spanish-speaking respondents, regardless of various regional versions of the Spanish language, had to be ensured.

4.3 Manuals/Miscellaneous Materials Development

4.3.1 Manuals

With the large increase in both the number of staff and the layers of management, the need for accurate, detailed manuals for both training and reference was great.

• Field Interviewer Manual: All field staff (from interviewers to the National Field Director) received a Field Interviewer Manual detailing all aspects of an interviewer's work requirements on the 1999 NHSDA. This manual was sent to all FIs for reading prior to the start of classroom training, was utilized throughout the training, and was a ready reference when questions arose during field work.



• Field Interviewer Computer Manual: This companion FI manual provided details about hardware use and care issues for both the Newton and the Gateway laptop computer, instructions for using the programs on each computer, transmission steps, and a troubleshooting guide to assist staff encountering technical difficulties. This computer manual was included with—but bound separately from—the FI Manual, so FIs could easily include it in their computer carrying case as a quick reference while working.



• PAPI Handbook: Also bound separately, this handbook provided PAPI interviewers with instructions for completing the 1999 NHSDA using the paper questionnaire. Only items which differed from the CAI version were discussed in this handbook.



• Field Supervisor Manual: This detailed manual for FSs included instructions and tips for recruiting field staff and managing the Counting and Listing effort and Screening and Interviewing work. Strategies for managing staff using information on the Web-based CMS were also presented, as were administrative issues for both the FSs and their staff.



• Field Supervisor Computer Manual: Explanations of the equipment provided for FSs (computer, printer, and pager) were included in this separate volume, as were instructions on using the various software tools (Windows/MS Word/MS Excel, Microsoft Network (MSN) for e-mail, Fed-Ex tracking). Detailed instructions on how to use the intricate and extensively informative Web-based CMS were provided for instruction and reference.



• Regional Supervisor Manual: This manual provided specific guidelines for RSs on supervising the FSs in their region and on reporting requirements to the Regional Directors. Separate chapters provided instructions for managing the various stages of NHSDA, including FI Recruitment, Counting and Listing, and Screening and Interviewing.



• Counting and Listing Manual: The standard RTI Counting and Listing Manual was given to all listers and management staff working on that phase of the NHSDA. Also provided was the RTI Counting and Listing Supplement for Subsegmenting.



• Guide to Controlled Access Situations: This manual, developed and utilized for previous NHSDAs, was given to all management staff.



• NHSDA Guide Book: With the phenomenal growth of staff (including headquarters staff), this guidebook was developed specifically for all project management and headquarters staff. It provided details about issues such as chain of command, use of the project network drive, and whom to include on various e-mails.

4.3.2 Miscellaneous Materials

Based on the 1998 versions, the following materials were updated to reflect the conversion to CAI and the increased sample size for 1999 or simply to update them for the new year:

• Lead Letter to all SDUs
• Statement of Confidentiality
• Question and Answer Brochure
• NHSDA Highlights
• Newspaper Articles

The following materials remained virtually unchanged from 1998 for use in 1999:

• RTI Fact Sheet
• "Sorry I missed you" cards
• Appointment cards

Refusal conversion letters—sent in previous years by field supervisors to reluctant respondents—were converted to an automated system for centralized mailing from RTI. When an interviewer first entered a refusal code for a case in the Newton and then transmitted the data to RTI's computer system, the interviewer's FS could see the letter on the Web-based CMS. After talking to the interviewer or simply relying on the specific code and notes entered by the FI, the FS could then either release the appropriate letter to be printed and sent or cancel the need for a letter. In attempting to cover most refusal situations, there were 17 different conversion letters, available in both English and Spanish.

4.4 Preparation for Interviewer Training and Field Data Collection

This section reviews the main steps necessary to prepare for training and distribute interviewing materials to all field staff completing training.

4.4.1 Home Study Packages

4.4.1.1 CAI Home Study

Prior to training, each FI hired for screening/interviewing work was sent a Home Study package containing:

• A 1999 Field Interviewer Manual
• A 1999 Field Interviewer Computer Manual
• A cover memorandum from the National Field Director
• Home study exercises
• A Computer Basics Introductory Video
• A Gateway laptop computer and all accessory equipment
• A Computer Equipment ID List
• An Equipment and Receipt Form (EARF) and an envelope.

Trainees were instructed to:

• read both manuals;
• complete the home study exercises;
• watch the videotape explaining how to unpack and set up the equipment;
• complete and mail the EARF to RTI;
• unpack and setup the Gateway;
• complete the tutorial on the Gateway; and
• listen to the audio Drug Pronunciation Guide on the Gateway.

The computer tutorial on the Gateway was the same practice session developed to familiarize survey respondents with using the computer. Bilingual interviewers were also instructed to listen to the Spanish version of the audio Drug Pronunciation Guide.

Completed exercises and all computer equipment were to be brought to training. Exercises were collected at registration, graded, and returned to the appropriate training team. Appendix A contains the CAI home study memorandum, while Appendix B contains the home study exercises.

4.4.1.2 PAPI/Bilingual Home Study

Additional home study materials were sent to FIs hired to work in areas with a PAPI sample and to all bilingual FIs (who would also attend PAPI training). The following items were included in this packet:

• A memorandum from the National Field Director



• A PAPI Handbook



• Home study exercises



• A Questionnaire/Answer Sheet set



• A Spanish version of the Questionnaire/Answer Sheet set (for RTI-Certified bilingual FIs)

These additional home study exercises were to be completed and brought to training. Appendix C contains the PAPI home study memorandum, while Appendix D contains the home study exercises.

4.4.2 Handling of Survey Supplies

To prepare for training and data collection a master list of needed supplies was developed. Using this list, all supplies were developed, ordered (if necessary), and stored in preparation for data collection activities throughout the survey year.

4.4.2.1 Printed Materials Related to Training

While using computers greatly reduced the production of printed questionnaires used throughout the year, many paper forms were still necessary, particularly for training. A detailed, near-verbatim guide was prepared for each member of the team of trainers assigned to each room. Along with the two-volume training guide were numerous printed materials, including:

• Data Collection Agreements for all trainees to signify they agreed to follow procedures and maintain confidentiality.



• A Training Workbook that contained necessary exercises, printed examples, screening scripts, and additional instructions.



• A Training Segment Kit with example listing and locating materials for the practice segment used in training.



• Mock Scripts separately bound for four different CAI paired mocks and including the screening mocks for the case.



• PAPI Materials including questionnaire booklets, answer sheet sets, and mock scripts for the PAPI training.



• Verification Forms printed in padded form and specifically for the various training cases.



• Reference Data Calendars for use during the practice interviews.



• Showcard Booklets and Pillcards for training and use during subsequent field work.



• Supplies to be used during the course of training, including the Lead Letter, the Statement of Confidentiality, and various tools used during obtaining participation, such as the laminated Federal Certificate of Confidentiality, RTI Fact Sheet, NHSDA Newspaper Articles handout, NHSDA Question and Answer brochure, Sorry I Missed You cards, NHSDA Highlights, and "Preliminary Estimates."

4.4.2.2 Training Videotapes

A videotape was developed that contained multiple segments for use throughout the course of FI training. The use of the videotape provided controlled, standardized, visual representations of the various tasks assigned to S/I interviewers. In addition to the dramatizations of these tasks, a history of the NHSDA by Project Director Tom Virag was included, as was the widely enjoyed, award-winning segment "It's a Big Job"¹ that introduced the management staff and many of the RTI NHSDA departments performing specific tasks (such as Technical Support and the Sampling Department).

4.4.2.3 Assignment Materials

Trainees performing well after the first days of training were given assignment materials for the cases assigned to them. The assignment materials consisted only of the Segment Materials Envelope (the envelope with the various maps and listing sheets for a segment). They did not receive lead letter packets since these were mailed from RTI in an effort to reduce both cost and the burden on the FIs, as well as to centralize the operation.

Electronic screening with the Newton also replaced the need for screening forms and control forms, and interviewers simply had to transmit at the end of training to pick up their assigned cases. Trainees struggling during training either received a small, partial assignment or received no assignment until the adequate completion of further training. Any unassigned or partial segment kits were sent to the FSs for later assignment.

4.4.2.4 Bulk Supplies

Bulk supplies were packed at RTI and shipped via Federal Express directly to the homes of staff completing training successfully. During the year, periodic resupply requests were sent by e-mail from FSs to the Field Distribution Center. Requested items were sent directly to the FIs needing supplies.

4.5 Website Development

Using the power of the Internet to enhance communication, staff developed two Websites.

4.5.1 Project Case Management System

The project's Web-based CMS revolutionized the way supervisors managed their staff by enhancing the ability of all levels of management to make informed decisions based on current field conditions. Each night, data was transmitted to RTI from the interviewers' Newtons and Gateway laptops for inclusion in the CMS. The next morning, each supervisor and manager had access to the results of the previous day's work and its effect on the totals for that quarter.

Besides case work reports, the Website also contained many helpful tools, such as logs to enter new recruits, links to other pertinent sites, project calendars, and other administrative tools.

Access to this secure Website was tightly controlled with system wide security provided through secure links to the network from each user's computer. Additionally, several levels of passwords were required to enter the system. Supervisors had access limited to the information needed to manage their areas (e.g., an FS could only see data about his/her staff, while an RS viewed details about all cases and staff in his/her region).

4.5.2 NHSDA Respondent Website

For computer savvy respondents, an informative public NHSDA Website was developed. Visitors to the site could access a variety of topics such as project description, confidentiality, and frequently-asked questions. Brief information was included about both SAMHSA and RTI, with links to the Websites of both organizations.

4.6 Equipment Acquisition

Significant effort was expended in researching, acquiring, and storing the necessary equipment for the NHSDA. Interviewers needed Newtons and Gateway laptops, and protective cases for the Newton had to be designed and manufactured by a contractor. Supervisors and managers needed those plus management laptops, printers, faxes, and pagers. Sufficient Toshiba Multimedia Laptop Projectors were required to supply all the training sites during the large January interviewer training sessions. An extensive inventory system was developed to monitor the disbursement and location of all equipment. In addition, technical assistance to the users of the equipment was an important and necessary task.

4.7 Problems Encountered

Due to heroic efforts by dedicated staff, the above preparations were completed so that training and the start of data collection began as scheduled.

• Volume: The sheer increase in magnitude caused numerous problems. Instead of 15 FSs, there were 81. Instead of 2,670 segments, there were 7,200. Given the increase in the volume of work, the traditional way of performing each task had to be re-examined. All such changes had to be tested and documented. Additionally, the large quantities of materials required meant allowing longer periods for printing, handling, and distribution.



• Inexperienced staff: The tremendous increase in RTI staff meant the number of experienced writers, coordinators, and other development staff was inadequate to complete or supervise the work required. While mentoring new employees is important for their growth and experience, having to do so under extremely tight deadlines was difficult.



• Schedule: Even without the sample size increase, the large number of major changes in the NHSDA procedures would have been challenging to implement in a comfortable time frame. From the moment of the contract award, the proposed schedule was already behind. The implemented schedule had many tight deadlines RTI worked very hard to meet. The SAMHSA staff, also burdened, were not always able to review items and provide comments in the time allowed. Because of the overall tight schedule, there was no cushion, and late comments did not allow staff sufficient time to make changes.



• CAI conversion: The conversion to CAI meant reworking many processes which had been fine-tuned during the years of PAPI data collection. In combination with the volume, the conversion meant a simple task like storing equipment turned entire rooms into warehouses of Gateway computer boxes. Also, because of the significant effort and time required to load large number of computers with the final CAI system, the instrument had to be finalized earlier than in the past.



• Electronic screening: The decision to switch from paper screening to using the Newton meant developing, testing, and documenting a new and tricky (but worthwhile) computer program. As with the conversion to CAI, the change in screening meant additional effort to either rework or replace past processes. It certainly complicated the development of manuals and training materials. For example, a question on a paper screening form could be typed into the manual text, whereas displaying the question as seen on the Newton screen meant capturing the screen as a digital photograph, converting it to an image file, then sizing it for inclusion in the manual.



• Spanish translations: With limited experienced bilingual personnel, the overseeing of translations, taping of WAV files, and reviewing of computer programs was a significant amount of work for personnel already overburdened with other critical tasks.



• Last minute requests: As in previous years, some requests for alterations in the screening and instrument text were received either just before—or after—the established deadlines. This increased workloads which were already too heavy. Any requested change, however slight, required another round of extensive testing to be sure the change did not impact another area of the program (this varies considerably from altering a paper version of the instrument). The Spanish versions also had to be changed and checked. If the requested change impacted the ACASI sections of the CAI, it meant reworking WAV files in both English and Spanish. Changes could also ripple through manuals and drafted training materials. Requested last minute changes burdened programmers, software testers, manual writers, and training developers and shortened the computer loading schedule.



• PAPI sample: Including the PAPI sample further complicated the CAI conversion, since past paper processes had to be maintained. Most often, the past processes had to also be reworked so that they would mesh with the CAI systems. It also doubled development work for some materials, such as needing two Showcard Booklets and two versions of the Question and Answer Brochure.

¹ "It's a Big Job" won the 1999 Silver Reel Award for effective use of training videos.

This page was last updated on June 03, 2008.

SAMHSA, an agency in the Department of Health and Human Services, is the Federal Government's lead agency for improving the quality and availability of substance abuse prevention, addiction treatment, and mental health services in the United States.