National data      State level data       Metropolitan and other subState area data

Substance Abuse in States and Metropolitan Areas:
Model Based Estimates from the 1991-1993 National Household Surveys on Drug Abuse

Chapter 4

4. Method Evaluation:

4.1 Evaluation Strategies

4.2 Results of the Evaluations

4.3 Summary of the Evaluations

In any small area estimation (SAE) project, questions arise as to how well the model describes the outcome variables, that is, in this case, how effective are the small area estimation models in describing substance abuse in the States and MSAs. When producing small area estimates, there usually are no A gold-standards,@ (i.e., direct estimates with known reliability and validity) that can be used to evaluate the overall methodology. In fact, if there were such estimates available, policy makers could use those and would not need small area estimates. Thus, in the absence of such gold-standards for use in an evaluation, one uses multiple evaluation procedures which, when considered jointly, give an overall indication of the quality of the small area estimates. In this small area estimation study, a large variety of evaluations were conducted. The following summarizes some of these evaluations. ^{(footnote#17)}

• The statistical evaluations based on examining the agreement between direct NHSDA estimates and the SAE estimates within evaluation subgroups (see below for an explanation of how this was done), indicated that the SAE estimates had moderate to high correlations with direct estimates, that there were few significant differences between the direct estimates and SAE estimates, and that the SAE estimates adequately reflected the range of prevalence in substance abuse.

 

• The SAE national estimates closely track the NHSDA direct estimates.

   

• Statistical evaluations based on comparisons to external data on cigarette and alcohol use indicated that the SAE estimates were at about the same level as the external data, that the correlations with the external data were moderate to high, and that the estimates were good to very good at reflecting the range of prevalence of cigarette and alcohol use across the States.

 

• Statistical evaluations based on comparisons to external data from administrative data on arrest and past year treatment for substance abuse indicated that the SAE estimates for States were lower than those based on administrative data (as are the NHSDA estimates at the national level) and had only poor to fair correlations with the corresponding administrative data.

 

• Substance abuse characteristics that have higher prevalence were more adequately reflected by the modeling than those characteristics with lower levels of prevalence.

  

1) Goodness-of-fit tests

2) Comparisons to external measures of substance abuse

3) Comparisons of the final SAE model to other models

• The big-city-subsample model is similar to the full SAE in that it is a composite estimator that includes both an indirect and direct component; however, in this case a subsample of the respondents from the six large cities was selected and pooled with the full sample from the remainder of the nation before fitting the models. This model was investigated because it reflects the composition of continuing NHSDA surveys which do not have large oversamples from these big cities.

• The SAE indirect estimator drops the local area direct effects from the full SAE model. This model was fit to evaluate the usefulness of using the composite estimator. Differences between this model and the full SAE model give one an idea as to whether adding the local area effects to the model improves its ability to predict substance abuse. Also, comparing this model with the next simpler model, the county demographic model (below), measures the impact of the addition of the other demographic variables.

Three simpler models were used a basic demographic model, a demographic model that used six large city/non-large city splits, and a model that included the basic demographics and county level predictions.

Using these three strategies - -  goodness-of-fit tests, comparisons to external data, and comparisons to other models - -  we evaluated the SAE model by: 1) carrying out goodness-of-fit tests for the final SAE model using the evaluation subgroups, 2) conducting cross-validation goodness-of-fit tests on the final SAE and several other models, and 3) comparing estimates from the SAE model and the other types of models to external data by calculating rank correlations and range ratios for the various sets of estimates. The results are given in the next section.

Exhibit 4.2 - - Summary of Evaluations of Final SAE Model and Comparisons to Alternative Models Based on a Cross-validation Goodness-of-fit Tests. Correlations, χ²- Probabilities*, and Range Ratios.

Exhibit 4.2 summarizes the cross-validation analyses. ^{(footnote#23)} In this table, we also introduce for the first time four of the other models that were examined. For example, looking at the first row of Exhibit 4.2, which shows the correlations between the predicted and the direct estimates in the evaluation subgroups, we note that the final SAE model has the highest correlation, the SAE model without the direct local area effects the next highest, and so on.

Exhibit 4.2

• The cross-validation approach presents a somewhat less optimistic view of the final SAE model than did the goodness-of-fit tests using evaluation subgroups. The correlations are lower and the range ratios are higher than those observed in Exhibit 4.1. This probably means that the final SAE model included some predictors that should not have been included. This is sometimes called A over-fitting@ in that some of the factors that were identified as being good predictors of substance abuse in one sample were actually not very good predictors when applied to a different sample. ^{(footnote#24)}

• The models that used only the demographic characteristics are the poorest performers in almost all cases. The range ratios indicate that simply using demographic characteristics to predict drug use in a small area will probably not reflect the true range in prevalence estimates across the small areas. This indicates that improvements can be achieved by including more predictors in the models.

• The county demographic model worked fairly well.  ^{(footnote#25)}  Although the full SAE model was the overall best performer, both the county demographic model and the SAE model without the direct effects worked fairly well. The county demographic model estimates are simpler to calculate than the SAE model without the direct effects; therefore, it may be a good candidate for future SAE modeling activities.

For each of the six models examined, Exhibits 4.3 through 4.6 compare the estimated prevalences to those from other sources using both the rank correlation and the range ratios. Methodological and definitional differences between the NHSDA and the other sources are considered in making these comparisons. Adjustments were made to the external data in some cases, but it was not possible to fully account for the differences. The other data sources and the adjustments made for comparison with SAE estimates are described below:

Behavioral Risk Factor Surveillance System (BRFSS): Comparisons of the SAE estimates of the prevalence of past month alcohol and cigarette use are made to estimates from the BRFSS without making any adjustments. The BRFSS is a telephone survey conducted in all 50 States under cooperative agreements with the Centers for Disease Control and Prevention. Definitions used are comparable to NHSDA definitions, since the BRFSS estimates reflect past month use. Studies have shown that reporting of substance use behaviors may be lower in telephone surveys than in face-to-face surveys, particularly for illicit drugs. The BRFSS State estimates are simple averages over the three years 1991 through 1993.

National Drug and Alcoholism Treatment Unit Survey (NDATUS): SAE estimates of the number of persons receiving treatment for drug abuse are compared to estimates constructed from NDATUS. NDATUS is an inventory of all specialty substance abuse treatment facilities in the U.S. Based on reporting by State substance abuse agencies, it provides estimates (including adjustments for nonresponse) of the number of clients in treatment at a given point in time. To develop an estimate of persons treated during a year, the NDATUS client counts (including drug only and combined drug and alcohol clients) were multiplied by the reciprocals of average lengths of stay, and adjusted to account for multiple treatment episodes in a year by the same individual. Estimates of length of stay and multiple episodes were obtained from the Drug Services Research Survey, conducted in 1990. These calculations were done within categories of treatment modality and and applied separately to each state. No adjustment was made to account for the inclusion in the SAE estimates of persons reporting treatment through self-help groups, private physicians, or emergency rooms, none of which are counted in NDATUS. The State estimates are averages over only 1992 and 1993 since the 1991 estimates could not be adequately adjusted for nonresponse.

Uniform Crime Reports (UCR): The SAE estimates of the number of persons arrested in the past year are compared to estimates derived from the UCR. The UCR compiles data from local jurisdictions on the number of arrests. For comparison with the SAE estimates, an adjustment to the UCR data was made to account for persons arrested more than once during a year, so the adjusted UCR estimates reflect number of persons arrested at least once. This adjustment was made within the four Census regions, using data on multiple arrests reported by arrestees in the NHSDA sample. The State estimates are simple averages over the three years 1991 through 1993.

For each of the six models examined, Exhibits 4.3 through 4.6 compare the estimated prevalences to those from other sources using both the rank correlation ^{(footnote#26)}  and the range ratios.  ^{(footnote#27)}  Of note in these exhibits is the following:

Exhibit 4.3 presents the results that compare the final SAE estimates for the BRFSS estimates for alcohol use. The rank correlations between the final SAE estimates and the BRFSS estimates are quite high (over 0.85) and the range ratio is good (nearly 0.6). This indicates that the SAE model produced estimates that are very consistent with what is found in the BRFSS. In addition, both sources estimate a similar prevalence of use at the national level with the NHSDA estimates being somewhat larger. This higher level of reporting is consistent with findings from methodology studies which have shown that telephone surveys yield lower reports of use than self-administered surveys.  ^{(footnote#28)}  In addition, we note that the rank correlations and range ratios are much better for the final SAE model than the corresponding statistics for the two demographic models. This indicates that the SAE model is performing much better than typical synthetic estimators which states might construct by applying the NHSDA rates of use to their population distribution.

The comparison of the final SAE estimates to the BRFSS smoking data (Exhibit 4.4) presents a similar picture. The range ratio is very good (over 0.9) and the rank correlations are moderate (about 0.5). The fact that the BRFSS uses a somewhat different question than the NHSDA may account for some of the lack of comparability. Again, we observe that the NHSDA estimates higher levels of cigarette use than the BRFSS, which is consistent with the difference in interview methodology. The demographic model does not do as well as the final SAE model, indicating that the SAE model is better than using a synthetic estimator.

Exhibits 4.5 and 4.6 present the results for the past year drug treatment and past year arrest. Although we again observe that the SAE models perform better than the demographic models, the rank correlations are still only 0.38 for treatment and 0.35 for arrest. These low correlations are probably due to the lack of correspondence in methodology between the divergent data sources, the low prevalence rates of these items, and the less restricted range across States.  ^{(footnote#29)}

One of the characteristics that is desirable in a small area estimation procedure is that it produce estimates that adequately reflect the range of differences across areas. Because the estimated NHSDA prevalence rates for treatment and arrest are lower than the corresponding estimates from NDATUS and UCR, the calculated range ratios are going to appear artificially small. That is, the range ratios in Exhibits 4.5 and 4.6 are not a good indication of how well the models reflect the range of differences across areas because of the differences in overall prevalence levels

As was noted in the beginning of this section, a variety of approaches must be used to evaluate small area estimation methods. Considering all of the evidence presented,  ^{(footnote#30)} the following findings are noteworthy:

• The full SAE model is generally the best model in that it tends to have the highest correlations with the direct estimates, to adequately reflect the range of prevalence, and to have exhibited few significant differences when compared to direct estimates in Goodness-of-fit tests.

• The full SAE model is a better predictor for the more prevalent behaviors than for the less prevalent behaviors and better at reflecting the differences in prevalence rates across areas when there is a wider dispersion of rates across the areas.