Robert Wood Johnson

To enhance sampling design efficiency consideration should be given to sampling schools rather than districts as the primary sampling units (PSU). The would both enhance the efficacy of and contribute to the feasibility of designs stratifying PSUs by unit characteristics such as region, socio-economic status, enrollment, public-private, and ethnic composition.

First, the essence of this report including tables and figures will be placed on the website of the Metropolitan Research and Policy Institute, University of Texas, San Antonio. Second, the results and recommendations will be used to create fact sheets to support the work of the Latino Council on Alcohol and Tobacco (LCAT) in Washington. Third, policy papers and manuscripts of substance findings will be submitted for publication as well as in unpublished format by the project investigators, and the Metropolitan Research and Policy Institute in the Occasional Policy Paper Series. Fourth, a copy of this report will be sent to the states we have assessed and to NASADAD.

The major goal of this study was to examine data collected by individual states for the purposes of evaluating the implications for policies addressing adolescent substance use at state and national levels. Specific focus was on determining policy implications for Latino populations nationally. The first step in this process was to examine the methodology and data collected by states for the purposes of determining rates of substance use among high school populations. During the initial phase of the project, we were required to choose from two options in the examination of state youth drug use surveys. These options were:
Acquisition of raw data from states with large Latino populations, and analyses of such data, or Examination and comparison of state drug use survey reports.

Early in the implementation of the project it became evident that the only viable alternative was the second option. The reasons for this are informative in terms of future attempts to foster greater integration and comparability of state youth drug use surveys.
First, since most states subcontract the survey to research firms or universities, it is difficult to track the subcontractors responsible for the survey in a given year. In some states the same firm has had the subcontract for many years. However, in other states the firms have changed, and sometimes there is not sufficient institutional memory both in the private firms who received the state contracts and the state agencies in charge of the surveys.

Second, in some states different agencies are responsible for surveys of high school students and middle school students. In some instances it is the department of education for one survey, and the department of substance use for the other (e.g., New Jersey). Moreover, in other states surveys are conducted at the school district level (e.g., Florida).
Third, in terms of examining policy implications and making recommendations, the actual state reports are the tools that inform policy decisions for state governments.

Sampling considerations: All surveys examined utilize multi-stage cluster sampling designs. The most typical approach selects school districts, then schools within districts followed by intact classes within schools. Districts and/or schools are commonly selected under a stratified or lattice design building-in control for region. To reduce costs and the potential burden or disruption of pulling students out of classes, the universal approach of surveying intact or entire classes has been adopted. Accommodation of these practical constraints to use of intact classes creates the tension of large sample sizes in the final stage forcing smaller sizes in earlier stages. Given a fixed total sample size, from a statistical standpoint, smaller n's at the final, classroom stage would facilitate smaller sampling error terms through the selection of more districts, schools, and classrooms. Given within-cluster (class) correlations, a multi-stage, district-school-class-student design using 40x5x5x5=5000 would be superior to a more typical one using 25x2x4x25=5000. Also, given within school district homogeneity, better designs could well omit sampling school districts altogether and use schools primary sampling units (PSU). This approach would offer the benefit of improved and more meaningful sampling from strata or design cells employing geographic, socio-economic, and/or public-private, school size, controls. Selecting more schools through this approach would increase the study school recruitment burden in that more schools would mean more solicitations of district as well as school approval. It is; nevertheless, possible that districts are more likely to view such approval as limited to a few schools rather than a blanket, district-wide commitment. The already high levels of non-participation at the district level reported in some studies would appear to lower the risk of additional non-participation.

Group administration to intact classes has become the virtual norm. While this has clear justification on the grounds of administrative efficiency and minimization of classroom disruption, it has the potential to create subjective over-estimations of study precision through focus on sample size alone. Given machine-readable questionnaires and consequent low data processing costs, the cost differential between surveying an entire class or some fraction thereof tends to be minimal. The estimate precision gain from surveying n=25 or all students in a class; however, is not equivalent to the gain to be derived from an alternative of sampling 5 students from 5 classes. Given the preference for intact class sampling, large sample sizes are both to be expected and necessary. Estimation of standard errors for the purposes of reporting confidence intervals or significance tests for differences are complicated by multistage designs and require specialized software applications. It is not readily apparent that sampling design efficiency( lowest standard error for a given sample size) has driven design choices or that appropriate estimation procedures are used in all of the studies examined.

Non-response at each sampling stage is problematic across the studies. The true sampling frame in all of these studies is students: 1) who were in attendance on the day of administration, 2) who chose to participate, 3) who returned their consent form, 4) whose parent(s) consented, and 5) whose a) teacher, b) school, and c) district agreed to participate. Technically, district and school stage non-participation limit inference to "students who attend schools/districts which chose to participate". With reported compliance rates at these stages near 50% in some studies, the utility of the entire study is seriously compromised. Although replacement methods are typically employed, these are a poor alternative to successful recruitment. This problem becomes particularly onerous when it is difficult to find a true match because there are few entries in given cells of a stratification matrix.

Active (written) parental consent is reported as being especially problematic to these studies. As this is a policy matter, the researcher must adapt by improving acquisition of consent forms. Rates of negative parental replies appear to be about 5%, much of the balance of the problem appears to fall upon students failure to deliver the forms either to their parent or return them to the school. It is unknown what portion of the non-returns represent an implied or latent positive or negative parental response. Clearly, more successful protocols for collecting active consent forms need to be developed as well as a better understanding of the actual causes for the breakdown of the process. In addition to increasing motivation to participation the part of both parents and students, careful advanced planning accompanied by multiple attempts as well as follow-up telephone calls could reduce non-response at this stage.

Because student participation is voluntary, biases resulting from non-participation pose a serious threat to study integrity. Rates of participation within schools and classes have been reported to vary widely. In addition to efforts to enhance participation, systematic efforts to examine and adjust for potential biases must be addressed. Intact class surveying can serve to motivate both participation and non-participation and it is quite possible that in schools encountering student resistance, alternate methods should be evaluated.