Background Since 2001, the District of Columbia Department of Health has been using an emergency ... more Background Since 2001, the District of Columbia Department of Health has been using an emergency room syndromic surveillance system to identify possible disease outbreaks. Data are received from a number of local hospital emergency rooms and analyzed daily using a variety of statistical detection algorithms. The aims of this paper are to characterize the performance of these statistical detection algorithms in rigorous yet practical terms in order to identify the optimal parameters for each and to compare the ability of two syndrome definition criteria and data from a children's hospital versus vs. other hospitals to determine the onset of seasonal influenza. Methods We first used a fine-tuning approach to improve the sensitivity of each algorithm to detecting simulated outbreaks and to identifying previously known outbreaks. Subsequently, using the fine-tuned algorithms, we examined (i) the ability of unspecified infection and respiratory syndrome categories to detect the start...
Background The COVID-19 pandemic has led to an avalanche of scientific studies, drawing on many d... more Background The COVID-19 pandemic has led to an avalanche of scientific studies, drawing on many different types of data. However, studies addressing the effectiveness of government actions against COVID-19, especially non-pharmaceutical interventions, often exhibit data problems that threaten the validity of their results. This review is thus intended to help epidemiologists and other researchers identify a set of data issues that, in our view, must be addressed in order for their work to be credible. We further intend to help journal editors and peer reviewers when evaluating studies, to apprise policy-makers, journalists, and other research consumers about the strengths and weaknesses of published studies, and to inform the wider debate about the scientific quality of COVID-19 research. Results To this end, we describe common challenges in the collection, reporting, and use of epidemiologic, policy, and other data, including completeness and representativeness of outcomes data; th...
Immediately following September 11, 2001, the District of Columbia Department of Health began a s... more Immediately following September 11, 2001, the District of Columbia Department of Health began a syndromic surveillance program based on emergency room visits. The number of patients is recorded on the basis of chief complaint coded the following syndromic categories: death, sepsis, rash, respiratory complaints, gastrointestinal complaints, unspecified infection, neurological, or other complaints. Extending a previous analysis with the addition of two years of data and optimally chosen parameters for detection algorithms, we compared the timing of first alerts of the flu season in the DC syndromic surveillance data and other syndromic and non-syndromic data systems. Comparing the earliest dates at which various systems flag, we found that unspecified infection cases, and especially those from a children's hospital, consistently flags earlier than any of the other six hospitals and before a multivariate analysis of the other six hospitals in our analysis. In addition, we found tha...
Permission is given to duplicate this electronic document for personal use only, as long as it is... more Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page. ... The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the ...
Background After Action Reviews (AARs) provide a means to observe how well preparedness systems p... more Background After Action Reviews (AARs) provide a means to observe how well preparedness systems perform in real world conditions and can help to identify – and address – gaps in national and global public health emergency preparedness (PHEP) systems. WHO has recently published guidance for voluntary AARs. This analysis builds on this guidance by reviewing evidence on the effectiveness of AARs as tools for system improvement and by summarizing some key lessons about ensuring that AARs result in meaningful learning from experience. Results Empirical evidence from a variety of fields suggests that AARs hold considerable promise as tools of system improvement for PHEP. Our review of the literature and practical experience demonstrates that AARs are most likely to result in meaningful learning if they focus on incidents that are selected for their learning value, involve an appropriately broad range of perspectives, are conducted with appropriate time for reflection, employ systems frame...
The third paper in a series on how learning health systems can use routinely collected electronic... more The third paper in a series on how learning health systems can use routinely collected electronic health data (EHD) to advance knowledge and support continuous learning, this review describes how analytical methods for individual-level electronic health data EHD, including regression approaches, interrupted time series (ITS) analyses, instrumental variables, and propensity score methods, can also be used to address the question of whether the intervention "works." The two major potential sources of bias in non-experimental studies of health care interventions are that the treatment groups compared do not have the same probability of treatment or exposure and the potential for confounding by unmeasured covariates. Although very different, the approaches presented in this chapter are all based on assumptions about data, causal relationships, and biases. For instance, regression approaches assume that the relationship between the treatment, outcome, and other variables is pro...
About this PDF file: This new digital representation of the original work has been recomposed fro... more About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.
About this PDF file: This new digital representation of the original work has been recomposed fro... more About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.
This report is part of the RAND Corporation draft series. The unrestricted draft was a product of... more This report is part of the RAND Corporation draft series. The unrestricted draft was a product of the RAND Corporation from 1993 to 2003 that represented preliminary or prepublication versions of other more formal RAND products for distribution to appropriate external audiences. The draft could be considered similar to an academic discussion paper. Although unrestricted drafts had been approved for circulation, they were not usually formally edited or peer reviewed. ... Permission is given to duplicate this electronic document for personal use only, as long as it is ...
Abstract: The level of bioterrorism preparedness across California's jurisdictio... more Abstract: The level of bioterrorism preparedness across California's jurisdictions is uneven, ranging from excellent to poor. There are wide variations in every aspect of preparedness strategy, development, and implementation. The system suffers from inefficiency and waste. Strong leadership will be required to develop a shared understanding of public health organization and responsibilities.
RAND was contracted by the US Department of Health and Human Services to develop resources and pr... more RAND was contracted by the US Department of Health and Human Services to develop resources and prepare analyses that help describe and enhance key aspects of state and local public health emergency preparedness. As part of this work, RAND was asked to compile a repository of exemplary practices in public health preparedness. This report provides an overview of the methods and criteria used to select and nominate 13 candidate practices for the repository and provides brief descriptions of the practices that have been ...
The growth of university-industry research relationships in biotechnology has raised questions co... more The growth of university-industry research relationships in biotechnology has raised questions concerning their effects, both positive and negative, on universities. A survey of over 1200 faculty members at 40 major universities in the United States reveals that biotechnology researchers with industrial support publish at higher rates, patent more frequently, participate in more administrative and professional activities and earn more than colleagues without such support. At the same time, faculty with industry funds are much more likely than other biotechnology faculty to report that their research has resulted in trade secrets and that commercial considerations have influenced their choice of research projects. Although the data do not establish a causal connection between industrial support and these faculty behaviors, our findings strongly suggest that university-industry research relationships have both benefits and risks for academic institutions. The challenge for universities is to find ways to manage these relationships that will preserve the benefits while minimizing the risks.
I s California's public health system prepared for a bioterrorist attack? Or a deadly new infecti... more I s California's public health system prepared for a bioterrorist attack? Or a deadly new infectious disease such as SARS? A RAND Corporation team found that even in California-widely regarded as one of the best-prepared states-the level of preparedness ranged from excellent to poor. Californians' level of public health protection depends on the public health jurisdiction in which they happen to live.
Legal preparedness is a critical component of comprehensive public health preparedness for public... more Legal preparedness is a critical component of comprehensive public health preparedness for public health emergencies. The scope of this study was to assess the usefulness of combining didactic sessions with a tabletop exercise as educational tools in legal preparedness, to assess the impact of the exercise on the participants' level of confidence about the legal preparedness of a public health system, and to identify legal issue areas in need of further improvement. The exercise scenario and the pre- and postexercise evaluation were designed to assess knowledge gained and level of confidence in declaration of emergencies, isolation and quarantine, restrictions (including curfew) on the movement of people, closure of public places, and mass prophylaxis, and to identify legal preparedness areas most in need of further improvement at the system level. Fisher exact test and paired t test were performed to compare pre- and postexercise results. Our analysis shows that a combination of didactic teaching and experiential learning through a tabletop exercise regarding legal preparedness for infectious disease emergencies can be effective in both imparting perceived knowledge to participants and gathering information about sufficiency of authorities and existence of gaps. The exercise provided a valuable forum to judge the adequacy of legal authorities, policies, and procedures for dealing with pandemic influenza at the state and local levels in Massachusetts. In general, participants were more confident about the availability and sufficiency of legal authorities than they were about policies and procedures for implementing them. Participants were also more likely to report the need for improvement in authorities, policies, and procedures in the private sector and at the local level than at the state level.
Abstract Power transformations can often simplify the task of describing behavior in data that co... more Abstract Power transformations can often simplify the task of describing behavior in data that consist of positive measurements or counts. Tukey's diagnostic techniques based on two-dimensional plots are useful in searching for a suitable transformation in two types of data structures: multiple batches at different levels, and two-way tables. To these, we add plots for finding transformations in two other structures: a single batch, and y versus x. The basis for plots that indicate transformations for symmetrizing and for straightening is explored using ...
Abstract: Over the past three years, the US Department of Health and Human Services (HHS) has mad... more Abstract: Over the past three years, the US Department of Health and Human Services (HHS) has made significant investments in state and local public health in an effort to enhance public health emergency preparedness. The RAND Corporation was contracted to work with the HHS Office of the Assistant Secretary for Public Health Emergency Preparedness (OASPHEP) to develop resources and to prepare analyses to help describe and enhance key aspects of state and local public health emergency preparedness. As ...
Background Since 2001, the District of Columbia Department of Health has been using an emergency ... more Background Since 2001, the District of Columbia Department of Health has been using an emergency room syndromic surveillance system to identify possible disease outbreaks. Data are received from a number of local hospital emergency rooms and analyzed daily using a variety of statistical detection algorithms. The aims of this paper are to characterize the performance of these statistical detection algorithms in rigorous yet practical terms in order to identify the optimal parameters for each and to compare the ability of two syndrome definition criteria and data from a children's hospital versus vs. other hospitals to determine the onset of seasonal influenza. Methods We first used a fine-tuning approach to improve the sensitivity of each algorithm to detecting simulated outbreaks and to identifying previously known outbreaks. Subsequently, using the fine-tuned algorithms, we examined (i) the ability of unspecified infection and respiratory syndrome categories to detect the start...
Background The COVID-19 pandemic has led to an avalanche of scientific studies, drawing on many d... more Background The COVID-19 pandemic has led to an avalanche of scientific studies, drawing on many different types of data. However, studies addressing the effectiveness of government actions against COVID-19, especially non-pharmaceutical interventions, often exhibit data problems that threaten the validity of their results. This review is thus intended to help epidemiologists and other researchers identify a set of data issues that, in our view, must be addressed in order for their work to be credible. We further intend to help journal editors and peer reviewers when evaluating studies, to apprise policy-makers, journalists, and other research consumers about the strengths and weaknesses of published studies, and to inform the wider debate about the scientific quality of COVID-19 research. Results To this end, we describe common challenges in the collection, reporting, and use of epidemiologic, policy, and other data, including completeness and representativeness of outcomes data; th...
Immediately following September 11, 2001, the District of Columbia Department of Health began a s... more Immediately following September 11, 2001, the District of Columbia Department of Health began a syndromic surveillance program based on emergency room visits. The number of patients is recorded on the basis of chief complaint coded the following syndromic categories: death, sepsis, rash, respiratory complaints, gastrointestinal complaints, unspecified infection, neurological, or other complaints. Extending a previous analysis with the addition of two years of data and optimally chosen parameters for detection algorithms, we compared the timing of first alerts of the flu season in the DC syndromic surveillance data and other syndromic and non-syndromic data systems. Comparing the earliest dates at which various systems flag, we found that unspecified infection cases, and especially those from a children's hospital, consistently flags earlier than any of the other six hospitals and before a multivariate analysis of the other six hospitals in our analysis. In addition, we found tha...
Permission is given to duplicate this electronic document for personal use only, as long as it is... more Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page. ... The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the ...
Background After Action Reviews (AARs) provide a means to observe how well preparedness systems p... more Background After Action Reviews (AARs) provide a means to observe how well preparedness systems perform in real world conditions and can help to identify – and address – gaps in national and global public health emergency preparedness (PHEP) systems. WHO has recently published guidance for voluntary AARs. This analysis builds on this guidance by reviewing evidence on the effectiveness of AARs as tools for system improvement and by summarizing some key lessons about ensuring that AARs result in meaningful learning from experience. Results Empirical evidence from a variety of fields suggests that AARs hold considerable promise as tools of system improvement for PHEP. Our review of the literature and practical experience demonstrates that AARs are most likely to result in meaningful learning if they focus on incidents that are selected for their learning value, involve an appropriately broad range of perspectives, are conducted with appropriate time for reflection, employ systems frame...
The third paper in a series on how learning health systems can use routinely collected electronic... more The third paper in a series on how learning health systems can use routinely collected electronic health data (EHD) to advance knowledge and support continuous learning, this review describes how analytical methods for individual-level electronic health data EHD, including regression approaches, interrupted time series (ITS) analyses, instrumental variables, and propensity score methods, can also be used to address the question of whether the intervention "works." The two major potential sources of bias in non-experimental studies of health care interventions are that the treatment groups compared do not have the same probability of treatment or exposure and the potential for confounding by unmeasured covariates. Although very different, the approaches presented in this chapter are all based on assumptions about data, causal relationships, and biases. For instance, regression approaches assume that the relationship between the treatment, outcome, and other variables is pro...
About this PDF file: This new digital representation of the original work has been recomposed fro... more About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.
About this PDF file: This new digital representation of the original work has been recomposed fro... more About this PDF file: This new digital representation of the original work has been recomposed from XML files created from the original paper book, not from the original typesetting files. Page breaks are true to the original; line lengths, word breaks, heading styles, and other typesetting-specific formatting, however, cannot be retained, and some typographic errors may have been accidentally inserted. Please use the print version of this publication as the authoritative version for attribution.
This report is part of the RAND Corporation draft series. The unrestricted draft was a product of... more This report is part of the RAND Corporation draft series. The unrestricted draft was a product of the RAND Corporation from 1993 to 2003 that represented preliminary or prepublication versions of other more formal RAND products for distribution to appropriate external audiences. The draft could be considered similar to an academic discussion paper. Although unrestricted drafts had been approved for circulation, they were not usually formally edited or peer reviewed. ... Permission is given to duplicate this electronic document for personal use only, as long as it is ...
Abstract: The level of bioterrorism preparedness across California's jurisdictio... more Abstract: The level of bioterrorism preparedness across California's jurisdictions is uneven, ranging from excellent to poor. There are wide variations in every aspect of preparedness strategy, development, and implementation. The system suffers from inefficiency and waste. Strong leadership will be required to develop a shared understanding of public health organization and responsibilities.
RAND was contracted by the US Department of Health and Human Services to develop resources and pr... more RAND was contracted by the US Department of Health and Human Services to develop resources and prepare analyses that help describe and enhance key aspects of state and local public health emergency preparedness. As part of this work, RAND was asked to compile a repository of exemplary practices in public health preparedness. This report provides an overview of the methods and criteria used to select and nominate 13 candidate practices for the repository and provides brief descriptions of the practices that have been ...
The growth of university-industry research relationships in biotechnology has raised questions co... more The growth of university-industry research relationships in biotechnology has raised questions concerning their effects, both positive and negative, on universities. A survey of over 1200 faculty members at 40 major universities in the United States reveals that biotechnology researchers with industrial support publish at higher rates, patent more frequently, participate in more administrative and professional activities and earn more than colleagues without such support. At the same time, faculty with industry funds are much more likely than other biotechnology faculty to report that their research has resulted in trade secrets and that commercial considerations have influenced their choice of research projects. Although the data do not establish a causal connection between industrial support and these faculty behaviors, our findings strongly suggest that university-industry research relationships have both benefits and risks for academic institutions. The challenge for universities is to find ways to manage these relationships that will preserve the benefits while minimizing the risks.
I s California's public health system prepared for a bioterrorist attack? Or a deadly new infecti... more I s California's public health system prepared for a bioterrorist attack? Or a deadly new infectious disease such as SARS? A RAND Corporation team found that even in California-widely regarded as one of the best-prepared states-the level of preparedness ranged from excellent to poor. Californians' level of public health protection depends on the public health jurisdiction in which they happen to live.
Legal preparedness is a critical component of comprehensive public health preparedness for public... more Legal preparedness is a critical component of comprehensive public health preparedness for public health emergencies. The scope of this study was to assess the usefulness of combining didactic sessions with a tabletop exercise as educational tools in legal preparedness, to assess the impact of the exercise on the participants' level of confidence about the legal preparedness of a public health system, and to identify legal issue areas in need of further improvement. The exercise scenario and the pre- and postexercise evaluation were designed to assess knowledge gained and level of confidence in declaration of emergencies, isolation and quarantine, restrictions (including curfew) on the movement of people, closure of public places, and mass prophylaxis, and to identify legal preparedness areas most in need of further improvement at the system level. Fisher exact test and paired t test were performed to compare pre- and postexercise results. Our analysis shows that a combination of didactic teaching and experiential learning through a tabletop exercise regarding legal preparedness for infectious disease emergencies can be effective in both imparting perceived knowledge to participants and gathering information about sufficiency of authorities and existence of gaps. The exercise provided a valuable forum to judge the adequacy of legal authorities, policies, and procedures for dealing with pandemic influenza at the state and local levels in Massachusetts. In general, participants were more confident about the availability and sufficiency of legal authorities than they were about policies and procedures for implementing them. Participants were also more likely to report the need for improvement in authorities, policies, and procedures in the private sector and at the local level than at the state level.
Abstract Power transformations can often simplify the task of describing behavior in data that co... more Abstract Power transformations can often simplify the task of describing behavior in data that consist of positive measurements or counts. Tukey's diagnostic techniques based on two-dimensional plots are useful in searching for a suitable transformation in two types of data structures: multiple batches at different levels, and two-way tables. To these, we add plots for finding transformations in two other structures: a single batch, and y versus x. The basis for plots that indicate transformations for symmetrizing and for straightening is explored using ...
Abstract: Over the past three years, the US Department of Health and Human Services (HHS) has mad... more Abstract: Over the past three years, the US Department of Health and Human Services (HHS) has made significant investments in state and local public health in an effort to enhance public health emergency preparedness. The RAND Corporation was contracted to work with the HHS Office of the Assistant Secretary for Public Health Emergency Preparedness (OASPHEP) to develop resources and to prepare analyses to help describe and enhance key aspects of state and local public health emergency preparedness. As ...
This technical report aims to take this work a step further by creating a linkage between the cap... more This technical report aims to take this work a step further by creating a linkage between the capabilities listed in the logic model for PHEP in the EU Member States – which apply at the emergency preparedness system level – with competencies for individuals who work in the system by generating a competency-based model. A subsequent phase, to be completed in 2017–2018, will involve the development and pilot testing of competency-based curricula.
Although informed by public health emergency practitioners, this report is primarily written for educators who will develop competency-based training programmes. It should also be useful for organisations that employ professionals trained in public health emergency preparedness. Public health emergency preparedness is very broad, diverse, and rather complex societal process of preparing and responding to health threats caused by communicable diseases and other cross-border threats to health.
ECDC has initiated a medium-term project aimed at designing competency-based training curricula for experts working in preparedness at the national level in the Member States. ‘Competency-based’ means that training participants acquire knowledge and skills based on a defined set of core competencies they need to best fulfil their job responsibilities. The competency-based approach enhances harmonisation and helps to ensure similar levels of competency throughout the EU, regardless of the background of those working in the field.
These competencies are intended to describe the knowledge and skills needed by experts working in preparedness at the national level such as national preparedness committee members or their equivalent. We recognise that the preparedness and emergency response structures of EU Member States vary, and use the term ‘national preparedness committee’ generically to describe the group of individuals who work on these issues in any given country. In general, however, public health preparedness is included in the emergency management structures in a country, which are usually led by interior affairs or civil protection staff. When an emergency is of a health nature, then health personnel typically take the lead.
While the PHEP logic model and competencies can be useful in preparedness planning, it is not ECDC’s goal to mandate any particular emergency response approach, structure, or capabilities. These are clearly the responsibility of the Member States and necessarily reflect each country’s own resources, administrative and political structures, as well as their responsibilities under Decision 1082 and the International Health Regulations (2005). Similarly, it is not ECDC’s intention to specify the knowledge or skills needed by any particular group of preparedness workers. Rather, our goal is to describe the competencies that different groups of professionals need – in addition to the basic knowledge and skills required for their profession – when they are called upon to serve as members of national preparedness committees or teams in order to facilitate an effective national and cross-border response to health threats in Europe.
In 2017, the European Centre for Disease Prevention and Control (ECDC) developed a competency mod... more In 2017, the European Centre for Disease Prevention and Control (ECDC) developed a competency model for individuals who work in public health emergency preparedness (PHEP) in European Union (EU) countries. The model serves as the basis for developing competency-based training programmes to support professionals in PHEP efforts at the country level. The competency model describes the knowledge and skills professionals need when working in national-level PHEP, such as preparedness committee members or their equivalents. In order to develop the model, existing competency statements were reviewed, as well as case studies and reports. Fifty-three professionals from the EU and other countries provided feedback to the model by participating in a three-stage consultation process. The model includes 102 competency, 100 knowledge and 158 skill statements. In addition to specifying the appropriate content for training programmes, the proposed common competency model can help to standardise terminology and approaches to PHEP training.
Uploads
Papers by Michael Stoto
Although informed by public health emergency practitioners, this report is primarily written for educators who will develop competency-based training programmes. It should also be useful for organisations that employ professionals trained in public health emergency preparedness. Public health emergency preparedness is very broad, diverse, and rather complex societal process of preparing and responding to health threats caused by communicable diseases and other cross-border threats to health.
ECDC has initiated a medium-term project aimed at designing competency-based training curricula for experts working in preparedness at the national level in the Member States. ‘Competency-based’ means that training participants acquire knowledge and skills based on a defined set of core competencies they need to best fulfil their job responsibilities. The competency-based approach enhances harmonisation and helps to ensure similar levels of competency throughout the EU, regardless of the background of those working in the field.
These competencies are intended to describe the knowledge and skills needed by experts working in preparedness at the national level such as national preparedness committee members or their equivalent. We recognise that the preparedness and emergency response structures of EU Member States vary, and use the term ‘national preparedness committee’ generically to describe the group of individuals who work on these issues in any given country. In general, however, public health preparedness is included in the emergency management structures in a country, which are usually led by interior affairs or civil protection staff. When an emergency is of a health nature, then health personnel typically take the lead.
While the PHEP logic model and competencies can be useful in preparedness planning, it is not ECDC’s goal to mandate any particular emergency response approach, structure, or capabilities. These are clearly the responsibility of the Member States and necessarily reflect each country’s own resources, administrative and political structures, as well as their responsibilities under Decision 1082 and the International Health Regulations (2005). Similarly, it is not ECDC’s intention to specify the knowledge or skills needed by any particular group of preparedness workers. Rather, our goal is to describe the competencies that different groups of professionals need – in addition to the basic knowledge and skills required for their profession – when they are called upon to serve as members of national preparedness committees or teams in order to facilitate an effective national and cross-border response to health threats in Europe.