Patient Safety in the Operation Room: Quantitative Article Critique

Introduction

In the appraisal of research articles, there are certain criteria that must be present in order to correctly ascertain whether or not the research results are valid. According to Polit and Beck (2012), one must follow certain procedures in quantitative studies in order to assure validity. This is important because contrary to qualitative studies, quantitative studies strive to generalize the results of their study to the population stated Polit and Beck. Having a structured set of questions to see if all of the criteria are fulfilled helps the reader to knowledgeably assess an article for content and validity, as well as assist a researcher to ensure that all essential elements are present. This critique will assess an article from van Beuzekom, Boer, Akerboom, and Hudson (2012) about safety in the operating room. The questions to be answered will be stated and the responses will be provided. While there were many strong points in Van Beuzekom et al.’s study, there were some points where the authors could have elaborated a bit further.

Questions for Critiquing Research Problems, Questions, and Hypotheses

What is the research problem? Where is it located? How clearly is it stated? How well does the problem statement build a cogent and persuasive argument for this study?Van Beuzekom et al. (2012) stated that the research problem is the amount of OR errors occurring due to systemic faults. They state it clearly in the beginning, establishing the problem of errors due to systemic factors and then addressing the problem of OR errors contributing to a significant portion of the rate of errors in a hospital setting. They clearly establish the need for the study, to reduce these errors, because they are preventable errors if the systemic issues are addressed.

Describe how well the authors have established that the problem has significance for health care. How might the research contribute to nursing practice, administration, education, or policy? Van Beuzekom et al. (2012) did a very good job at establishing the problem of operation room (OR) safety. They provided statistics showing that half of all safety related problems were due to operational errors performed in the OR, and that the mistakes that occurred were due to systemic problems, such as understaffing. There were also issues with perioperative nursing changes. Polit and Beck (2012) stated that it is important to establish a clear problem in quantitative research, and that quantitative research should aim to propose and perform a procedure or protocol that will address the problem, and that the results of applying the procedure to the problem can be measured in numerical data to demonstrate effectiveness or ineffectiveness.

As Van Beuzekom et al. (2012) stated, their research demonstrated the effectiveness of an educational and training system upon OR errors. Polit and Beck (2012) stated the importance of research making a contribution to the field of study, focusing upon producing valid studies proving or disproving the effectiveness of a procedure or protocol.

Discuss the presence or absence of formal statements of purpose, research question, and/or hypotheses within the report. How clearly and concisely is this information communicated, and how a logical and useful is its location in the report (for the reader)?The purpose is clearly stated under its own section. The research question is not delineated, but one can infer from the paragraph describing the goals of the research what the research question would be. According to Van Beuzekom et al. (2012), the goal of the research is to see if using a questionnaire to drive a training program addressing systemic issues in care would be effective. Their hypothesis is clearly stated. They hypothesized that directly addressing latent risk factors (LFRs) would lead to improved rates of OR error.

Describe the appropriateness of the wording of purpose statements or questions. For example, discuss whether the authors have identified key concepts/variables along with their population of interest?The purpose statement is clearly delineated in Van Beuzekom et al.’s (2012) study, as described above. The hypothesis stated above does not clearly delineate the role of the questionnaire and subsequent training program.

If there are no formal hypotheses, is their absence justified? If there are hypotheses, do they flow from a theory or previous research? How justifiable are the predictions?The hypothesis is clearly stated in Van Beuzekom et al.’s (2012) study. A clear statement of the dependent and independent variables is always advantageous to producing valid results within a study, according to Polit and Beck (2012). However, the basic hypothetical statement is there and identifiable, and they do draw a clear line from the statement of the problem (high OR error rates), and their solution to the problem (discovering through a questionnaire the problems, and directly addressing those problems in the OR). Because of the clear line of logic, the predictions are justifiable.

Questions for Critiquing Design Elements and Study Validity in Quantitative Studies

Was there adequate statistical power?In van Beuzekom et al.’s (2012) study, the p values were recorded and showed a significant difference between the beginning values taken with their survey (LOTICS survey) and after the interventions were administered. A t-test analysis was used to compare the beginning test scores and end test scores for the survey, and an ANCOVA was used to analyze the covariate factors (such as age and years of experience).

Discuss whether the manner in which the independent variable was defined and operatíonalìzed and how this allowed strong contrasts that enhanced statistical power?In Van Beuzekom et al.’s (2012) study, the independent variables were clearly defined and operationalized - the use of the LOTICS survey to detect systemic LRFs present in hospitals in the Netherlands and the application of training to directly address the issues detected after the application of the survey. The t-test analysis revealed that the results were significant. However, as Polit and Beck (2012) emphasized, even when results are significant or demonstrate statistical power, it does not prove the effectiveness of the protocol, because the results could have been due to an error, or chance. One can say that the results of Van Beuzekom et al.’s (2012) study demonstrates a correlation of administering of the survey and training program and decreased perceived OR error.

How did the investigators control confounding variables and did this enhance precision?As stated above, the authors performed an ANCOVA to control for confounding variables. Their analysis revealed that factors such as age and years on the job did have an effect upon the results. Their analysis helped the authors understand how those variables affected the results of the study.

If the study had an intervention under the control of the researchers, describe evidence (if avaìlable) that attention was paid to intervention fidelity. For example, did they discuss staff training and was this adequate? Was the implementation of the intervention monitored? Was attention paid to both the delivery and receipt of the intervention?Van Beuzekom et al. (2012) the intervention was fully controlled by the researchers, and they presented adequate evidence that they controlled those factors, although not all factors were fully explained. For instance, they did not describe the training provided to the staff. The focus of their discussion was upon the results the training produced. They also did not discuss any details about their intervention or the monitoring of the progress, or delivery or receipt of the intervention. Their discussion was very strong in terms of discussing the results, and the results were significant, but the reader is not provided with a discussion upon the training or measures provided to address systemic issues affecting OR performance. The authors could have made their study stronger by providing a full description of the training measures.

What evidence does the report provide that selection biases were elimìnated or minimized?Van Beuzekom et al. (2012) used purposive sampling to obtain their participants, ORs in the Netherlands. According to Polit and Beck (2012), this is used when conducting a study when the population is a group of experts, and usually, this does not involve a discussion on sampling procedures, including eliminating biases. The fact that the study does not explain how they eliminated biases does not hurt the sampling procedure, since they need to target ORs and their personnel for the study. According to Polit and Beck, it is common for a discussion of the sampling procedure and elimination of biases to be lacking for this sampling technique.

What steps were taken to control confounding participant characteristics that could affect the equivalence of groups being compared? Were these steps adequate?While Van Beuzekom et al. (2012) did an ANCOVA to test for the effects of participant characteristics upon the results of the study, they did not control for confounding participant characteristics, nor was it necessary. The authors went through great measures to establish that individuals should not be blamed for OR errors. A more productive approach was to address the systemic conditions of the OR within the hospitals that contributed to OR personnel error, taking blame away from the individual actions of the OR personnel. The authors did a good job in assessing how factors such as age and years of experience affected the results of their study, but it wasn’t necessary or desirable to control for those factors because then it wouldn’t have replicated real life situations. If a study is altered to the point of not reflecting real life situations, this can negatively affect the validity of a study and the generalization of the findings, according to Polit and Beck (2012).

To what extent did the study design rule out the plausibility of other threats to internal validity, such as history, attrition, maturation, and so on?van Beuzekom et al.’s (2012) study design naturally lent itself towards protection against internal threats to internal validity, which Polit and Beck (2012) define as research design construct. However, in Van Beuzekom et al.’s (2012) study, the experimental group was younger and had less experience than the control group. However, the authors seemed to control for this variable, which they found to have a significant effect upon the results.

What are your overall conclusions about the internal validity of the study?The authors did their best to control for threats to internal validity and did a good job overall. They could not help that one group of OR professionals were younger and less experienced than the other group of OR professionals, but the ANCOVA analysis helped them understand those effects and they were able to control for those factors in their results.

Were there any major threats to the construct validity of the study?To hedge against construct validity to the study, van Beuzekom et al. (2012) used the LOTICS, an instrument having demonstrated reliability and validity. According to Polit and Beck, using valid, reliable instruments of measurement help to protect studies against threats to construct validity.

In intervention studies, was there a good match between the underlying conceptualization of the intervention and its operationalizatìon?van Beuzekom et al.’s (2012) study goals were consistent throughout the study. The instrument utilized was efficient in carrying out the goals of the study: to identify LRFs so that an intervention specific to the problems experienced by the OR staff of the intervention group were addressed specifically. The result indicated that their intervention made a difference in a number of rates, such as turnover rate and the amount of perceived OR errors before and after the intervention. As Polit and Beck (2012) stated, when a hypothesis is proven, it speaks to factors such as strong research design and less of a chance errors occurred in the study. Intervention conceptualization and successful operationalization are proven with successful data results as it was in this study.

Was the intervention “pure” or was it confounded with extraneous content, such as researcher expectations? The intervention was not confounded with bias for van Beuzekom et al.’s (2012) study. The intervention administered was first assessed using a valid instrument with a Likert-type rating scale. Afterwards, the intervention was based upon the results of the assessment instrument (LOTICS). The measurement for power of the t-test results, the before and after scores of the LOTICS, concluded there was an effect. The effects were conducted using constructs and instruments that are reliable and valid, minimizing the effects of bias.

Was the setting or site a good example of the type of setting envisioned in the conceptualization?van Beuzekom et al. (2012) operationalized their study about OR safety in two OR settings, so the answer is yes.

Was the context of the study sufficiently described to enhance its capacity for external validity or generalizability?van Beuzekom et al. (2012) sufficiently described their study in a manner that demonstrated the generalizability of their results. For instance, the problems associated with systemic OR errors, such as a nursing shortage in the OR, was demonstrated to be an industry-wide issue. The researchers provided interventions to help retain OR employees and reduce employee turnover, which, if the researchers had explained their interventions in more detail, could be applied to more hospitals around the world if employee turnover was identified as a problem in OR errors.

Were the settings or participants representative of the types to which results were designed to be generalìzed? The settings and participants were representative of any OR setting.

Guidelines for Critiquing Data Collection Plans in Quantitative Studies

Was collection of data using structured methods (in contrast with unstructured methods) consistent with study aims? Polit and Beck (2012) stressed the importance of using valid and reliable data collection instruments to obtain data for analysis. van Beuzekom et al. (2012) use of the LOTICS instrument to test for an effect of their interventions measured specifically what they needed it to measure to achieve their study aims and prove the validity of the hypothesis.

Were the right methods used to collect the data (self-report, observation, etc.)? Should supplementary data collection methods have been used to enrich the data available for analysis?Yes, the use of a survey in van Beuzekom et al.’s (2012) study was a good way to collect the data to prove their hypothesis. This study was very strong in its results and reporting the data. A way to improve upon the data available for analysis was not immediately apparent.

Discuss the validity of the research instruments or measures, in terms of congruence with underlying constructs, data quality, reputation, efficiency, and so on.The LOTICS, according to van Beuzekom et al. (2012), is an instrument that has proven reliability and validity: “The LOTICS has been validated with respect to factor structure and reliability of the scales, as well as its content and discriminative validity” (p. 3).

The authors described the data collection methods in sufficient detail. It was understood that the procedures were conducted according to those stipulated in the LOTICS. The only information not provided was who was in charge of the data collection and their training to administer the test and the training, which are factors to consider in assessing a research study and the validity of its findings. The findings of the report could have been emphasized if the researchers included their training on the LOTICS and who provided the training sessions after the administration of the first round of the LOTICS test.

Conclusions

The study by van Beuzekom et al. (2012) was strong in many regards. They provided adequate detail regarding their research methods, the analyses of the data, and proving their hypothesis. However, the study could have been improved with inclusion of more information such as who administered the LOTICS, what was their training and did they have to have training on the LOTICS, and other details listed above that could have made the case for the validity of their study even stronger. Overall, this was a good study with sound internal validity and construct, sound, valid research methods, and the use of valid and reliable instruments used to collect their data. Without the questions to guide the analysis, it would have seemed like a great study without any missing components. The questions were a good guide to find the missing components, and are a great guide to use when constructing one’s own research design to ensure that all possible points are conveyed to the reader so validity of the study is not in question.

References

Polit, D. F. & Beck, C. T. (2012). Nursing research: Generating and assessing evidence for nursing practice (9th ed.). Philadelphia, PA: Wolters Kluwer Health/Lippincott Williams & Wilkins.

van Beuzekom, M., Boer, F., Akerboom, S., Hudson, P. (2012). Patient safety in the operating room. An intervention study on latent risk factors. BMC Surgery, 12(10).