Application of recently developed computer algorithm for automatic classification of unstructured radiology reports: Validation study

PURPOSE: To validate the accuracy of Lexicon Mediated Entropy Reduction (LEXIMER), a new information theory-based computer algorithm developed by the authors for independent analysis and classification of unstructured radiology reports based on the presence of clinically important findings (F_T, where _T represents "true") and recommendations for subsequent action (R_T). MATERIALS AND METHODS: The study was approved by the Human Research Committee of the institutional review board. Consecutive de-identified radiology reports (n = 1059) comprising results of barium studies (n = 99), computed tomography (n = 107), mammography (n = 90), magnetic resonance imaging (n = 108), nuclear medicine (n = 99), positron emission tomography (n = 106), radiography (n = 212), ultrasonography (n = 131), and vascular procedures (n = 107) were independently analyzed by two radiologists and then with LEXIMER to categorize the reports into F_T and F _T-0 (containing or not containing clinically important findings) categories and R_T and R_T-0 (containing or not containing recommendations for subsequent action) categories. Accuracy, sensitivity, specificity, and positive and negative predictive values of LEXIMER for placing reports into F_T and F_T0 and R_T and R _T0 categories were assessed by using appropriate statistical tests. RESULTS: There was strong interobserver concordance between the two radiologists for placing radiology reports into F_T and R_T categories (K = 0.9, P < .01). For the LEXIMER program, accuracy, sensitivity, specificity, and positive and negative predictive values, respectively, were 97.5% (95% confidence interval [Cl]: 96.6%, 98.5%), 98.9% (95% Cl: 97.9%, 99.6%), 94.9% (95% Cl: 93.1%, 96.0%), 97.5% (95% Cl: 96.6%, 98.0%), and 97.7% (95% Cl: 95.8%, 98.8%) for placing radiology reports into F_T and F_T0 categories and 99.6% (95% Cl: 99.2%, 99.9%), 98.2% (95% Cl: 95.0%, 99.6%), 99.9% (95% Cl: 99.4%, 99.99%), 99.4% (95% Cl: 96.3%, 99.9%), and 99.7% (95% Cl: 98.9%, 99.9%) for placing reports into R_T and R _TO categories. CONCLUSION: LEXIMER is an accurate automated engine for evaluating the percentage positivity of clinically important findings and rates of recommendation for subsequent action in unstructured radiology reports.