Measuring Audit Results

Why Statistical Literacy is Crucial for Auditors

Written by Joanne Byron, LPN, BS, CCA, CIFHA, CHA, COCAS, CORCM, CHCO, HPOC, OHCC, CMDP, ICDCT-CM/PCS

Information provided below is a basic overview of common statistical terminology used when Auditing for Compliance and not intended as being comprehensive, legal or consulting advice. Please consult a professional for more information regarding the importance of using statistical measures for your healthcare organization.

Why Is This Important When Software Generates the Statistics?

It is essential for chart auditors to understand statistical terms even when software generates the statistics because automated tools cannot interpret context, detect hidden biases, or make judgment calls regarding data quality.

While software speeds up the analysis of large datasets, an auditor's understanding of statistics is required to validate that the results are meaningful, accurate, and truly answer the audit's objective rather than just identifying coincidental correlations.

Advantages of Using AI - Artificial intelligence (AI) and software tools are transforming medical chart audits from infrequent, retrospective sampling into continuous, comprehensive, and automated processes. These tools primarily utilize Natural Language Processing (NLP) and Machine Learning (ML). AI integrates with electronic health records (EHRs) to analyze 100% of patient records continuously, rather than relying on limited retrospective samples, providing the ability to identify documentation gaps (such as missing signatures, late entries, unsupported coding), coding errors, and compliance risks in real-time.

Advanced, AI-enabled health information systems can now analyze raw, disparate data from Electronic Health Records (EHR)—including clinical notes, lab results, and patient-reported metrics—to automatically calculate complex health scores like Metabolic Equivalents (METs) and identify declining kidney function.

AI-driven tools identify patterns of documentation errors, allowing auditors to proactively manage risks related to payer audits and recoupments or situations which can trigger investigative external audits.

Human Review is Necessary

Statistical software works on the principle "garbage in, garbage out" (GIGO). Auditors must know if the data was collected properly, if there are missing values, and if the data is skewed, as automated tools may process flawed data without flagging it.

Auditors ensure data integrity – that the data accurately reflects the patient encounter or financial transaction before software analyzes it. Understanding statistical distribution helps auditors know when a simple "average" is misleading and when they need to look at the median or standard deviation.

Then there is the importance of contextual interpretation. Human auditors are better at interpreting the context and intent of a clinical note, such as differentiating "CTA" (clear to auscultation) from "CTA" (CT-angiogram) based on the surrounding narrative. Also, auditors can integrate information not explicitly written together in a single section. It is important to insert the human factor because auditors can spot subtle indicators or nuance not easily quantifiable by algorithms.

Risk Oversight - Ethical and Regulatory Accountability

Human oversight is crucial to prevent the "black box" problem where AI makes decisions without transparency, mitigating potential bias in automated audit systems. It prevents algorithmic bias and "automation complacency" where humans over-rely on AI.

By keeping human judgment in the loop, especially when auditing complex datasets or making critical decisions, the integrated process ensures the logic behind a decision is interpretable, transparent, and legally sound.

Importance of Statistical Literacy

Compliance should be the focus of all your audit functions. Measuring where you are now and improvements achieved is accomplished by applying statistics to understand data collected during the baseline audit and subsequent audits over time.

In healthcare chart audits, statistics are primarily used to summarize coding, billing and documentation compliance as well as clinical performance, identify variations in care, and determine if quality improvement (QI) initiatives are successful. These audits rely on both basic descriptive measures and more specialized tools for monitoring trends.

Auditors use descriptive statistics to summarize data and describe the basic features of a set of patient records. Instead of reading hundreds of individual charts, auditors use these "snapshots" to see the big picture—like how well a clinic is following safety rules or what the "typical" patient looks like. Common techniques include frequency distribution, percentages, and proportions to assess compliance with rules, regulations and reimbursement standards. Visual tools such as bar charts, histograms, and run charts analyze trends over time, providing a visual illustration of the data.

Key Statistical Measures Auditors Should Know

Statistics provide a "snapshot" of performance, helping to identify areas for improvement in clinical care, documentation accuracy, and compliance without making broad generalizations about the entire population. Here are the most common descriptive statistics used in healthcare audits explained in simple terms:

Finding the Middle or Central Tendency (the “typical”)

Used to find the average or typical value in audit data. Auditors use these to identify the most common or "average" value in a group of charts. These statistics help identify the center or "middle" of the data set. Terminology associated with central tendency are:

Mean (average): The average value, calculated by adding all values and dividing by the total count. The sum of all values divided by the number of cases. It helps identify the average performance, such as the average length of stay.
Median (middle value): The middle value, often used to avoid skewing data with extreme outliers, especially in run charts. For instance, let’s say you have 5 patients waiting 10, 15, 20, 25, and 100 minutes to see the provider. The mean is 34 ((10+15+20+25+100)/5), but the median is 20. The median is better for spotting typical patient experience when a few outliers (like the 100-minute patient) skew the average.
Mode (most common value in the data set): The most frequently occurring data point. The mode helps auditors identify anomalies. If a provider's billing pattern shows a "mode" that differs significantly from peers (e.g., almost all visits are coded as complex), it serves as a red flag for review.

Measures of Dispersion (Variability or Spread)

Measures of Dispersion (also known as variability or spread) in a chart audit tell you how consistent or scattered your data is. While the average (mean) tells you where the center of the data is, the dispersion tells you if most records are close to that average or wildly different.

In a chart audit, high dispersion often means high variability in clinical practice, which might suggest a need for better standardization (e.g., in documentation, timing of care, or drug dosages).

Standard Deviation (SD): Measures the spread of data; a small standard deviation indicates data is tightly clustered around the mean. – An example – if the average audit score was 90% with an SD of 5% means most charts fall between 85% and 95%. SD is the most common, precise measure, but best used when data is roughly bell-shaped (normally distributed).
- Low SD = Data is consistent (most nurses/doctors documenting similarly).
- High SD = Data is inconsistent (wide variation in practice).
Range & Interquartile Range (IQR): Identifies the highest/lowest values and the spread of the middle 50% of the data. Excellent for skewed data or when you have outliers, as it ignores the extreme top and bottom, focusing on the "typical" records. It is a robust method identify the "normal" range of data while excluding extreme outliers that might skew results.

In a chart audit, high dispersion often means high variability in clinical practice, which might suggest a need for better standardization (e.g., in documentation, timing of care, or drug dosages). In summary, dispersion tells you if your performance is reliable (low spread) or unreliable (high spread).

Frequency & Proportions

Frequency and Proportions are the two primary, simple statistics used to turn raw medical record data into actionable information. Frequency measures how often a specific event, behavior, or error occurs in a set of charts. It is a simple raw number or count. Proportions (often presented as percentages) measure the frequency relative to the whole. It tells you what part of the total population or sample had the characteristic, rather than just the raw count.

Frequency Distribution (Raw Count): Illustrates how often specific criteria are met. Frequency is simply counting how many times something happened. It tells you the total volume.
- Example: You audit 50 charts to see if doctors signed their notes. You find that 40 charts have signatures. The frequency? 40.
Proportion (Percentages): Used to define compliance rates (e.g., % of charts with documented allergies). Proportion puts that count into context by comparing it to the total. It tells you the "score" or the rate of success.
- The Formula: (Number of times it happened) ÷ (Total number of charts checked). Example: Using the same 50 charts, you take the frequency (40) and divide it by the total (50). The Proportion? 0.80 or 80%.
Why use both?
- Frequency is great for understanding workload (e.g., "We had 100 falls this month").
- Proportion is better for measuring quality (e.g., "Only 2% of our patients had falls").

If you check 10 charts and find 5 errors, the frequency is low (only 5), but the proportion can be horrifying (50%).

Conclusion

Compliance auditors must understand audit statistics to ensure their findings are defensible, accurate, and scalable. A firm grasp of statistical concepts allows auditors to identify high-risk patterns of non-conformance while minimizing the risk of "false positives".

Furthermore, when regulatory bodies like CMS or the OIG perform audits, they often use extrapolation to project error rates into massive financial recoupments; an auditor who understands the underlying math can effectively validate or challenge these high-stakes calculations

For more information, consider enrolling in the Auditing for Compliance online course. Tuition includes online, proctored certification to earn your Certified Healthcare Auditor (CHA^SM) credential.

About the Author

Joanne Byron, BS, LPN, CCA, CHA, CHCO, CHBS, CHCM, CIFHA, CMDP, COCAS, CORCM, OHCC, ICDCT-CM/PCS is an educator with Officer of the American Institute of Healthcare Compliance, a Licensing/Certification non-profit partner with CMS. She shares her experience of over 40 years as a nurse, consultant, auditor and investigator in the healthcare field.

References

American Institute of Healthcare Compliance