The Metrics of Quality Culture
The long awaited, anxiously anticipated FDA guidance on quality metrics is now finally distributed for comment. The official title of this guidance for industry is Submission of Quality Metrics Data Guidance for Industry. As a result, many executives believe it will continue to increase the conversation around Quality Culture and help accelerate best practice.
The intent of FDA to establish quality metrics first emerged when Congress passed the Food and Drug Administration Safety and Innovation Act (FDASIA) enhancing FDA’s capability to proactively react to, prevent, and alleviate drug shortages. Specifically, Title VII Section 705 of the Act states FDA “shall inspect establishments described in paragraph that are engaged in the manufacture, preparation, propagation, compounding, or processing of a drug or drugs.”
Section 706 of the same act allows FDA to request certain information from companies in advance of or in lieu of inspections. The guidance goes on to state “records or other information that the Secretary may inspect” under this section. Technical experts involved with “the manufacture, preparation, propagation, compounding, or processing of a drug shall” will be responsible.
The intent of FDA to establish quality metrics . . . to proactively react to, prevent, and alleviate drug shortages.
In the Federal Register Notice, FDA asked the industry to assist with “drafting a strategic plan on drug shortages” per the FDA Safety and Innovation Act. This notice asked a series of thought-provoking questions that invited collaboration from across the industry.
Questions like “What metrics do manufacturers currently use to monitor production quality?” offered feedback from companies of all sizes. Another question – “How frequently would such metrics need to be updated to be meaningful?” – invited an internal discovery on when assessments should be conducted.
After a years of actively engaging with the industry in a variety of venues, this new guideline has helped shape a new analysis on quality culture.
The culture of a company dictates the veracity of their metrics.
The metrics proposed in the guideline are not new to the pharma industry. Many of them are currently being used by companies to internally measure performance. In some cases, the specified metrics are also reported to the agency via annual report or annual product review. The biotech and pharmaceutical industry needs to ensure meaningful data continues to be provided to measure success.
Defining Quality Culture
The underlying and understated tenet used to determine a company’s well-being is a measure of their quality culture. The culture of a company dictates the veracity of their metrics. The best way to ensure the data has merit is to assess the quality culture of the submitting organization. It is in this area that the FDA guidance lacks clarity.
The guidance leaves the opportunity open to establish quality-culture metrics by stating, “these metrics are not intended to be an all-inclusive set”. As innovation grows, there will be new and useful metrics to assess a product and manufacturer’s state of quality.
For example, management commitment to quality is an evaluation factor for the overall health of the PQS [pharmaceutical quality system]. The industry dialog to share benchmarks and standards could provide more streamlined metrics from management team to team. A robust strategy would demonstrate the commitment to measurability while inviting these new evaluation factors.
Quality Culture Metrics
FDA proposes three voluntary metrics to try to get at the elusive quality culture. The first optional metric proposed is intended to measure senior management engagement by assessing whether the head of the quality unit and the head of the operations unit have signed the annual product review (APR) or product quality review (PQR).
The second optional metric proposed is corrective action and preventive action (CAPA) effectiveness. The measurement for this metric is to indicate the percentage of corrective actions that required retraining of personnel.
The assumption for retraining is the root cause of original deviation was determined to be insufficient or ineffectual training. Another proposed metric is intended to measure a firm’s process capabilities through a checklist series of questions. The real question should be if these new metrics, optimized together, can shed new light on the quality culture.
Achieving a quality culture requires an environment where responsibility, accountability, and reliability are paramount. Each employee understands the role they play in delivering a high-quality product to the customer. Each function recognizes that sustaining performance over time means process improvement and training.
Management must educate employees and provide the tools where employees perform in an atmosphere that encourages excellence. Assigning the head of quality and the head of operations the task of signing the APR or the PQR does not ensure management engagement nor does it mean that the quality culture is lacking.
It is up to an organization to establish the appropriate level of responsibility and signing authority for APR and PQRs. It is up to senior management to provide the people charged with these activities the necessary resources to complete the task in a timely manner with the expectation that they will be held accountable for the contents.
The second optional quality culture metric is specific to CAPA. The proposed metric is to report the percentages of corrective actions involving the retraining of personnel. Without context supporting the retraining of personnel, this metric does not offer insight into the true culture of an organization.
It could be argued that any CAPA that results in a reduction or elimination of a recurring deviation would require an element of training personnel. For instance, retraining of personnel on the CAPA issue, how it was solved, and implementing change shows management engagement. It should be expected that a majority of CAPAs involve some retraining of personnel.
Critical quality attributes
The third quality-culture optional metrics involves trying to use critical quality attributes (CQA) as a key indicator of a quality culture. Of the three optional metrics proposed, this one does provide some measurement of the existence of a quality culture.
On the surface, the questions just seem to be a regurgitation of information contained in the APR or PQR. Upon closer evaluation, however, it is clear that FDA is trying to measure whether a company drives for continuous improvement through their review and assessment of threshold levels established with CQAs.
Companies that have established CQAs and linked them to a requirement to issue a CAPA when they exceed the established threshold levels have demonstrated a commitment to continuous improvement. Continuous improvement programs are, in fact, reliable indicators of the presence of a quality culture.
The establishment of simple quality metrics don’t simply measure the quality of the product. Metrics set a baseline for the quality culture required to assist FDA in establishing a risk-based audit program.
The problem is that it can be difficult to measure culture with cold, hard data. The remaining question is: If taken together, are the proposed optional metrics indicative of a quality culture? The answer is subjective and ongoing.
Careful thought and consideration should be exercised when determining what to measure and how often to measure. Driving positive change involves how to interpret and communicate the data to the team. Likewise, clarity on interpreting expectations for using the data also drives positive collaboration.
As a result, management needs to be cognizant of the fact that whatever metrics are reported, they must be developed, evolved, and adjusted over time to maximize their impact on driving positive change.
Careful thought and consideration should be exercised when determining what to measure, how often to measure, how to interpret and communicate the data, and what the expectation is for using the data to drive positive change.
When choosing a metric, it is important that the architects of the metric are aware of unintended consequences that may inadvertently drive negative behavior. Management attempting to incentivize achievement of the goal such as offering a financial award if the goal is achieved, may lead to inappropriate behaviors that do not address the real issue.
In these cases, it is generally not the metric that will drive the behavior but rather use of behavioral rewards. Reward for achievement rather than analysis of the real underlying causes will not lead to sustainable positive change. When managed properly, metrics are an important tool to help drive positive change and quality process improvements.
Upon observation, an unhealthy quality culture is easy to identify. People in a poor culture do not understand their job and its importance to the business. They often appear stressed, and they hide their mistakes or blame others for their errors. Likewise, there is no evidence of teamwork. People work in silos and rarely, if ever, seek input or advice from others.
One metric that could be potentially used to assess a poor culture include employee turnover. Deviations due to human error may demonstrate an employee’s lack of pride delivering a quality performance. For instance, deviation training may be reconsidered if the employee is not a quality culture fit in general.
Alignment of Goals
In contrast, a robust, healthy quality culture can be evidenced by alignment of goals between quality and operations. Work teams who focus on continual improvement promote the attitude of incorporating quality into their daily jobs. Above all, they are not afraid to offer suggestions for improvement to their colleagues or about the process.
People understand the importance of their jobs and respect each other and their management. This culture views these inspections as a tool in their continual improvement initiatives. Metrics used to measure a healthy quality culture include a small employee turnover, deviations that identify a root cause other than human error, and employee pride in their performance.
The metrics chosen must be meaningful and written to provide a clear analysis of ongoing activities.
When establishing a metrics program, companies should evaluate numerous data input points including, but not limited to, product-quality attributes, manufacturing site performance, people metrics, and quality-system metrics. For product-quality metrics, companies should consider reporting on batch-specific data. Most importantly, this includes trending drug product, drug substance, and stability test results against complaint rates.
Indirect product-quality metrics should include environmental monitoring, water trend results, and yield rates. When establishing site metrics, the company can look at inspection history including internal audit findings and maintenance history such as equipment age versus defect-failure rates. People metrics should consider ongoing job-specific training and education. Skills and experience assessments should be measured against employee turnover rate. Analyzing the turnover rate by job function and site can show quality success or failures.
Quality systems metrics will analyze change control, investigation root-cause trends, and release-testing cycle times. There is no set requirement on which metrics a company should track to measure their overall performance.
Each company should determine which metrics to track based on their operations, number of facilities they operate and where they are located. Above all, what types of products they manufacture and the existing culture is foundational.
The metrics chosen must be meaningful and written to provide a clear analysis of ongoing activities. It is important for operations and quality to agree on how to report them to management to avoid overreaction to the data. It is not sufficient to simply report the data. The interpretation of the data is of crucial importance because it may include a root-cause analysis of its own.
Additional Blog Updates
(1) FDA, Request for Quality Metrics Guidance for Industry, (Rockville, MD, July 2015).
(2) The Food Drug Administration Safety and Innovation Act, Pub. L. 112-144, 126 Stat. 993 (2012).
(3) FDA, “Food and Drug Administration Drug Shortages Task Force and Strategic Plan; Request for Comments” (Federal Register Feb. 12, 2013).