Quality Metrics

Susan Schniepp, Distinguished Fellow at Regulatory Compliance Associates^®, discusses the quality metrics journey in determining the suitability of pharmaceutical products.

 

 

One of the most discussed and debated topics on today’s pharmaceutical landscape is the issue of quality metrics. Establishing, maintaining, and interpreting quality metrics to determine the suitability of pharmaceutical products has become a high priority for the FDA. To understand the issue of quality metrics it is important to start at the beginning.

 

FDASIA

 

The first stop on this journey took place in 2012 when Congress passed the Food Drug Administration Safety and Innovation Act (FDASIA) which enhanced FDA’s capability to proactively react to, prevent, and alleviate drug shortages. This direction is embedded in the language contained in Title VII—Drug Supply Chain and Title X—Drug Shortages.

 

Specifically, Title VII Section 705 of the Act states FDA:

 

“…shall inspect establishments described in paragraph that are engaged in the manufacture, preparation, propagation, compounding, or processing of a drug or drugs (referred to in this subsection as ‘drug establishments’) in accordance with a risk-based schedule established by the Secretary”.

 

QA Risk Factors

 

This section also describes the risk factors considered in establishing the inspection schedule. One of the risk factors listed in this section is:

 

“Any other criteria deemed necessary and appropriate by the Secretary for purposes of allocating inspection resources.”

 

Section 706 of the same act authorizes FDA to request certain company information in advance of or in lieu of inspections by stating

 

“Any records or other information that the Secretary may inspect under this section from a person that owns or operates an establishment that is engaged in the manufacture, preparation, propagation, compounding, or processing of a drug shall, upon the request of the Secretary, be provided to the Secretary by such person, in advance of or in lieu of an inspection…”

 

In addition, Title X section 506C-1 (Annual Reporting on Drug Shortages). Title X section 506C-1 requires FDA to annually provide Congress “a report on drug shortages…”

 

Quality Metrics Examples

 

The second part of the journey occurred in the Federal Register Notice. In that notice, FDA asked the industry to:

 

“assist the Food and Drug Administration in drafting a strategic plan on drug shortages as required by the Food and Drug Administration Safety and Innovation Act…”

 

This notice asked a series of thought-provoking questions about more specific quality metrics including:

 

“What metrics do manufacturers currently use to monitor production quality?” and “How frequently would such metrics need to be updated to be meaningful?”.

 

The industry reaction to this information was varied. Many trade organizations responded to the questions in the Federal Register. Some prepared white papers while others held meetings to discuss the issue with their members. The general consensus was that industry needed to seriously engage with FDA to define which quality metrics would provide information to the agency supporting efforts to eliminate drug shortages and to be used in establishing a risk-based approach to inspections.

 

FDA Quality Metrics

 

The third leg of the journey was the issuance of a draft guidance titled Request for Quality Metrics. In between the issuance of the Federal Register and the release of this original draft guidance there were many industry comments submitted to the agency suggesting various metrics that might be employed by the agency.

 

In addition, industry trade organizations held conferences and seminars to discuss the issue with the stakeholders and agency representatives. A number of these trade organizations published white papers based on the proceedings from these conferences and put forth the position that the need to submit metrics to the agency was redundant because of the requirement to submit annual product reviews. The agency listened to industry and some of the feedback was incorporated into the guidance.

 

Quality Performance Indicators

 

The quality performance indicators that were chosen as metrics for the updated guidance include:

 

• Lot Acceptance Rate
• Product Quality Complaint Rate
• Invalidated OOS Rate
• Annual Product Review
• Product Quality Review
• On Time Rate

 

The guidance also contained three optional metrics intended to measure quality culture:

 

• Measuring Senior Management Engagement
• CAPA Effectiveness
• Process Capability/Performance

 

This version of the quality metrics guidance generated 83 comments from industry submitted to the docket. A number of these comments focused on the practicality of submitting the requested metrics. In response to this concern the agency issued the Quality Metrics Technical Conformance Guide: Technical Specifications Document that explained how companies would be expected to submit the data to the agency.

 

Quality Management

 

The fourth part of our journey came when the second version of the guidance, titled Submission of Quality Metrics Data Guidance for Industry was released for comment. The docket for commenting on this most recent version of the guidance closed. There were 25 comments submitted. 12 from trade associations, 10 from individual firms, one from a hospital group, one from an academic institution and one from a private citizen.

 

The guidance clearly stated the

 

“…metrics can be also useful to FDA: to help develop compliance and inspection policies and practices…. improve the agency’s ability to predict, and therefore, possibly mitigate future drug shortages; and to encourage the pharmaceutical industry to implement state-of-the-art, innovative quality management systems for pharmaceutical manufacturing.”

 

Quality Metrics

 

The FDA also clarified the goal of the metrics program by stating

 

“As described in this guidance, FDA is initiating a voluntary reporting phase of the FDA quality metrics reporting program.”

 

This current version of the document requires companies to report on only three metrics:

 

• Lot Acceptance Rate
• Product Quality Complaint Rate
• Invalidated OOS Rate

 

The optional metrics intended to measure quality culture were removed.

 

Total Quality Management

 

These proposed metrics are not new to the pharmaceutical industry. Many of them are currently being used by companies to measure their total quality management performance. In order to be successful, companies need to review and analyze the information the FDA is asking for, as well as other metrics they are collecting, and identify potential problem signals. Quality leadership teams should look to solve issues and self-correct before regulatory inspections occur.

 

The problem with the proposed metrics, however, is that they are lagging indicators of performance. There is no set requirement on which metrics a company should track to measure their overall performance. Each company should determine which metrics to track based on their operations, number of facilities they operate and where they are located, what types of products they manufacture, and what type of culture exists in their places of business. Any metrics chosen must be meaningful and written to provide a clear analysis of ongoing activities.

 

Quality Assurance Management

 

It is important for operations and quality to agree on any type of quality assurance metrics. Additionally, how to report the results to management is as important as the results themselves. The interpretation of the data is a crucial element because it may include a root-cause analysis of its own and may help to promote continuous process improvements.

 

When choosing a metric, it is important that the architects of the QA metric are aware of consequences that may inadvertently drive negative behavior. Management attempting to incentivize achievement of the goal such as offering a financial award if the goal is achieved may lead to inappropriate behaviors that do not address the real issue. In these cases, it is generally not the metric that will drive the behavior but rather use of behavioral rewards. Reward for achievement rather than analysis of the real underlying causes will not lead to sustainable positive change.

 

Quality Control Management 

 

When managed properly, metrics are an important tool to drive positive design controls and process improvements. Regardless of what metrics a company chooses to measure their performance, achieving a quality culture is important in assuring reported quality control metrics are accurate and reliable. A quality culture requires management and employees to establish an environment where responsibility, accountability, and reliability are paramount. This helps each person deliver a high-quality product to the customer and sustained performance over time.

 

Management must educate employees and provide the tools and environment where they can perform their functions in an atmosphere that encourages excellence and continuous improvement. Continuous improvement programs are, in fact, reliable indicators of the presence of a quality culture. Where the industry’s and the agency’s destination on this journey ends is yet to be seen. What is evident is that companies must have a robust quality metrics program so they may continue to supply quality products without interruptions to patients.

 

Planning Quality Management

 

Companies need to keep in mind that when planning a quality management program, they should evaluate numerous data input points. This would include, but not limited to:

 

• Product-quality attributes
• Manufacturing site performance
• People metrics
• Quality-system metrics

 

For product-quality metrics, companies should consider reporting on batch-specific data such as trending drug product, drug substance, and stability test results against customer complaint rates.

 

Internal Audit

 

Indirect product-quality metrics could include environmental monitoring, water trend results, and yield rates. When establishing site metrics, the company could look at inspection history including internal audit findings and maintenance history such as equipment age versus defect-failure rates.

 

People metrics should consider ongoing job-specific training and education, skills and experience assessments, and employee turnover rate by job function and site.

 

Root Cause

 

Quality systems metrics might look at change control, investigation root-cause trends, and release-testing cycle times. Companies that can successfully establish a robust metrics program that helps them continuously improve should have no trouble meeting the FDA metrics requirements.

 

To begin the Regulatory Compliance Associates scoping process today, please enter your information in the blue form below and click the submit button at the bottom of the webpage. You may also email us at [email protected].