Review and Reporting Software

QORM Consultants have on various occasions assisted clients with so called interim controls. These are 3rd party reviews of batch records, deviations, validations, and change controls. In most cases such reviews are part of a regulatory settlement, but sometimes it is also done on a voluntary basis to alleviate concerns by regulatory agencies about the suitability of material and systems.

The value of such reviews is obvious: A 3rd party is reviewing the records and ensures that, once the review is done and all observations by reviewers are are addressed, will withstand regulatory scrutiny.

However, the value of such reviews can be multiplied if the learnings of the review are communicated clearly to senior management, repeat issues are identified at their root cause and processes are improved such that they are more reliable and possibly also accelerated as has been described in another case study here. In this case study we describe QORM's proprietary Review and Reporting Software focusing on batch record review, only. The software, however, can and has successfully been used for other types of 3rd party reviews.

High Level Review Process

Review Process

Screen Shot of Review and Report DB

This screenshot shows the overview for one particular batch record. The observations can be viewed individually in a popup window.

The categorizations of observations are used create statistics to highlight which part of batch record execution and review pose problems

On a high level, the 3rd party review process is quite simple (see above). The client is giving documents to be reviewed to the consultants. Any issues are communicated to the client, who then rectifies the issues. The consultant in turn makes some record that the reviewed record now is compliant and returns the record to the client for archiving

QORM Consultants have seen many instances in which this process was handled in Excel spreadsheets. While certainly possible, Excel quickly reaches its limits in this application. For instance, storing record identifying information and multiple observations in one spreadsheet are nearly impossible. Tracking the progress on resolving observations, publishing statisitcs on observation types and handling multiple users almost invariengly will result in chaos.

QORM's proprietary Review and Reporting Software handles all of these tasks and more.

Record Handling:
- Check-in and check-out of record to 3rd party review group
- Record location management from issuance to floor through QA to 34rd party review group to archive using bar codes
- Report on length-of-stay in different review stages
Observations:
- Handling of multiple obsevations for each record
- Assignability of who made obsevations and what is expected to resolve them
- Communicating individual observations with stakeholders
Reporting:
- Providing statistics and reports in reusable formats (e.g., Excel)
- On-demand reporting
Product release:
- Link record trees (and deviations, etc.) to releasable product and report on readiness of product
User experience
- User friendly web based access
- Multi-user access with role based access (e.g., client is allowed to view observations and type in responses, 3rd reviewer allowed to view responses and type in observations)

System Interface Design

The system is highly customizable to ensure the right fit for the client environment. It is implemented in MS Web Access on QORM's Sharepoint site. Obviously, each client has its own site that is completely isolated from that of other clients.

By using QORM's Review and Reporting database, clients realized the following benefits:

For one client we were able to reduce the number of consultants from 16 FTEs to 10 FTEs.
For another client the number of FTEs required to manage corrections for batch records decreased from 6 to 2 FTEs
Statistics developed for and delivered to the client shaped a major training program hat improved Right-First-Time from mid 50% level to over 90% level.
For one client we were able to reduce the time to prepare 3rd party review related product release activity from 2 days to 2 hours using 1 person rather than 2.
For that same client the 3rd party review completely disappeared from the critical path for product release.
At one client the number of documents or parts of documents lost per year dropped from an average of 12 to none. No material had to be destroyed after starting to use the database.

Screen Shot of Review and Report DB

This screenshot shows how trees are constructed within the database. In many cases the trees can be imported from client sources, however, at one client actually the database was used to construct and maintain the tree.

Review Findings

The primary goal of external (and internal) review may be to ensure the compliance of documentation of manufacturing or laboratory processes. However, when applied properly, the review findings can be used to better understand recurring error sources and ultimately improve the execution and review of documentation.

For instance: For Batch Record Review (BRR), QORM Consultants have developed a 3 prongue framework to categorize errors found in the records:

Documentation errors, such as omitting dates or sigantures, calculation errors and the like
Operator errors: Erros where operators on the floor (or lab technicians) have omitted steps or not executed them exactly as per procedure.
Issues showing that operators and other parts of the organizations identified an issues but then did not properly address it. Such issues are typically represented in unclear margin notes

The importance of these categories is that their repeated occurence would point to different weaknesses in the organization and therefore require different approaches to remediation.

Examples for typical findings sorted by category described on the left.

Statistics

Screen Shot of Review Statistics Available to the Client On-Demand

The Review and Reporting system provides detailed statistics about many aspects of the review, from throughput of the review, times in various review stages (which can extend all the way back into the organization). The most important statistics are arguably those that are showing progress with respect to improving execution and review activities by the organization.

QORM Consultants found that the statisitcs are best communicated through on-demand Excel files, making use of already existing capabilities at the clients. These same statistics could also be disseminated through tools such as Power BI or other dashboard software.