After Leeds pathology lab's technology failure, biomedical editor and author Brian Nation looks at the lessons to be learnt and Clinical Chemistry Operational Manager Jamie West explains lab information management systems.
Last September, The Leeds Teaching Hospitals NHS Trust witnessed what can happen when the computer systems on which modern healthcare relies break down. A major pathology computer hardware failure resulted in significantly reduced capacity to process samples and meant the trust was unable to report patient test results electronically.
Immediately following the crash, it became apparent to the Leeds team that finding a simple solution was going to be extremely challenging, and that ongoing pathology service disruption was likely.
It was agreed that the inability to provide electronic pathology results to the Leeds Teaching Hospitals, Bradford Teaching Hospitals and the respective GP communities was a business continuity incident. Therefore a meeting was convened involving the establishment of director-to-director contact between the two trusts and the local Clinical Commissioning Group (CCG).
Potential approaches
The meeting involved everyone who had any connection to, or interest in, the incident. Out of this meeting emerged four potential approaches, which were:
- Complete restore. At some later point this option was dismissed due to the inability to restore from an alternative back-up file.
- Rebuild the system. This would have required access to clean images of the different laboratory disciplines and a multi-agency approach.
- Use of order communication system. This would have reduced the reliance on the Leeds computer system and possibly have provided a means to distribute results.
- Send samples to other hospitals. In doing so, there would be reliance on their technology to distribute results.
In the main, the four approaches (with the possible exception of the last option) describe different aspects of disaster recovery, which may have helped the IT teams prioritise their work and maximise focus.
While some services were restored within a few days, others were disrupted for a number of weeks, and services to Bradford Teaching Hospitals NHS Foundation Trust and GP practices in Leeds and some parts of Bradford were also affected.
To ensure the safety of patients and for business continuity, the trust reverted to paper-based operating systems and postponed over 140 non-urgent planned procedures for patients.
Multiple disk failure
The disruption to services in Leeds was caused by an unusual occurrence of three individual disk failures over a short period of time, causing the system to fail, with warnings indicating imminent disk failures not identified. In addition, the system could not be restored after the disk failure, as there was an incomplete data back-up. This back-up failure had occurred over a period of time as a result of the increasing volume of system data.
Subsequently, an independent review was commissioned by the trust last October to ensure a fuller understanding of the circumstances surrounding the failure, and, more importantly, that lessons were learned from the incident in order to minimise the possibility of a future reoccurrence.
Applicable to all
While the recommendations contained in the review
of the Leeds IT failure were specifically aimed at the Yorkshire trust, they also have a generic value as a reminder of what can go wrong if best practice is not followed by those responsible for pathology computer management. The 10 recommendations in the report are summarised as follows:
- The Service Level Agreement that describes the managed service currently provided should be reviewed and strengthened, where required. The trust must be assured that the level of service covered by this and similar contracts is adequate and that they do not expose the trust to any unnecessary risks.
- All contracts with external service providers must contain a statement of compliance with regard to adherence with standard operating instructions covering the major aspects of work for which the third party is responsible/liable.
- The trust should work closely with the provider to seek confirmation that the successive failure of three disks in the array was indeed coincidental and not something more sinister that could reoccur in the future.
- Where possible, system back-ups should be part of a centrally managed process. Regular checks (daily) must be in place to ensure that back-ups are complete and are capable of running a full restore. This process should be extended to all of the critical systems across the trust, or, at the very least, all of the major departmental applications.
- A clean image of the current version of the operating system and application across all laboratory disciplines must be retained and stored off-site. This process should be extended to all systems across the trust, or at the very least the major departmental applications.
- A system must be in place with immediate effect to record accurately the the names of those who are entering any of the trust’s data centres, as well as all the activities that took place during access periods.
- Electronic alerting and warning systems must be in place in order to ensure that all those who have responsibility for data centres, system back-ups and electronic storage receive automated assurance that back-ups are intact and complete.
- Large departmental systems, their associated networks and data centres should be managed within the structure of the corporate information management and technology service. Members of the pathology IT team who spend more than 50% of their time managing information systems should equally move to the corporate service. This principle should also be applied to the other large departmental systems that have dedicated resources.
- The trust corporate IT team should work with the pathology IT team and primary care system suppliers to understand how technology can provide a more cohesive solution to recognise the relationship between practices and pathology services.
- In relation to the above, the trust should also re-examine any Memoranda of Understanding or reciprocal arrangements that it has in place with other NHS providers to take account of critical system and service failures in the future. Included in this review should be any system or service that relies heavily on technology.
Collaborative computing
In 2011, Leeds put together an ambitious IT strategy based around a clinical portal, using open source code. It is also at the centre of the Leeds Care Record, a secure virtual health and social care record that is another open source project that provides a shared care record for the city. As part of the National Programme for IT in the NHS, Leeds was due to deploy the Lorenzo electronic care record; however, delays mean it still runs a much older patient administration system.
While undoubtedly the computer failure and human error at the core of the Leeds catastrophe last autumn was a nightmare in reality – the solutions implemented and the progress made owe much to collaborative working that is a hallmark of the relationship between laboratories, the commercial sector and the wider healthcare sector.
Brian Nation Editor and author, specialising in biomedical science
Laboratory information management systems: an introduction
Laboratory information management systems (LIMS) allow the management of samples and data within the laboratory. Historically these have often been a central information system containing patient, sample and test data. But in the modern clinical laboratory there are often multiple interlinked information systems, which may include a central LIMS system, “middleware”, operating software for analysers, order communications software (which allows users to request pathology investigations and review the results) and quality control software. Laboratory systems often report into external software, for example in GP surgeries, or supporting specific areas of care, such as maternity or cancer.
Such technology has greatly improved the timeliness and quality of pathology services, and is now critical to delivering effective healthcare. But this does mean that IT systems are a vulnerability as well as a benefit to healthcare providers, and there have been a number of high-profile disruptions to IT systems in the UK which have adversely affected healthcare services. As such, the management of information systems is a critical responsibility of the laboratory.
The most recent edition of ISO 15189:2012 was the first to have specific clauses relating to the management of information systems, again emphasising the laboratory’s responsibility.
Here we aim to give a basic introduction to LIMS, demystify some of the jargon, and look at how an understanding of such systems supports laboratory accreditation, helps manage risk and maintains service continuity.
Systems configuration
Larger information systems are commonly configured in a client-server model, which involves a central server providing a database from which client terminals operate. These information systems communicate most commonly across networks either within or between organisations. The communication protocols for the systems are standardised, defining the rules upon which the communication is based.
The lab’s responsibility
As information systems become more prevalent and complex, the implementation and maintenance of software systems commonly involves a number of parties, including suppliers, IT departments, pathology IT leads and laboratory management and staff. Laboratory management teams do, however, have responsibilities relating to information systems. If the laboratory has devolved these responsibilities to another party, this should be documented in a formal agreement.
Requirements of the ISO 15189:2012 standard, which either directly relate to information management, or can be applied to information management, include the following.
Director responsibility
Included in the responsibilities of the laboratory is the requirement to “design and implement a contingency plan to ensure that essential services are available during emergency situations or other conditions when laboratory services are limited or unavailable”.
Contingency plans should be periodically tested, and are crucial in the kind of major IT incidents seen in the UK recently. Here the laboratory has three key responsibilities:
Reviewing vulnerabilities – laboratories should be aware of where their systems are hosted and what the vulnerabilities are. Disruption can occur to isolated pieces of software or across the whole or part of a network.
Establishing contingency plans for service disruption – plans should cover disruptions to individual systems and across networks, and should include both short- and long-term disruption.
Testing business continuity plans – plans to maintain services in the event of disruption should be tested and verified to ensure that laboratories can maintain critical services.
Equipment management
Laboratory software must also be considered under the requirements for the management of laboratory equipment (clause 5.3.1), which includes the following:
Selection and purchase of equipment – and its suitability for purpose to maintain the service.
Acceptance testing – verifying that the software has the required functionality prior to implementation. Such validation processes are the result of requirements for the MHRA regulation. There are a number of strategies for validation exercises, and the full scope of functionality of the information system should be covered.
Maintenance and repair – this may include back-up processes and the handling of faults and defects. If these aspects of information systems management are handled outside of the laboratory (for example by an IT department or third party) then an agreement should be in place for the expectations of each party.
General requirements
ISO 15189:2012 is the first to have specific requirements for the management of information systems. This includes requirements for the following:
Security and access – this includes processes for the management of access to data, the ability to enter or change patient data, and release the results of examination processes for reporting. In addition, access to make changes to the system should be restricted.
Verification of changes – as well as the requirement for acceptance testing prior to introduction to use, changes to systems should be subject to documented verification processes, which include authorisation prior to implementation.
Reporting to external systems – the laboratory should ensure that any software that reports to external software accurately reproduces reports.
The above is not a full description of the requirements for accreditation, or an exhaustive list of good practice. There are a number of alternative approaches to the management of information systems, with varying levels of input from the laboratory and support from IT teams and third parties. Laboratories should be aware of the processes and understand that included is a responsibility to ensure that critical services are maintained when key systems are not operational.
Jamie West Clinical Chemistry Operations Manager at Peterborough and Stamford Hospitals NHS Foundation Trust
