How custom software is hurting your hospital

Through the work that we do, we get to talk to a lot of hospitals about their IT and Decision Support initiatives, and their successes and failures. We’ve seen a pattern where hospitals are facing increasing problems with custom-developed software (internally created or by consultants) whether they are small data management applications or large data warehouses.

NOTE: Any custom application code, script, database, report, Excel macro, or analytical view that needs to be developed and maintained by the hospital is considered “software” within the context of this article.

SYMPTOMS OF THE PROBLEM

1. The majority of work being assigned to technical resources surrounds patching or fixing existing software rather than creating new value. As a result, new initiatives are not being adequately addressed.
2. There are increasing complaints from technical staff that they don’t understand the software that was developed by someone else and wind up wasting a lot of time trying to get up to speed. They often say the software was “poorly written”.
3. End-users lose faith in the reliability and data quality of the software and complain about increasing lead times to implement new features or fix bugs.
4. The hospital starts looking for “magic bullet” software to purchase, or “training” to solve their problems.
5. There is constant blame assignment, large unordered lists of parallel tasks assigned to people, staff turnover and/or low job satisfaction by the technical staff.

Once a tipping point is reached, it can have devastating consequences for a hospital and its ability to use its internal technical resources to help improve hospital operations. The result is an unwieldy heap of siloed software assets with no clear ownership or practical long term consolidation and management strategy.

WHY THE PROBLEM EXISTS

1. There are many facets of hospital operations and significant operational differences between countries (even provinces). Standardized software that address regional issues doesn’t always exist for the hospitals to purchase. EHR systems often ignore specific country/regional idiosyncrasies that need to be addressed. Also, consulting firms don’t have an abundance of properly trained software people with the requisite healthcare background.
2. In general, Healthcare IT is relatively immature, with inadequate attention to interoperability, reusable software/data assets, and proper data infrastructure. Hospitals are forced to operate within this broader environment. This excellent article by the NHS articulates many of these issues and offers a general strategy for addressing the problems.
3. Some hospitals have a policy of letting vendors do all the development work while others have a policy of doing everything internally because they focus on the cost of vendors but not necessarily their value. Both approaches have significant drawbacks and a mix is almost always better.
4. Employees and consultants are budgeted differently and external help often can’t be easily utilized without an onerous procurement process.
5. It has historically been very difficult for managers and team leaders to understand the difference between a highly-skilled professional software developer and a relative novice. “Years of experience” for a list of technical categories on a resume doesn’t begin to tell the full story and vendors routinely oversell their consultants’ abilities. Underqualified technical resources are often out of their depth and it largely goes unnoticed until too late.
6. There is a massive shortage of qualified developers in Canada that understand the intricacies of healthcare data, which is a significant hiring problem for both vendors and hospitals.
7. Stakeholders and managers may understand the business value of the software, they don’t understand the necessity and experience required to create maintainable software. Sometimes they don’t want to admit that what they’re doing is software development in order to justify shortcutting best practices and proper processes. They get into deep “Technical debt.“

Once a hospital gets into technical debt, the only solution is to take a step back and undergo the costly process of rework (refactoring). No executive wants to hear that new hospital initiatives are stalled because software that “works for the most part” needs a large amount of additional investment without any tangible benefit. This can begin a devastating downward spiral.

THINGS YOU CAN DO TO HELP

1. Manage risk. Small projects that build incremental value towards a larger, integrated infrastructure (designed by a reputable architect) have many advantages. Make it clear to vendors if they don’t meet their short term commitments, they’ll be replaced for the next phase. It forces a team to come up with a structured plan with a stepwise approach, discrete deliverables, and modular components. You’ll provide value sooner, identify issues faster and be able to adapt and recover. Large, monolithic projects where vendors take the helm over a long time without comprehensive and verifiable indicators of real progress is a massive risk.
2. Manage the process. Make sure that a qualified project manager oversees development and uses a proper task management process to keep a finger on the pulse of the project. It doesn’t have to be complex but it’s required to break large projects into independent components and it keeps people (both vendors and employees) accountable for deliverables and timelines. Also, find a development methodology (e.g. scrum, agile) that works best for your culture.
3. Have a clear delineation between what is done internally versus what is always outsourced, and budget for it. Some tasks should be done internally, as vendor support wouldn’t be practical. Some skills are not worth developing internally and this is a very tough pill for many people to swallow. Your average internal hospital IT resource deals with a massive breadth of different subject areas and will not develop the same technical skillset as dedicated consultants with decades of experience in complex, focused areas. It also reduces the amount of technical knowledge your team needs to do their work so they can focus and become more proficient in those areas.
4. Re-evaluate the actual value of technical training (especially boot camps). This is often an attempt to find that silver bullet to solve technical debt problems. From my experience, managers put far too much hope into the outcomes from external training for their teams. For technical training to be effective it requires regular and ongoing application of learned concepts, which is often overlooked. Also, it’s better to create cross-functional teams rather than try to have everyone be proficient in every aspect of all technologies.
5. Assign individual ownership for software assets to minimize excessive overlapping of training and effort. Keep track of assets and maintenance plans in a central location (even an Excel spreadsheet works). Outsource ownership and maintenance to vendors, where possible. This frees up your internal resources to do more important work and it’s keeps their list of tasks (and things to know how to do) manageable.
6. Make maintainability a priority. Hospitals generally care about the usefulness of the software and gloss over everything else. This leads technical staff to create solutions without including effort that leads to better software maintainability. This has disastrous long term, often irreparable, consequences. If total-cost-of-ownership analysis, documentation, deployment guides, regression testing plans, and ongoing support plans are not part of every initiative, you’re in trouble. You need to assign ownership and budget for ongoing costs of all custom software.
7. Consider breaking people into dedicated teams. Internal innovators often don’t know a thing about proper software development but they understand the problem space and make things happen. They create prototypes and minimal viable products that tease out great solutions. We don’t want to bog them down with the minutiae of making things production-ready, which should be the responsibility of another team. Trying to develop staff that are proficient at the entire lifecycle (jack of all trades) of software development is not optimal. Some people won’t like this approach but there are many different skillsets and a division of responsibilities is best for the hospital.  This is the same reason that developers and testers should never be the same people.

Ultimately the technology required to successfully deliver most hospital software projects has existed for a very long time. Most project failure occurs because of inexperience, unrealistic stakeholder expectations, vendors that over-promise, and woefully inadequate IT infrastructure in the province. A hospital is not a professional software development shop and likely doesn’t have the internal experience to properly create custom software tools. However, creating software is a necessity for the foreseeable future and cannot be entirely avoided.

These are incredibly difficult problems to solve and good leadership, great processes, proper risk management, and an ability to critically self-evaluate are the only real immediate solutions.