Technology is transforming healthcare by providing clinicians with new diagnostic tools and life-changing treatments. Whilst these interventions might grab the headlines, one of the main burdens that doctors face is more mundane: doctors are spending 50% of their time filling in electronic medical records. To help solve this problem, the IT and Digital Innovation team at Leiden University Medical Center (LUMC) is using AI to reduce the amount of time doctors spend on administrative overhead, and give them more time to treat patients.
“As we’re a research facility, education faculty, and provide care to patients, LUMC is in a unique position,” Marjolein Elderhorst, Manager of the Digital Innovation team at LUMC explains. “We have a clinical focus, and work closely with physicians and patients to make sure the solutions we develop answer their needs.” Doctors at LUMC asked the team to find a way to reduce admin during the initial patient-doctor consultation (anamnesis), so they could focus on their patients and what they are saying. The team also wanted to structure the record data to make it easier to research, and make the recording available to the patient.
To do that, the team looked for a way to input conversations automatically. As well as being able to record and transcribe consultations, the team wanted it to be secure and easily replicated in other departments. Working closely with Cloud Technology Solutions and Google Cloud LUMC found a solution to this challenge.
“LUMC focusses on value-based healthcare and delivering a good experience for patients,” says Marjolein. “Our solution should be scalable and secure. That’s why using cloud technology is the ideal choice for us.”
A good Speech-to-text solution - structured and non-verbal
When developing tools to automate manual processes, it’s important to make sure that the outputs are not only accurate, but also fit for purpose. The digital innovation team at LUMC faced a number of challenges when building a tool to handle anamnesis consultations. “In order to be useful, the text needs to be structured, and also account for non-verbal communication, such as nods of the head,” Simone Cammel, AI Specialist in the Digital Innovation team at LUMC explains. “We also need the structuring of the text, so it’s available for future data analysis.”
Together with Cloud Technology Solutions, the digital innovation team built a back-end infrastructure on Google Cloud to process anamnesis recordings. When a recording is made, it’s stored in Google Cloud Storage. It’s then automatically transferred to Google App Engine by Google Cloud Functions and Google Cloud Pub/Sub, and pushed to Google Cloud Speech-to-Text for transcription.
To structure the transcriptions into a conversation and recognize where a non-verbal response had taken place, the team developed custom algorithms. ”Developing the algorithms was quite some work and took us several months. We are happy with the successful use of Google Cloud Speech-to-Text with customized solutions,“ says Simone.
Once the transcript has been created, Google App Engine transfers the data back to Google Cloud Storage. It can then be accessed by the physician through the front end, running on Google Firebase.
“The physician receives the transcribed answers to each of the anamnesis questions they asked: how long has the patient been unwell, are the symptoms worse at certain times than others, and so on,” says Enno de Haan, ML Team Leader at Cloud Technology Solutions. “The physician can add comments, and then copy the text into the electronic patient record, a final step that we plan to automate in the near future.”
“We looked at other solutions, but we went with Google because we find the level of recognition of medical terminology to be very good. We can also add specific medical vocabulary into the dictionary,” says Simone.
Collaborating with Cloud Technology Solutions on a scalable infrastructure
LUMC’s collaboration with Cloud Technology Solutions has been an important factor in the project’s success. “Google advised us to work with Cloud Technology Solutions, because they are experts in creating Machine Learning solutions and have a lot of experience in the medical sector. We’ve been working together for over a year now,” says Simone. However, the actual infrastructure was implemented much faster. “In close collaboration with the team at LUMC, we were able to put the architecture in place within around 10 weeks,” explains Enno.
Once the solution was ready, the team began recording fictional conversations with colleagues to see how well it functioned before introducing it to clinicians to test in the real world. As the data gathering process is complex and requires a lot of attention, the team at LUMC uses Google Cloud to provide a reliable infrastructure. “I don’t have to think about it, I know the infrastructure is there,” says Simone. “That’s very helpful, as it accelerates what we do. I can focus on the challenges that matter.”
The final step will be to develop a mature solution that can be implemented and used in daily practice at LUMC, and has the potential to scale up. “In the future, we have the ambition to roll out the solution can be rolled to other LUMC departments and can be used to improve healthcare in general,” says Marjolein.
Creating healthcare solutions for the 21st century
Following initial tests, the tool has already had a successful trial run with patients. “The physicians are pleased with the initial results,” says Marjolein. “Around 200 patient interviews have been recorded so far, and 800 more will be recorded in the coming months. We now have a department of 10 physicians who are keen to help us develop it further.” Also, patients approached by LUMC for the trial all agreed to have their conversations recorded and their data stored in the cloud. Google Cloud is GDPR-compliant, which is important for LUMC in terms of keeping patient data secure. “The patient always comes first at LUMC,” she explains. “Calls are never made available to a third party. Google Cloud provides us with a service where all data remains the property of the patient, and is never analyzed by any other party.”
In the next phase, the tool will be fine-tuned before it can be put into use with data from real electronic health records. The team also plans to develop a diagnostic function, so it won’t just transcribe the anamnesis, but also analyze it and offer suggestions to the doctor in real-time.
The digital innovation team at LUMC is also working with Cloud Technology Solutions to develop new solutions for other challenges, such as a digital assistant to help surgeons access radiographic images during surgery. “Rather than taking off his or her gloves or requiring a nurse to do it, the surgeon will be able to use their voice and pull up any diagnostic information they need,” explains Simone.
“We believe that healthcare should answer the needs of the 21st century,” says Marjolein. “We want to provide our patients with the best possible care, as well as playing a leadership role in the Netherlands and beyond. Together with Cloud Technology Solutions and Google Cloud, we can work on developing digital tools to keep innovating healthcare.”