Best practice machine learning clusters for training Generative AI with Ray on AWS.

The Client

Phare Health develops AI technology for the back office to address the systemic challenges of healthcare finance. Their advanced AI tools streamline administration, maximise fair reimbursement, and unlock valuable insights from data, thereby improving hospitals' cash flow and coding quality. Their solutions transform unstructured data into actionable insights, providing a deeper understanding of hospital operations, particularly by providing AI-powered options for clinical coding.

The Challenge

A crucial part of Phare's AI-driven healthcare financial solutions is developing a secure, efficient, and scalable environment for their data scientists and machine learning engineers to work in training new Machine Learning (ML) models for processing healthcare data. These models are used by Phare's data ingest and transform system built in AWS, which brings in client data and processes it at scale. By processing clinical data with these intelligent ML models, Phare can connect information about what is happening in the clinic to hospitals' financial processing needs, saving costs and ultimately resulting in better patient outcomes.

ML Models are trained in a training environment with access to de-identified patient data.  De-identified patient data is that which has identifying characteristics of the patients replaced by randomised information. Because under some circumstances patient identities can be recovered (imagine for example being the only person with a particular illness in your region), it is still sensitive data and needs to be managed very carefully. As such, Phare is concerned to ensure that the training environment follows the same high standards of privacy, security, and auditability of their production data pipelines and user-facing product. A natural related requirement is to be HIPAA (Health Insurance Portability & Accountability Act of the USA) compliant. 

In addition to data security and privacy, Phare data scientists must be able to carry out machine learning training "at scale". In practice this means leveraging a large amount of computer power by deploying training "clusters" - many computers working in the cloud, coordinating the machine learning training together. This training is what produces the ML models to take data and, with the help of the cleverness of data scientists and machine learning engineers, create great models for processing patient data.

The ML environment data scientists and machine learning engineers are to work in also requires best practices for deployment of the solution as well as integration with existing infrastructure and expertise. Cloud infrastructure on AWS is deployed with Terraform and due to previous experience and its wide range of cross-cloud capabilities, Phare has opted to manage the cluster with  Python-based machine learning cluster software Ray, which comes with the ability to automatically scale cluster size according to demand and offers a variety of tools for machine learning at scale.

Our Approach

Daemon worked with Phare to determine where we could create the most impact and get Phare enabled on Ray as quickly as possible. Working closely with Phare's data scientists and consulting with Phare's in-house DevOps specialists, Daemon was tasked with ensuring Ray auto-scaling capability within AWS best practice security and optimisations of the Ray cluster. It's worth noting that hackers are known to have exploited Ray in the past, making it essential to apply best practice security across the Ray stack.

The technical decisions made are as follows:

• Using the Ray-provided Ray Autoscaling for on-demand cluster scaling, to get Phare enabled and scalable quickly.

• AWS SSM Session Manager to maintain network isolation and access control when accessing the instances, but EC2  Instance Connect where Ray lacks compatibility with session manager.

• EC2 optimisations and integrations for fast startup of cluster and work environments, including Packer EC2 AMI build.

• Closely applying and communicating expert AWS advice as encapsulated in their HIPAA whitepaper and well-architected framework.

• Advice from across data, AI/ML and cloud practices on optimal and secure ways to run data and ML workloads and recommendations for forward-compatible aspirational architectures from Daemon's principal data architects.

Below is the interim architecture that was achieved by the end of the delivery period. 

The Outcome

By implementing a secure and scalable training environment, we enabled Phare to train machine learning models using their preferred open source distributed computing framework while keeping data and compute locked down within AWS. The system allows Phare's data scientists to prototype and experiment with GPU-based instances with right-sized levels of access, perform distributed training with Ray and leverage autoscaling for efficient resource use.

Benefits

Using Ray on AWS with Daemon's support, Phare is ready to streamline and scale its AI development process, enhance the accuracy and efficiency of its financial health tools, unlock valuable insights from data for its clients while maintaining stringent security standards, and ultimately support its mission to improve hospitals' financial health.

Testimonials

“We are excited to share our experience with Daemon, whose exceptional support and expertise have been pivotal in setting up our AI infrastructure. Their team consistently demonstrated professionalism and clear communication throughout the project. By implementing secure and scalable solutions, Daemon has enabled us to train our machine learning models with sensitive healthcare data using Ray on AWS, ensuring full HIPAA compliance. Their thorough understanding of AWS best practices and Ray's auto-scaling capabilities have allowed us to continually optimise our workflows and improve the efficiency of our data scientists.

Daemon's contributions will significantly elevate our AI development process and allow us to scale our experimentation with GPU-based instances and perform distributed training more easily. Their commitment to security and performance ensures that our infrastructure remains robust and scalable. We anticipate that the foundation Daemon has built for us will enable us to unlock even more valuable insights from our data and enhance the accuracy of our financial health tools, ultimately improving the financial health of hospitals.”

Tymor Hamamsy (Cofounder)

Labels

• Health
• Finance
• HIPAAAI / Artificial IntelligenceGenerative AI
• AI-powered coding
• ML / Machine Learning
• AWS
• EC2
• Ray
• Python
  
• Nvidia
  
• Terraform