Daemon worked towards Aligned AI's reinforcement learning proof of concept, bringing experience in AWS Sagemaker, Huggingface, LLMs, and the generative AI ecosystem.

The Client

Aligned AI is a startup operating at the bleeding-edge of research and industry, building solutions for AI alignment. A core offering is an API (Aligned API) that interfaces with in-production AI systems, monitors them, and suggests intelligent updates to auto-correct them should the need arise. Aligned AI is offering a safer, more precise and controllable AI, which currently has huge global appeal. Fortune 50 recently named Aligned AI as one of its 50 AI Innovators List.

The Challenge

With rapidly growing interest in their product, Aligned AI are looking at scaling massively in the near future. The AI models developed by Aligned AI rely on proprietary algorithms that are integrated into, among others, a niche form of deep-learning called Reinforcement Learning with Human Feedback (RLHF). 

AWS provides an array of relevant services, targeted at production-scale software, and the compute resources necessary to run resource-hungry AI algorithms.

Daemon was tasked with assisting Aligned AI in utilising AWS services, in particular, Sagemaker, to advance the training of models using Aligned AI's proprietary methods. We were able to simultaneously assist with the development of an early stage product, on a very tight schedule. Given the nascency of this field and type of work, there are few tools readily available in the broad marketplace today that enable this type of work. As a result, this challenge sits at the bleeding-edge.

Our Approach

While Daemon is comfortable with delivery-based engagements, in this case, the needed way of working was time-scoped online consulting and knowledge transfer with a tight feedback loop so that Aligned AI's expert personnel could benefit immediately. 

That is, Daemon provided their "AI/ML in AWS" expertise in just the way it was needed at this point in Aligned AI's AWS journey:

• Daemon facilitated the provisioning of in-demand NVidia A100 GPU-based Sagemaker notebook instances with 8 GPUs connected by high-speed NVLink, set up the development environment and debugged dependency and driver issues. 
• Daemon assisted with deploying parallel training (supervised fine tuning and reinforcement learning from human feedback) with TRLX, running across 8 GPUs and utilising the Huggingface Accelerate backend. 
• Daemon contributed toward an initial POC using DeepSpeed across multiple compute nodes.
  
  

The Outcome

By working with Daemon, Aligned AI realised their vision and implemented their planned RLHF training workload on AWS. Aligned AI is now in a position to exploit AWS's compute capabilities and services for research and as a springboard for their expanding product offering.

Testimonials

It was a pleasure to work with Daemon - with their help, we were able to effectively solve our large model training problems, deploy these models, and integrate our technology, with its unusual requirements, with all the standard tools in the area. 
- Stuart Armstrong, CTO, Aligned AI

Labels

• AWS

• Sagemaker

• ML / Machine Learning

• LLMs / Large Language Models

• RLHF / Reinforcement Learning from Human Feedback

• Jupyter + Conda

• Nvidia

• Reinforcement Learning

• Huggingface