Harnessing AI to showcase the store of the future - AWS Boundless Store

As world leaders in on-demand cloud computing technology, Amazon Web Services (AWS) are helping retailers envision and enable the future of the retail shopping experience. The ‘Boundless Store’ concept showcases the power of Computer Vision using Machine Learning (ML) & Artificial Intelligence (AI) for forecasting stock, personalizing the shopping experience and allowing ‘frictionless’ shopping via an auto-detect basket.

The challenge

The demonstration area was designed to help retailers see how existing AWS services can be used to provide a cutting edge retail experience. In this case, enabling a seamless shopping experience for customers to be able to make purchases without having to interact with a physical checkout: simply take what you want and leave. The showcase demo included a shelf with embedded Computer Vision technology which detects items being taken from and returned to the shelf and adjusts a customer's bill accordingly. As an AWS Partner Network (APN) Advanced Consulting Partner, Daemon was tasked with helping to ensure this demo was up and running at the NRF Big Show in four weeks.

The scope of the project included:

Improving/rebuilding an existing object tracking and detection model which could consistently determine which products are on a shelf by processing a real-time video feed
Implementation of decision-making logic which could accurately determine when an item is added to or removed from the shelf, and increment a customer's bill accordingly
Build a serverless back end on AWS to process the shopping list data
Design, build and deploy a UI to reflect how both customers and store staff could interact with the shelf

Our Approach

In order for an automated retail system to understand where products are on shelves and how customers are interacting with them from moment to moment, a solid Computer Vision solution was required. To deliver this, the Daemon team took ownership of the full data pipeline for the Boundless Store Computer Vision system, from data collection and training of neural networks, all the way down to the ‘bare metal’ of a deployed Computer Vision system.

Along the way, the team produced:

An improved object detector for finding objects in a real-time video feed
A robust object tracking system for figuring out how those objects are moving over time
A decision-maker for deciding whether events like the shopping of a product have occurred.

Daemon applied machine learning best practice to measure the performance of different object detection learners. This disciplined approach allowed the team to explore the possible solutions with confidence and ultimately choose the approach for which there was plenty of evidence. Specific techniques included object detection learners, collecting new data-sets aimed at training robust and "generalisable" object detectors, all within Amazon SageMaker.

In the area of Computer Vision, the team created an object tracker that would not just understand the products on the shelf that the camera was seeing, but to follow them over time. This tracking capability provided an interpretable stream of information about objects in the store, allowing the system to make decisions about what is happening on the shelf so that those events flow through to the rest of the inventory management and shopping cart system. The aim was to build a robust, well-tested, and configurable Computer Vision solution, which was then deployed on the accelerated AWS DeepLens system.

By deploying all of the ML and Computer Vision code to smart cameras, the team was able to perform all image processing and associated logic on the shelf itself, keeping bandwidth requirements and cloud costs to a minimum. This 'edge processing' is crucial for an in-store solution to be viable at scale, where bandwidth and network reliability can be limited.

When an item was taken from or returned to the shelf, the cameras would trigger events which were sent to a serverless AWS back end. Using AWS IoT GreenGrass, AWS Lambda, Amazon DynamoDB and AWS AppSync, these events modified the user's shopping list to accurately reflect the items which the user had taken from the shelf. This was integrated into an AWS Amplify front end which would update the shopping list in real-time.

The UI also showed how this type of service could integrate into a broader retail solution, giving real-time recommendations to customers by using Amazon Personalize and helping store staff anticipate inventory problems before they happen using Amazon Forecast.

Outcome

The entire solution was delivered in under four weeks, in-time for demonstration at the NRF Big Show. The retraining resulted in significant improvement in the accuracy of detecting items on a shelf: the initial detector had an mAP (mean average precision, a standard measurement of object detector quality) of 41% when calculated against new test scenarios. Our final detector had raised this to 75%, high enough to consistently detect all objects present on the shelf with a high degree of accuracy. The images below show a comparison between the output of the original detector and that of the detector after retraining. The old detector would miss many of the objects on the shelf and sometimes incorrectly identify products. On the same shelf scene, the new detector detected all items correctly and with a high degree of certainty.

This also ‘generalised’ the object detector, meaning that it was better equipped to function in unseen environments - where lighting, camera angle, positions of products and the physical appearance of the shelf can be different - without the requirement for further data collection or model training.

The UI was also demoed to help attendees understand how the output of the cameras could be processed in real-time and provide customers and in-store staff alike with an enhanced retail experience.

Results

Improved object detector that could consistently detect all items on the shelf
Object detection accuracy increased by 81%
Object detection certainty increase from about 50% to over 90%
Less than 5 seconds total time from item removal to add to user shopping list
Full project was delivered in under four weeks