Test it out: an online shopping demo experience with Gemini and RAG

Earlier this year, tens of thousands of developers gathered in Las Vegas for Google Cloud Next ’24, which culminated in hundreds of sessions and over 200 announcements. During the Developer Keynote, we showcased how Gemini can help with the shopping experience of an online store. Let’s dive into this demo and how it all worked in more detail!

Problem statement

Imagine you are the owner of Cymbal Shops, an online store with thousands of products. Each item has some metadata– a title, a description, a cost, and more. Traditional built-in search engines can easily handle queries relating to this metadata, but what if the shoppers are more interested in abstract questions such as “what piece of clothing pairs well with my outfit?”, or “what kind of furniture would complement my living room?”

This kind of user need sounds like a great use-case for integration with a large language model (LLM) that can process both text and images, such as Gemini!

But “wait a minute…”, I hear you say. An LLM is trained on an extensive amount of datasets that are not necessarily the collection of products that Cymbal Shops sells, and you quickly notice that the results are fairly generic and irrelevant:

This is where Retrieval-Augmented Generation (also known as RAG) comes in.

Retrieval-Augmented Generation (RAG)

RAG improves the accuracy and relevance of LLM outputs by augmenting (adding onto) the input prompt with relevant pieces of data stored in an external database or search index. This can involve for example embedding the prompt and documents into a shared vector space, then using similarity metrics to rank the pieces of data by relevance.

When a new user prompt is received (in our case, when the user talks to the shopping assistant), we embed it in the same format as our online store’s inventory (a series of numerical values) which allows us to retrieve the most likely relevant items, which we then include as part of the original prompt to the LLM.

In this scenario, Cymbal Shops was already using a PostgreSQL database to store product information, so we will leverage AlloyDB, Google Cloud’s 100% PostgreSQL-compatible database, to also store vectorized products. Leveraging Vertex AI’s built-in RAG capabilities may be preferable in other scenarios.

Without RAG, the prompt sent to Gemini might have looked like this:

With RAG, the prompt generated now looks like this:

Notice the difference?

Let’s deploy this new version of Cymbal Shops and try the same user prompt (“Suggest furniture pieces that pairs well with this room”) alongside the same living room space photo:

The shopping assistant now only recommends real products from the Cymbal Shops inventory!

What’s next?

Try out the demo and see how Gemini can improve your customer experience for yourself! The Cymbal Shops sample application is open source, where you can find the instructions to deploy this demo.

Learn more about how you can leverage the power of RAG in Google Cloud:

• What is Retrieval-Augmented Generation (RAG)?

• Infrastructure for a RAG-capable generative AI application using Vertex AI

• Build enterprise gen AI apps with Google Cloud databases