KubeCon + CloudNative North America 2023

Generative AI technologies such as LLMs are accelerating the pace of innovation. However, using such models come at a steep computational cost and is a major barrier to entry for many businesses and researchers. This talk proposes a novel approach to improve LLM efficiency, for complex reasoning tasks, by composing containerized LLMs as an inference decision tree called Tree Of Thoughts (ToT) and orchestrating it using various K8S ecosystem projects. ToT is an inference framework in combinatorial problem space, where each node represents a specific train of thought. This approach improves inference efficiency by refining prompts from the initial inputs through ‘conversations’ between models. All this sounds abstract, so using CNCF projects such as Operator Framework, Redis, Neo4j, and KEDA you will learn how to build a concrete ToT implementation. With real time visualization of graphs of intermediate LLM prompts and thoughts, you will see how ToT on Kubernetes enhances the computational efficiency of LLM inferencing.