The Sched app allows you to build your schedule but is not a substitute for your event registration. You must be registered for KubeCon + CloudNativeCon North America 2023 to participate in the sessions. If you have not registered but would like to join us, please go to the event registration page to purchase a registration.
Please note: This schedule is automatically displayed in Central Standard Time (UTC -6). To see the schedule in your preferred timezone, please select from the drop-down menu to the right, above "Filter by Date." The schedule is subject to change.
Like many tech companies, Honeycomb released a feature using an API backed by a Large Language Model (LLM) this year. However, unlike most APIs, those that call LLMs are non-deterministic and inherently unreliable. So how do we know that it's doing what we need? What are the different factors that matter to our users, and how can we measure them? Subtle changes to a prompt in an LLM can have huge changes on its behavior, so how do we understand the impact of our prompt changes? We asked all these questions and more during development, and we think we have a good way to answer them through careful instrumentation and Observability practices. In this talk, I'll go through how we instrumented our feature, what we tracked, what SLOs we set up, and how we measured our improvements as we iterated on the feature. Attendees should come away with a good idea of how they can blend the worlds of prompt engineering and Observability to build better products.
Phillip is on the product team at Honeycomb where he leads their AI initiatives and works on a bunch of different things. He's an OpenTelemetry maintainer -- chances are if you've read the docs to learn how to use OTel, you've read his words. In a past life, he worked on developer... Read More →
