Enhancing the Performance Testing Process for gRPC Model Inferencing at Scale

Explore the intricacies of performance testing for gRPC model inferencing at scale in this informative conference talk. Discover how to set up a Kubernetes cluster with KServe's ModelMesh for high-density deployment of machine learning models. Learn about load testing thousands of models and utilizing Prometheus and Grafana for monitoring key performance metrics. Gain insights into the complexities of model deployment, scalability challenges, and the features of Model Mesh. Delve into the automation of performance testing, including the setup of testing environments, QFlow pipeline, and K6 load tools. Witness a demonstration of the testing process, analyze testing logs and results, and understand the implications of cashmiss actions. Evaluate the benefits of using Model Mesh for your specific use case.