Why s3fs Fails in AI/ML and How to Achieve Scalable POSIX Access Anyway

Explore the critical limitations of s3fs in AI/ML environments and discover how to build scalable POSIX-compliant storage solutions for Kubernetes workloads. Learn why mounting S3-compatible storage via s3fs creates major problems in real AI/ML applications, including crashes from incomplete writes, vanished checkpoints, inconsistent metadata, and unpredictable I/O latency when training with PyTorch or TensorFlow. Understand how to overcome these challenges by designing a distributed file system that maintains POSIX compliance while leveraging cost-effective object storage, rather than abandoning it entirely. Gain insights into architectural trade-offs, implementing POSIX compliance in user space, integrating with Kubernetes through CSI drivers and Operators, and establishing observability benchmarks based on real production AI training clusters. Examine practical solutions for platform engineers, MLOps professionals, and Kubernetes architects who need reliable, scalable storage for data-intensive workloads. This intermediate-level conference talk requires familiarity with object storage, file storage concepts, and basic Kubernetes CSI driver principles, presented by Rui Su from Juicedata at SNIA SDC 2025.