Configuration-Driven Reporting on Large Datasets Using Apache Spark

Explore a 27-minute conference talk from Databricks on configuration-driven reporting for large datasets using Apache Spark. Dive into the challenges of processing petabytes of financial transactional data and learn about American Express's next-generation reporting framework. Discover how this highly configurable enterprise solution enables the generation of hundreds of different reports for thousands of partners without additional development. Gain insights into dynamic scheduling, data transformation, aggregation, and filtering using Spark's in-memory and parallel processing capabilities. Understand the implementation of business rules and the integration of template engines like FreeMarker and Mustache. Examine the technical components, configuration file structure, and various stages of the reporting process, including schema application, data lookup, transformation rules, and template application. Assess the success metrics of this innovative approach to handling large-scale reporting needs in the financial industry.