Balancing customer demands for faster, more personalized services with the need to comply with evolving regulations has put banks on an accelerated digital transformation. Banks must balance this speed with robust risk management as they underpin operations with new technology to make rapid decisions on credit, underwriting, KYC, etc.
Amidst this dynamic landscape, the American subsidiary of Japan’s largest financial institution, with its unique blend of Japanese cultural values and American innovation, sought to enhance its decision-making capabilities to maintain its competitive edge.
A siloed data landscape resulted in operational and compliance bottlenecks
The bank faced critical business challenges as its cloud data platform, Databricks, evolved. Manually managing the Databricks environment led to inconsistent workspace configurations and governance issues, which hindered the ability to make data-driven decisions. The loosely defined governance practices created issues in data quality, auditing, and metadata management, impacting reliability.
The absence of a standardized approach to organizing data layers and fragmented data quality checks made enterprise-wide data assurance difficult. Furthermore, the rising costs associated with scaling SQL endpoints for large datasets added financial strain. Lack of a consistent data modeling strategy hindered query performance and optimization, further impacting access to actionable insights.
Transforming the Databricks architecture for streamlined data management
The client recognized these challenges and partnered with Persistent to design a scalable, governed, and cost-efficient data management framework. We delivered a comprehensive data modernization strategy focused on automation, artificial intelligence (AI), and data best practices.
Starting with Infrastructure as Code (IaC) powered by Terraform, Persistent automated the creation of workspaces, access controls, and the bank’s Unity Catalog permissions, with monitoring and rollback features. This automation tripled administrative control and monitoring visibility, ensuring consistent governance across the organization.
We also developed a Data Management Playbook, tailored to the bank’s unique needs, to formalize best practices for organizing data layers, naming conventions, data modeling (normalized silver and star-schema gold), auditing, metadata management, and Change Data Capture (CDC) guidelines. This ensured 100% compliance with data governance policies, providing a reliable foundation for data-driven decision-making.
To address the limitations of Unity Catalog, Persistent integrated an AI-powered Data Quality Framework that delivered intelligent, scalable data checks, reducing manual validation by 70% and ensuring data reliability.
We shifted workloads to Azure to optimize cloud costs, especially analysis services and Cosmos DB for Postgres. These and refined architectural patterns reduced cloud storage and compute costs by 40%, optimizing the Databricks footprint while improving scalability and performance.
Improved compliance, speed, and decision-making
The Databricks transformation addressed technical challenges and delivered tangible business benefits, reinforcing the client’s ability to navigate the dynamic financial landscape with agility and confidence while fully adhering to regulatory guidelines.
The transformation enhanced data governance as standardized data models ensured consistent data quality, enabling faster and more reliable decision-making. The AI-driven data validation framework reduced the time and effort required for data quality checks by 70%, freeing up resources for more strategic initiatives such as new product development, personalized customer service, and strategic alliances.
The cost optimization measures, which reduced cloud costs by 40%, allowed the client to allocate resources more effectively. Improved query performance and data modeling strategies improved data processing, ensuring timely delivery of insights and helping the bank meet its objectives of serving customers better.
