As the healthcare ecosystem pivots to value-based care, stakeholders must prioritize improving care outcomes across touchpoints. A leading provider of global medical technology that collected and stored data on surgical equipment needed access to strategic insights that could help uncover trends in patient outcomes, operational efficiencies, and the effectiveness of various medical devices. Analyzing this data could help the client identify areas for further innovation, optimize resource allocation, and enhance patient care.
However, the client faced significant challenges in accessing these insights due to a time-consuming and error-prone data ingestion process in its AWS-hosted data warehouse. These inefficiencies hampered the client’s ability to leverage data effectively, impacting the overall performance and reliability of its medical devices, ultimately inhibiting it from delivering care outcomes down the healthcare value stream.
Data ingestion issues become roadblocks to business insights
The client struggled with prolonged data transfer times, causing delays in accessing critical insights and affecting decision-making processes. The frequent errors and instability within the data pipeline not only caused service interruptions but also compromised data integrity, making it difficult to maintain a consistent and reliable data flow.
The existing data transfer process involved full table loads, which increased the time required to transfer data to the cloud-based storage service, Amazon S3. Additionally, AWS database migration service (DMS) jobs ran into frequent errors due to source tables refreshing during the ingestion process, leading to system instability and interruptions. This unreliability in the data pipeline not only caused delays but also compromised the integrity of the data, making it difficult to maintain a consistent and dependable data flow. Moreover, the need for manual monitoring and intervention diverted the team’s focus from more strategic tasks, ultimately hindering the client’s ability to leverage data for strategic innovation and maintain its competitive edge.
The client turned to Persistent, an AWS premier tier partner, to help streamline its data ingestion process to unlock critical insights and free the team bandwidth for strategic business decisions.
Unlocking insights with Persistent’s data expertise & AWS data modernization services
With a long-standing relationship that spans more than a decade, Persistent had deep knowledge of the AWS data modernization services that proved instrumental in streamlining the client’s data ingestion process. Leveraging several AWS services, such as Amazon EC2 for necessary compute services, AWS Step Functions for automated and orchestrated workflows, Amazon SES for notifications, and Amazon EventBridge for event-driven workflow triggers, we helped the client transform its data landscape to deliver nimbler, more relevant, and time-sensitive insights.
We started by configuring parallel table loading for DMS jobs, allowing multiple tables to be ingested simultaneously. This significantly improved throughput and reduced overall ingestion time.
With our expertise in transforming data, we modified the workflow to better handle DMS job failures. We designed a system to detect when a DMS job either failed or became stuck mid-execution. In such cases, it would automatically use the last failed table as a trigger to restart the step function, ensuring recovery and continuation without manual intervention. This freed up the team’s bandwidth for more value-adding tasks, such as staying on top of customer demands, allowing leaders to make informed decisions, and keeping ahead of the competition.
Accessible data, faster insights, improved operational efficiency
The implementation of parallel table loading significantly reduced the total time required for data ingestion, allowing the team to access and analyze data much faster.
The enhanced AWS Step Functions workflow introduced a level of automation that was previously lacking. The system’s ability to detect and recover from DMS job failures without manual intervention resulted in a more stable and reliable data pipeline. This reliability ensured that data was consistently available for analysis, further empowering the business to make informed decisions.
The overall outcome was a streamlined and efficient data ingestion process that not only improved operational performance but also allowed the team to focus on strategic tasks. This shift enabled the client to drive innovation and improve patient care outcomes.
