Orchestration of SAS Data Integration Processes on AWS
About this paper
This paper talks about the orchestration of SAS® data integration processes based on the arrival of the SAS data integration input files in Amazon Web Services (AWS) S3. Our client runs a daily process where they generate credit statements for their customers. Each customer receives their statement once in a month. Every day, around 200K customers are processed, eventually reaching out to their entire customer base of roughly 6 million in a month. The process starts in their on-premises datacenter, followed by the APR calculations in SAS data integration in AWS, and finally culminates with the generation of the statements in the on-premises datacenter. The entire architecture is built using microservices; small independent and highly decoupled components such as AWS Lambda, Simple Notification Service (SNS), Amazon Simple Storage Service (Amazon S3), and Amazon Elastic Compute Cloud (EC2). This makes it easier to troubleshoot any issues in the data pipeline. The choice of using a Lambda function for the orchestration adds a certain complexity to the process. However, it also provides the most stability, security, flexibility, and reliability for an enterprise architecture. There are simpler alternatives like S3Fs and CloudWatch SSM, but they do not fit well for an enterprise architecture.
About Core Compete
Founded in 2012, Core Compete is a pioneer in Cloud Analytics and is based in Durham, North Carolina with offices in Dallas, London and India (Pune and Hyderabad). We currently have 200+ employees and over 450 cloud and technology certifications, with highly specialized skills including: domain consultants, data scientists, and cloud engineers. Core Compete is an ISO 27001 and SOC 2 certified organization.
We partner with AWS, Google Cloud, Microsoft Azure, SAS, Snowflake, Hortonworks and Tableau to deliver successful cloud analytics and big data transformations for major corporations worldwide.