By James Ferguson
Extract, Transform and Load (ETL) is a common method for taking disparate data from other systems and transforming then pushing to another system. We see this method used in many industries and processes repeated daily. However, the ETL process when using the cloud has not always been so straight forward.
You could argue there are many great tools such as Informatica, Decisions, and others, however, not many for cloud-specific ETL by the big three until recently.
ETL is used for the following primary functions:
- Extract data from databases
- Transform data using Merge, concatenate and other functions.
- Load data into new database or data services for consumption
All three cloud providers offer ETL tools now and have varying degrees of options and abilities. The following is a sample overview.
AWS just launched its new tool called “AWS Glue” which runs python code in the background while giving you a powerful tool for moving and transforming data across services. Using a data catalog and getting data locally as well as from existing stores in AWS.
Microsoft Azure has an “Azure Data factory” and works great at moving data from data centers to the cloud and back as well as across its varied services. They have also added some ties back into agents and VS for developers to make this integrated with other tools.
GCP support varied methods of getting data to and from its cloud offerings with services like “Cloud Dataflow”, “Big Query” and third-party vendors such as Matallion and others.
Regardless of which cloud provider you use or all three, you have many choices in the ETL space from which to choose.
Choosing the right cloud partner to help you with additional concerns around security, data encryption and data services architecture is crucial. Taos has extensive, practical, real-world experience partnering with our clients to implement cloud best practices. We have partnered with a vast number of companies from startup to enterprise across multiple verticals. Our assessments, recommendations, and execution enable smooth migrations, managed support and optimization. Our staff is skilled across the entire technology lifecycle and we look forward to solving your challenges with you.