SOUTHWORKS helps CODELCO introducing best DevOps practices to their data pipelines using Databricks

SOUTHWORKS x CODELCO | Case Study

Context

CODELCO is a Chilean copper mining company that is the world’s leading producer of copper mines, a leader in mineral reserves, and a driving force behind Chile’s development. It collects vast amounts of data through their extensive operations spanning seven mining divisions, plus smelter and refinery operations.

The CODELCO team had developed several Extract, Transform, and Load processes (ETLs) based on Azure Databricks for near real-time big-data processing. However, production releases posed challenges, and to maintain their core data flows, ultra-high uptime and advanced near realtime data processing, the market-leading company was looking for a partner to improve the ongoing efficiency and stability of their data pipelines.

‍

So CODELCO turned to SOUTHWORKS

Microsoft introduced SOUTHWORKS, a Microsoft and a Databricks partner, to CODELCO. SOUTHWORKS’ extensive experience in Data and AI and their partnership with Databricks made them the ideal team to collaborate with CODELCO’s data and development teams to achieve their objectives.

‍

“SOUTHWORKS has been an exceptional partner in understanding the opportunities in our data platform and helping us implement relevant enhancements, which enabled us to improve time to market indicators for our advanced analytics projects, accelerating our digital transformation. We appreciate their contributions and highly recommend their services to optimize data operations”.

‍Alberto Laudadio, Head of Data Management, CODELCO

‍

‍Solution

The CODELCO team wanted to improve critical key performance indicators (KPIs) such as deployment frequency, lead time for changes, mean time to restore (MTTR), and change fail rate. Additionally, they wanted to eliminate code duplication, introduce code accessibility, and implement additional control mechanisms to avoid disruptions when introducing enhancements and updates to the core data flows.

One of SOUTHWORKS Fireteams got to work right away, working side by side with CODELCO’s data and development teams to conduct architectural assessments, provide guidance, make code refactors in critical data pipelines and implement additional DevOps proven practices.

In order to achieve CODELCO’s quick strategic wins, embed best-practice, and build the foundation for future business acceleration, SOUTHWORKS focused on these key areas:

‍

Code duplication:

‍Refactor of Databricks Notebooks with a focus on eliminating code duplication, which helps to reduce the complexity of the ETLs and make them easier to maintain.

‍Code accessibility:

‍Refactor of the codebase, establishing the base structure, a blueprint that CODELCO’s team applied later to more repositories. SOUTHWORKS’ Fireteam introduced the use of Databricks Repos and implemented pull-request and branch policies to enhance change management.

Automation:

Addition of CI/CD pipelines to automate the deployment process and eliminate the need for manual intervention.

Unit testing and code coverage:

‍Introduction of unit testing to detect any possible errors in the codebase and improve the overall quality of the ETLs. The team also established a minimum threshold for code coverage to ensure that all critical areas of the codebase were thoroughly tested.

‍Static code analysis:

‍Introduction of tooling to detect errors early in the development cycle to ensure that the ETLs meets CODELCO’s coding standards.‍

Secrets management:

‍Introduction of secrets detection to ensure that no sensitive credentials could be exposed, while helping CODELCO’s team on using best development practices to store and retrieve them.

‍

‍

"With SOUTHWORKS we achieved better project structures, which enabled enhanced CICD workflows. We adopted some of the industry's best practices and defined a data pipeline gold standard that has operated as a model for the rest of the pipelines. Also, we made an important step towards an optimal DevOps culture and its subsequential benefits".

‍Paulo Ignacio Paillacar, Lead Data Architect, CODELCO

‍

‍Results

Performance Improvements: SOUTHWORKS’s assistance enabled CODELCO to improve performance in several critical areas. The project resulted in a higher deployment frequency, a shorter lead time for changes, a lower change fail rate, and a reduced mean time to restore(MTTR) – allowing CODELCO to provide an excellent experience for their end-users.

‍

Efficiency gains: By introducing automated pipelines and better code management practices, SOUTHWORKS was able to reduce the amount of time CODELCO spent on manual intervention, improving the overall efficiency of data pipelines. The implementation of unit testing and code coverage also contributed to the gains inefficiency, as potential issues are detected earlier in the development cycle.

‍‍

Improved quality and security: SOUTHWORKS also introduced additional control mechanisms to avoid disruptions when introducing enhancements and updates to the core data flows.The implementation of static code analysis and secrets detection helped to detect security risks early in the development cycle. These measures helped to improve the quality and security of CODELCO’s data pipelines, which is crucial in today’s data-driven world.

‍

--

‍

Ready to take your data pipelines to the next level?

Contact SOUTHWORKS today to learn how our expertise in Databricks and DevOps can help you improve efficiency and stability in your data management. We’ll help you achieve your quick strategic wins and accelerate your business.

‍