Data Engineer

job Description

Are you a Data Engineer with the expertise to orchestrate symphonies of insight from complex datasets? If you revel in transforming raw numbers into a golden tapestry of innovation, then we invite you to join us on an exciting journey. At Mukuru, we are in search of a Data Engineer extraordinaire to wield their magic wand and shape the backbone of our data architecture.

Role Overview:
The primary objective of this role is to prepare and manage large volumes of data for analytical and operational purposes. As a Data Engineer at Mukuru, you will be instrumental in designing, building, and maintaining the architecture used for data processing and storage.

Duties and Responsibilities

-Data Modeling and Design: Crafting efficient, scalable data models optimized for diverse storage and retrieval needs.
-Data Pipeline Construction: Building and maintaining robust data pipelines ensuring seamless data flow from various sources to storage systems, often involving ETL processes.
-Database Management: Managing and optimizing databases, ensuring speed, reliability, and security, with proficiency in SQL and NoSQL databases.
-Big Data Technologies: Utilizing cutting-edge technologies like Hadoop, Spark, and Kafka for large-scale data processing.
-Automation and Optimization: Automating data processes and optimizing data flow to enhance efficiency and reduce errors.
-Ensuring Data Quality and Governance: Implementing measures for data accuracy, consistency, and security while adhering to data privacy laws and regulations.

Duties and Responsibilities:
-Providing DataOps support to Analytics Engineers, overseeing AWS services, and managing CI/CD processes for deploying DDL to a Snowflake Data Warehouse.
-Managing data extraction from various source systems hosted on AWS RDS and EC2 instances.
-Maintaining and optimizing the extract, stage, delta load staging process for efficient data flow and integrity.
-Transitioning to and maintaining an Operational Data Store (ODS) using Change Data Capture (CDC) for data ingestion across multiple source databases.
-Collaborating with cross-functional teams to understand and meet data requirements, ensuring the data infrastructure aligns with organizational needs.
-Ensuring data security and compliance with financial industry regulations and best practices.
-Troubleshooting and resolving issues related to data systems and processes.
-Continuously evaluating and implementing new technologies and tools to enhance data systems and processes.

Qualifications and Experience

-Minimum of 3 years of experience in Data Engineering, preferably in a financial services environment.
-Strong experience with cloud data warehousing, including schema change management and DDL deployments (Snowflake CDW preferable).
-AWS Solutions Architect Associate certification is preferable.
-Proven experience in managing AWS services, including RDS and EC2, in a professional setting.
-Solid understanding and experience with CI/CD processes and tools, specifically related to data operations.
-Proficiency in SQL and familiarity with programming languages such as Python or Java.
-Experience with data modeling and warehousing, particularly in a Snowflake environment.
-Familiarity with Snowpark, Snowflake's native integration with Spark, or similar technologies.
-Experience with CDC-based data ingestion tools, particularly Debezium.
-Knowledge of data security and privacy standards specific to the financial industry.