Calling all Data Engineers who transform numbers into symphonies of insight!
If you're passionate about weaving technological wizardry to tame the wildest data streams and build the backbone of innovation, then buckle up because we're on the hunt for a Data Engineer extraordinaire. Join us on our quest to decode complexity, where your skills will be the magic wand that turns raw data into gold!
The main purpose of this role is to prepare and manage large volumes of data for analytical or operational uses. This role involves designing, building, and maintaining the architecture used for data processing and storage. This includes tasks such as:
Data Modeling and Design: Developing data models and structures that are efficient, scalable, and optimized for various data storage and retrieval needs Data Pipeline Construction: Building and maintaining robust data pipelines that ensure the smooth flow of data from various sources to storage systems and data processing applications. This often involves working with ETL (Extract, Transform, Load) processes Database Management: Managing and optimizing databases to ensure they are fast, reliable, and secure. This includes working with SQL and NoSQL databases Big Data Technologies: Utilizing big data technologies like Hadoop, Spark, and Kafka to handle large-scale data processing Automation and Optimization: Automating data processes and optimizing data flow to improve efficiency and reduce the likelihood of errors Ensuring Data Quality and Governance: Implementing measures to ensure data accuracy, consistency, and security, and complying with data privacy laws and regulations For this role you will need to have a strong background in computer science, engineering, or a related field, and should be proficient in programming languages such as Python, Java, and SQL. As the Data Engineer your work is crucial in enabling the organization to leverage data effectively for decision-making, business intelligence, and data-driven innovation.
Duties and Responsibilities
Duties and Responsibilities (Include but are not limited to):
Provide DataOps support to Analytics Engineers, including overseeing AWS services and CI/CD processes for deploying DDL to a Snowflake Data Warehouse Manage data extraction from various source systems hosted on AWS RDS and EC2 instances Maintain and optimize the extract, stage, delta load staging process, ensuring efficient data flow and integrity Transition to and maintain an Operational Data Store (ODS) using Change Data Capture (CDC) for data ingestion across multiple source databases, including MySQL, Postgres, SQL Server, and NoSQL data stores Collaborate with cross-functional teams to understand and meet data requirements, ensuring the data infrastructure supports these needs. Ensure data security and compliance with financial industry regulations and best practices Troubleshoot and resolve issues related to data systems and processes Continuously evaluate and implement new technologies and tools to enhance data systems and processes
Qualifications and Experience
Key Requirements:
Minimum of 3 years of experience in Data Engineering, preferably in a financial services environment Strong experience with cloud data warehousing, including schema change management and DDL deployments. (Snowflake CDW preferable) AWS Solutions Architect Associate certification preferable Proven experience in managing AWS services, including RDS and EC2, in a professional setting Solid understanding and experience with CI/CD processes and tools, specifically related to data operations Proficiency in SQL and familiarity with programming languages such as Python or Java Experience with data modeling and warehousing, particularly in a Snowflake environment Familiarity with Snowpark, Snowflake's native integration with Spark, or similar technologies Experience with CDC-based data ingestion tools, particularly Debezium Knowledge of data security and privacy standards specific to the financial industry Additional Skills:
Excellent problem-solving skills and attention to detail Strong communication and collaboration skills
