Senior Data Engineer

Job Title: Data Engineer

Location: Remote

Contract: £500 Per Day

Role Overview

Our client is seeking a Data Engineer with a solid foundation in both data engineering and business intelligence, capable of understanding and addressing business needs. This role involves managing and optimizing AWS-based data processes, orchestrating data flows, and supporting business intelligence capabilities. The ideal candidate will not only bring technical expertise but also a strong business acumen to facilitate effective collaboration with non-technical stakeholders.

Key Responsibilities

• Data Pipeline Management:
• Database Integration: Maintain MySQL databases on AWS (approx. 25TB) as primary data sources feeding the data pipeline.
• Real-Time Data Ingestion: Utilize Apache Kafka to manage data flows from MySQL into AWS S3 buckets, enabling real-time streaming.
• Data Processing: Implement ETL processes using AWS Glue and PySpark to build data lakes within S3 for further analysis.
• Business Intelligence (BI):
• Facilitate data availability for BI purposes, supporting daily updates and aiming to move from overnight batch jobs to near-real-time 15-minute refreshes.
• Collaborate on BI dashboard and reporting creation to provide actionable insights.
• Complex Data Reconciliation:
• Reconcile transaction records across multiple merchants, currencies, and network operators, addressing both straightforward and complex data relationships.

Technical Requirements

• Data Engineering Expertise:
• Proficiency with MySQL, Apache Kafka, AWS Glue, PySpark, and AWS S3 for managing data pipelines and ETL processes.
• Business Intelligence Skills:
• Experienced in designing and maintaining BI reports and dashboards with frequent data refresh capabilities.
• Data Reconciliation Experience:
• Skilled in reconciling transactional data across diverse systems, operators, and currencies to ensure data accuracy.

Business & Communication Skills

• Business Acumen:
• Ability to translate business requirements into data solutions and work collaboratively with non-technical stakeholders to develop reports and dashboards tailored to business needs.
• Communication:
• Strong communication skills to bridge technical and business needs, working closely with stakeholders to ensure data solutions support strategic objectives.