Johannesburg / Cape Town
Permanent
We are looking for a data engineer developer who is committed to harnessing the power of data and cloud technologies to drive innovation and informed decision-making. We are seeking an experienced AWS engineer with expertise in AWS Elastic and Python to join our collaborative team. As part of our team, you will play a vital role in building a data lake on the AWS platform, facilitating the seamless ingestion of real-time and local market data feeds into Elasticsearch, and enabling the creation of dynamic dashboards using Kibana.
The Role
As a data engineer with experience in AWS and a focus on data lake construction and Elasticsearch integration, you will be at the forefront of our data-driven initiatives. Your responsibilities will encompass working with a range of AWS services, including Lambda, S3, CodePipeline, and CodeCommit, as well as Elasticsearch, Kibana, Logstash, Beats, Elastic Common Schema (ECS), and Elastic Security. Your role will involve deploying, configuring, testing, and troubleshooting AWS services, ensuring efficient data indexing, querying, aggregations, and mappings. You will also serve as the subject matter expert for ELK (Elasticsearch, Logstash, Kibana) implementation across our shared service platform.
Responsibilities
- Collaborate with cross-functional teams to construct and manage a data lake on AWS for real-time and local market data feeds
- Implement and maintain Elasticsearch, Logstash, Beats, Kibana, Elastic Common Schema (ECS) and Elastic Security components
- Utilise AWS services such as Lambdas, S3, CodePipeline, and CodeCommit to automate data lake processes
- Develop and maintain data ingestion pipelines using Apache Flink and Apache Beam
- Deploy, configure, and optimise AWS services to support data storage, processing, and analysis
- Index data, design complex queries, create aggregations, and manage mappings within Elasticsearch
- Serve as the go-to expert for ELK implementation and configuration
- Collaborate with teams to integrate Elasticsearch with other operational data platforms and tools, including Kafka, SIEM, and more
- Bachelor's degree in Computer Science, Information Technology, or a related field
- Extensive experience as a developer with a strong focus on AWS Elastic services and Python
- In-depth knowledge of Elasticsearch, Logstash, Kibana, Beats, Elastic Common Schema (ECS) and Elastic Security
- Familiarity with AWS services, including Lambdas, S3, CodePipeline, and CodeCommit
- Proven expertise in data pipeline development using Apache Flink and Apache Beam
- Proficiency in Python for scripting and automation
- Background in cyber security or data engineering highly advantageous
- Strong problem-solving and troubleshooting skills
- Excellent communication and collaboration abilities
- Ability to excel in a fast-paced, team-oriented environment
