Senior Data Engineer (Kafka Streaming, Spark, Iceberg on Kubernetes)

Senior Data Engineer (Kafka Streaming, Spark, Iceberg on Kubernetes)
Build and scale a next-generation real-time data platform with cutting-edge open-source technologies.
100% Remote | R100 000 – R110 000 per month

About Our Client

Our client is a rapidly growing technology-driven organization building high-performance data platforms to enable advanced analytics, AI, and business intelligence. The team operates at the forefront of real-time data processing and distributed systems, leveraging modern cloud-native infrastructure. They foster a culture of technical excellence, continuous learning, and collaboration across multidisciplinary engineering teams.

The Role: Senior Data Engineer

As a Senior Data Engineer, you will design, build, and optimize next-generation data pipelines and platforms. Youll lead the architecture and implementation of scalable, real-time data solutions using Kafka, Spark, and Apache Iceberg deployed on Kubernetes. This is a hands-on, high-impact role within a forward-thinking data engineering team focused on performance, scalability, and innovation.

Key Responsibilities

• 5+ years of professional experience in data engineering or software engineering

• Design and implement scalable, highly available real-time data pipelines and architectures

• Build robust ETL and streaming pipelines using Apache Spark (Scala/Python) and Kafka Connect/Streams

• Develop and manage data lakes using Apache Iceberg with schema evolution and time travel capabilities

• Deploy and manage distributed data processing services on Kubernetes using containerization best practices

• Optimize performance and resource usage across Spark jobs, streaming apps, and Iceberg tables

• Define and uphold engineering best practices including testing, code standards, and CI/CD workflows

• Mentor junior engineers and contribute to building a high-performing data engineering team

About You

• 5+ years of experience in data engineering or related software engineering roles

• Advanced proficiency with Apache Spark (batch and streaming)

• In-depth experience with Apache Kafka (Connect, Streams, or ksqlDB)

• Hands-on experience with Apache Iceberg, including table evolution and performance tuning

• Skilled in Python (PySpark) or Scala

• Experience deploying and managing distributed systems on Kubernetes (Spark Operator is a plus)

• Solid understanding of data modeling and data warehousing concepts

• Advantageous: Experience with AWS, Azure, or GCP; familiarity with Flink or Trino

• Preferred: Bachelors or Masters degree in Computer Science, Engineering, or related field