We are Intellectsoft - a digital transformation consultancy group and engineering company that delivers cutting-edge solutions for global organisations and technology startups. Since 2007, we have been helping companies and established brands reimagine their business through digitalization. We're looking for an exceptional Senior Data Engineer to join our team. Are you up for a challenge?
Project description:
A biotechnology research project that earned recognition as a trusted provider of clinical genetic testing and an ideal collaborator for developing precision medicine solutions. Patients are central to the project's mission, driving partnerships with numerous non-profit organizations to support patients and healthcare professionals in their quest for answers. The main focus - is Clinical Cancer Diagnostic; however, the project is designed to expand into additional areas such as Molecular Diagnostic Testing, Cardio Genetics, Neuro Genetics, etc
Responsibilities:
- Develop connectors for Kafka to streamline the syncing of updates from source data repositories.
- Establish partitioned Kafka topics to optimize the synchronization of updates to destination data marts.
- Leverage Apache Flink for crafting intricate data analytics workloads to enable real-time monitoring and transformations.
- Deploy dashboards using Datadog and Cloudwatch to uphold system health and fulfil user needs.
- Institute schema registries to uphold data governance standards while catering to diverse data requirements.
- Collaborate closely with a West Coast-based scrum team, contributing to daily pull request submissions, code reviews, documentation maintenance, backlog management, and build validation across environments within sprint cycles lasting 2-4 weeks.
- Coordinate with other scrum teams to ensure coherence on data contracts, API specifications, and deployment timelines.
- Architect database schemas with a focus on query access patterns.
- Establish and manage CI/CD pipelines using infrastructure-as-code principles.
- Transition on-premises ETL jobs from PHP to AWS Flink and Glue processes gradually.
- Work alongside QA Engineers to develop automated test suites.
- Engage with end-users to troubleshoot service interruptions and champion our data product offerings.
- Maintain vigilant oversight of data quality, promptly addressing discrepancies, latency issues, and defects.
Requirements
Must have: Nice to have:
Benefits