Data Engineer

Data Engineer is responsible for designing, building, and maintaining the infrastructure and systems required for collecting, storing, and processing large datasets efficiently.

Education: Bachelor's degree in computer science with 8+ years of experience

Experience:

• Technical Skills
  • Programming Languages: Proficiency in Python, SQL, Java, or Scala for data manipulation and pipeline development.
  • Data Processing Frameworks: Experience with tools like Apache Spark, Hadoop, or Apache Kafka for large-scale data processing.
• Data Systems and Platforms
  • Databases: Knowledge of both relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).
  • Data Warehousing: Experience with platforms like Snowflake, Amazon Redshift and Azure Synapse.
  • Cloud Platforms: Familiarity with AWS, Azure Cloud for deploying and managing data pipelines. Having Good experience in Fabric is advantageous
  • Experience working with distributed computing systems such as Hadoop HDFS, Hive, or Spark.
  • Managing and optimizing data lakes and delta lakes for structured and unstructured data.
• Data Modeling and Architecture
  • Expertise in designing efficient data models (e.g., star schema, snowflake schema) and maintaining data integrity.
  • Understanding of modern data architectures like Data Mesh or Lambda Architecture.
• Data Pipeline Development
  • Building and automating ETL/ELT pipelines for extracting data from diverse sources, transforming it, and loading it into target systems.
  • Monitoring and troubleshooting pipeline performance and failures.
• Workflow Orchestration
  • Hands-on experience with orchestration tools such as Azure Data Factory, AWS Glue jobs, DMS or Prefect to schedule and manage workflows.
• Version Control and CI/CD
  • Utilizing Git for version control and implementing CI/CD practices for data pipeline deployments.

Key Skills:

• Proficiency in programming languages such as Python, SQL, and optionally Scala or Java.
• Proficiency in data processing frameworks like Apache Spark and Hadoop is crucial for handling large-scale and real-time data.
• Expertise in ETL/ELT tools such as Azure ADF and Fabric Data Pipeline is important for creating efficient and scalable data pipelines.
• A solid understanding of database systems, including relational databases like MySQL and PostgreSQL, as well as NoSQL solutions such as MongoDB and Cassandra, is fundamental.
• Experience with cloud platforms, including AWS, Azure and their data-specific services like S3, BigQuery, and Azure Data Factory, is highly valuable.
• Data modeling skills, including designing star or snowflake schema, and knowledge of modern architectures like Lambda and Data Mesh, are critical for building scalable solutions.

Role and Responsibilities:

• Responsible for designing, developing, and maintaining data pipelines and infrastructure to support our data-driven decision-making processes.
• Design, build, and maintain data pipelines to extract, transform, and load data from various sources into our data warehouse and data lake.
• Proficient in creating data bricks creating notebooks, working with catalogs, native SQL, creating clusters, Parameterizing notebooks, and administrating data bricks. Define security models and assign roles as per requirement.
• Responsible for creating data flow in Synapse analytics integrating external source systems, creating external tables, data flows and create data models. Schedule the pipelines using various jobs, creating trigger
• Design and develop data pipelines using Fabric pipelines, spark notebooks accessing multiple data sources. Proficient in developing Data bricks notebooks and data optimization
• Develop and implement data models to ensure data integrity and consistency. Manage and optimize data storage solutions, including databases and data warehouses.
• Develop and implement data quality checks and validation procedures to ensure data accuracy and reliability.
• Design and implement data infrastructure components, including data pipelines, data lakes, and data warehouses.
• Collaborate with data scientists, analysts, and other stakeholders to understand business requirements and translate them into technical solutions.
• Monitoring Azure and Fabric data pipelines, spark jobs and work on fixes based on the request priority.
• Responsible for data monitoring activities, having good knowledge on reporting tools like Power Bi and Tableau is required.
• Responsible for understanding the client requirements and architect solutions in both Azure and AWS cloud platforms.
• Monitor and optimize data pipeline performance and scalability to ensure efficient data processing.