Open to opportunities

Kyle Aidan

@kyleaidan

Message

Senior data engineer building cloud-native, reliable data platforms that drive fast business insights.

United States

Message

What I'm looking for

I’m seeking senior data engineering roles focused on cloud-native platform design, scalable pipelines, real-time streaming, strong DevSecOps practices, and opportunities to lead migrations and improve cost/performance.

I am a Senior Data Engineer with 7+ years designing and building modern data ecosystems across cloud and hybrid environments.

I’ve architected large migrations—moving 200+ TB from on‑prem Oracle and Hadoop into Azure Synapse and OneLake—improving query response times by 60% and cutting infrastructure costs by 27%.

I design medallion-layered data lakes, reusable Bronze/Silver/Gold pipelines with Databricks and Azure Data Factory, and build streaming systems with Kafka and Spark to deliver real-time insights under stringent latency requirements.

I emphasize reliability and compliance: implementing DevSecOps guardrails, self‑healing pipeline frameworks that reduced downtime by 85%, automated data quality with Great Expectations, and optimized ETL to reduce processing from 6 hours to 2 hours.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

RemX

Jan 2023 - Present (2 years 8 months)

Architected and led migration of 200+ TB from on-prem Oracle/Hadoop to Azure Synapse and OneLake, improving query response times by 60% and cutting infrastructure costs by 27%. Designed medallion-layered data lake and built cross-cloud streaming pipelines delivering fraud-detection insights under 4s while reducing downtime by 85% with self-healing pipelines.

Azure Synapse OneLake Databricks Azure Data Factory Apache Kafka Spark Structured Streaming DevSecOps Delta Lake Terraform

Data Engineer II

TrailiT

May 2020 - Jul 2023 (3 years 2 months)

Developed streaming and batch ingestion pipelines processing 1.5B daily events using Kafka, Spark and AWS Glue; partnered on dbt transformations and standardized Snowflake data marts to enable consistent risk-reporting. Optimized Scala ETL reducing nightly jobs from 6h to 2h and automated data quality checks catching 98% of upstream schema changes.

Kafka Apache Spark AWS Glue Snowflake DBT Scala Great Expectations ETL AWS S3

Data Engineer

Movement Labs

Aug 2018 - Mar 2020 (1 year 7 months)

Built and maintained ELT jobs for financial datasets with Airflow and Redshift supporting regulatory reporting; designed dimensional models and ERDs improving BI query performance 3x and created reusable Python ingestion modules for REST, SFTP and streaming sources.