Open to opportunities

Joshua Sommers

@joshuasommers1

Message

Senior data engineer designing distributed, near-real-time pipelines and analytics under tight SLAs.

United States

Message

What I'm looking for

I’m looking for a role building distributed data platforms—Kafka/Spark-driven pipelines with strong observability, DDL governance, and reliable SLAs—where I can optimize performance, reduce cost, migrate to cloud, and mentor engineers.

I’m a senior data engineer with more than 10 years designing and operating distributed data platforms across startup and enterprise environments. I deliver near-real-time ingestion, ETL, aggregation, and serving layers that stay reliable under tight SLAs.

In my most recent role, I architected Kafka-based ingestion pipelines and schema-registry flows to ingest 10–20 TB/day of event data, improving downstream availability. I optimized Spark and Spark Streaming workloads to cut aggregate job runtime by 60% and reduced cluster CPU usage by 35% across 1200 daily jobs.

I’m also hands-on across data quality and governance: I monitor regressions using Graphite and custom metrics, validate DDL changes, and enforce standards to reduce schema drift. I’ve tuned Presto/Trino for interactive reporting (cutting query latency by 40%), and reduced storage/retention costs by implementing partitioning and Apache Iceberg table formats.

I like building systems that teams can run confidently—automating orchestration with Airflow, containerizing workloads with Kubernetes and Docker, and strengthening observability with Prometheus/Grafana and other monitoring tools. I’ve mentored engineers and partnered across DBAs and stakeholders to scale on-prem Hadoop, manage cloud migrations to GCP/AWS, and improve operational outcomes like MTTR and alert fidelity.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

Fundrise

Nov 2022 - Present (3 years 8 months)

Architected Kafka-based ingestion pipelines and schema registry flows to ingest 10–20 TB/day of event data and improve downstream availability by 42%. Optimized Spark/Spark Streaming jobs, automated Airflow scheduling/dependencies, and improved data quality monitoring to raise reporting SLA accuracy to 99.7%.

Kafka Apache Spark Airflow Graphite Kubernetes Docker Presto Trino Hive Apache Iceberg Data Quality SQL Optimization

Senior Software Engineer

Scratchpad

Aug 2021 - Nov 2022 (1 year 3 months)

Designed Python ELT pipelines to centralize product and usage data into BigQuery, reducing analyst time to insight by 55%. Tuned SQL/Presto queries and Kafka-to-Spark Streaming ingestion to improve dashboard refresh speed and near-time aggregation latency.

Python BigQuery SQL Presto Kafka Docker Kubernetes Airflow Prometheus Grafana Looker

Full Stack Engineer (Data & ETL)

Solugenix

Jul 2020 - Mar 2021 (8 months)

Implemented batch ETL pipelines using Hive and Hadoop to ingest client transactional datasets (2–5 TB/month) into a centralized warehouse. Built Spark-based streaming transformations for reporting and created repeatable Docker-based ETL test environments.

Hive Hadoop Apache Spark Docker Python SQL Server PostgreSQL Luigi

Software Engineer

USAA

Oct 2019 - Jul 2020 (9 months)

Implemented Python data ingestion and validation pipelines to unify financial transaction feeds and improve downstream data consistency by 36%. Integrated Kafka with micro-batch Spark jobs for near-real-time metrics and built observability/alerting to reduce time-to-detect by 50%.

Python Kafka Apache Spark SQL Server Data Lake Graphite Observability

Senior Software Engineer

Cognizant

Oct 2018 - Oct 2019 (1 year)

Led migration of legacy ETL jobs to Spark and Hive, reducing nightly batch windows by 45% across a 10 TB dataset. Provisioned on-prem Hadoop resources, improved monitoring dashboards, and reviewed Java/Scala streaming code to increase throughput by 28% on critical paths.

Apache Spark Hive Kafka Hadoop Java Scala Data Modeling Capacity Planning Partitioning Bucketing

Software Developer

General Motors

Jun 2017 - Oct 2018 (1 year 4 months)

Implemented ETL data pipelines in Java and Python to collect vehicle telemetry into a central data store for diagnostics and analytics. Deployed and maintained on-prem Hadoop jobs and Hive tables, improving batch processing reliability by 31% and adding data validators to reduce late-arriving data by 65%.

Python Hadoop Hive SQL Server Data Reconciliation Backup Governance Java