Senior Databricks Engineer

Koniag Emerging Technologies, LLC is seeking a Senior Databricks Engineer with a Secret security clearance to design, develop, and deploy scalable data infrastructure using the Databricks platform. The ideal candidate will have 8+ years of dedicated Data Engineering experience and expert-level proficiency in PySpark and complex SQL.

Requirements

• Design, develop, and deploy scalable data infrastructure using the Databricks platform
• Drive key data engineering initiatives
• Collaborate with project managers, team members, and business owners
• Architect and implement scalable, highly available data pipelines
• Establish comprehensive data quality standards and implement monitoring dashboards
• Write clean, efficient, and production-ready code using PySpark and SQL
• Design and build robust data ingestion pipelines to reliably pull from various third-party external systems
• Implement automated deployments using Azure DevOps
• Develop and maintain custom Python libraries and modules
• Apply foundational and advanced data warehousing concepts
• Monitor, troubleshoot, and optimize Databricks clusters and Spark jobs
• Securely share data with external BI tools and develop dashboards natively within the Databricks environment
• Serve as a technical SME for the data engineering team

Benefits

• Health, dental and vision insurance
• 401K with company matching
• Flexible spending accounts
• Paid holidays
• Three weeks paid time off