Splunk Engineer

Responsibilities

• Architect, engineer, implement, and administer Splunk solutions in highly available, redundant, distributed computing environments.
• Lead design and deployment of new Splunk environments, including clustered, multi-site, and large-scale configurations.
• Perform Splunk forwarder deployment, configuration, and troubleshooting across diverse platforms.
• Integrate, curate, and normalize diverse log sources into Splunk, ensuring CIM compliance and high data fidelity.
• Configure and maintain Splunk dashboards, searches, and alerts to meet PCI DSS logging requirements, and deliver evidentiary reports to auditors to support compliance verification
• Develop advanced content for SIEM correlation, including custom correlation searches, dashboards, and alerts.
• Administer, maintain, and tune Splunk components (Indexers, Search Heads, Forwarders, Cluster Masters, Deployer, Deployment Server, and License Master).
• Proactively monitor platform health using internal logs, KPIs, and custom monitoring solutions to identify and address performance bottlenecks.
• Lead capacity planning, storage forecasting, and continuity of operations for large Splunk deployments.
• Optimize Splunk performance through configuration tuning, search optimization, and data model acceleration strategies.
• Troubleshoot complex ingestion, performance, and search-related issues, identifying root causes and implementing sustainable fixes or workarounds.
• Reproduce customer or internal issues, document findings, and work with Splunk Support or vendor engineers for resolution.
• Create, maintain, and enforce Splunk engineering documentation, including SOPs, design diagrams, architecture runbooks, and KB articles.
• Develop custom scripts and automation tools (e.g., Python, Bash, PowerShell) to improve Splunk administration, onboarding, and operational workflows.
• Utilize Splunk APIs for integration with enterprise tools and automation frameworks.
• Serve as a technical escalation point for Splunk Engineer I/II and Splunk Admin roles.
• Administer, tune, and troubleshoot Splunk Enterprise Security, maintaining data models, correlation searches, and notable events pipeline.
• Configure and manage HEC (HTTP Event Collector) connections and onboard new data sources.
• Manage Splunk RBAC (Role-Based Access Control) including SAML and AD group integrations for search heads and API endpoints.
• Collaborate with security, infrastructure, application, and DevOps teams to ensure Splunk aligns with enterprise monitoring, compliance, and operational goals.
• Design and implement Splunk solutions supporting compliance frameworks (e.g., PCI DSS, HIPAA, SOX), including dashboard/report development and audit evidence.
• Research, evaluate, and implement new Splunk apps, add-ons, and integrations to enhance platform capabilities.
• Mentor junior Splunk engineers and guide cross-functional teams on Splunk best practices, search optimization, and data onboarding.

Requirements

• 8+ years of IT experience in technical engineering, security operations, or infrastructure roles.
• 5+ years of direct, hands-on Splunk engineering and administration experience in large-scale, distributed environments.
• Expert-level knowledge of Splunk Enterprise and Splunk Enterprise Security, including architecture, clustering, and scaling strategies.
• Proficiency in Linux/Unix administration and shell scripting.
• Strong knowledge of Splunk APIs, including use for automation and tool integrations.
• Expertise in regex, field extractions, and key-value parsing.
• Strong programming/scripting skills in one or more languages (Python, Bash, PowerShell, Perl, JavaScript).
• Experience with storage systems (DAS, SAN, object storage) and understanding of their performance implications for Splunk indexing.
• Solid understanding of networking (switches, routers, firewalls, load balancers, DNS, SSL/TLS) and how it impacts Splunk architecture.
• Familiarity with Enterprise Management and automation tools.
• Experience with Splunk ITSI (preferred) and other premium Splunk apps.
• Strong knowledge of data formats including JSON, XML, and CSV.
• Demonstrated experience delivering Splunk-based compliance reporting and audit support.
• Strong communication skills for interacting with technical and non-technical stakeholders.
• Proven ability to lead projects, mentor team members, and provide architectural guidance.

Education & Certifications

• Bachelor’s degree in Computer Science, Information Systems, or related technical field (or equivalent experience).
• Splunk Certified Architect and/or Splunk Certified Consultant preferred.
• Additional certifications in security, cloud, or automation tools are a plus.

About Rackspace Technology

More on Rackspace Technology