Web Scraping Engineer (Python) (Remote - UK)

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Web Scraping Engineer (Python) in the United Kingdom.

This role offers the opportunity to design and operate high-scale web and AI/LLM scraping systems that deliver critical data to drive business insights. You will build resilient pipelines capable of handling anti-bot defenses and upstream changes while ensuring reliable, cost-efficient cloud deployments. Collaborating closely with Data Engineering, DevOps, and Product teams, you will influence architecture, performance, and system observability. The environment is fast-paced, innovative, and highly collaborative, allowing for autonomy and ownership. Your work will have immediate, tangible impact on global products and customers, while providing opportunities for technical growth and innovation.

Accountabilities

• Design, build, deploy, and maintain scalable web and AI/LLM scraping systems with high reliability and performance.
• Develop and implement anti-bot and evasion strategies, including proxies, headless browsers, CAPTCHAs, and request fingerprinting.
• Integrate scraped data into pipelines, including cleaning, deduplication, schema validation, and delivery to data warehouses.
• Ensure system resilience through monitoring, alerting, retries, and fallback strategies.
• Collaborate with cloud infrastructure and DevOps teams for containerized deployments and cost optimization.
• Proactively adapt systems to platform changes, anti-bot measures, and evolving AI data sources.
• Maintain strong documentation, best practices, and code hygiene, and communicate effectively across distributed teams.

Requirements

• Bachelor’s degree in Computer Science or a related technical field, or equivalent practical experience.
• Minimum 5 years of experience building and running production web scraping systems at scale.
• Strong Python skills, with experience in frameworks such as Scrapy, Playwright, or Selenium.
• Practical anti-bot experience: proxies, CAPTCHAs, JavaScript rendering, and fingerprinting.
• Experience with distributed systems and pipelines (Spark, Airflow) integrated with Redshift, PostgreSQL, or similar.
• Experience scraping LLM/AI sources (Perplexity, ChatGPT, Claude, Gemini).
• Experience deploying and managing workloads in cloud environments (AWS/GCP) using Docker and Kubernetes.
• Proven ability to build observable, resilient systems using Grafana, Prometheus, ELK, or similar monitoring stacks.
• Strong debugging, performance optimization, and reliability focus.
• Independent self-starter with excellent communication, able to take end-to-end ownership of projects.

Bonus / Desirable Skills:

• Awareness of legal, ethical, and compliance considerations in scraping (e.g., GDPR, terms of service).
• Familiarity with messaging/queuing systems (Kafka, RabbitMQ).
• Knowledge of Infrastructure-as-Code (Terraform, Ansible) and CI/CD pipelines (Jenkins, GitHub Actions, TeamCity).
• Experience with cloud cost optimization and scaling strategies for large-scale scraping.

Benefits

• Competitive salary and stock options to share in company growth.
• Remote-first work policy, with hybrid options at UK geohubs.
• Flexible working hours and full autonomy over workday structure.
• Private medical insurance and 24/7 mental health support via Employee Assistance Programs.
• Generous parental leave: up to 6 months fully paid maternity leave, 2 months paternity leave.
• Pension scheme from day one.
• Home office stipend to set up your ideal workspace.
• Swag welcome gift and birthday day off.
• Regular team socials, hackdays, and training days.
• Paid volunteer days to support social impact initiatives.

Jobgether is a Talent Matching Platform that partners with companies worldwide to efficiently connect top talent with the right opportunities through AI-driven job matching.

When you apply, your profile goes through our AI-powered screening process designed to identify top talent efficiently and fairly.
🔍 Our AI evaluates your CV and LinkedIn profile thoroughly, analyzing your skills, experience, and achievements.
📊 It compares your profile to the job’s core requirements and past success factors to determine your match score.
🎯 Based on this analysis, we automatically shortlist the 3 candidates with the highest match to the role.
🧠 When necessary, our human team may perform an additional manual review to ensure no strong profile is missed.

The process is transparent, skills-based, and free of bias — focusing solely on your fit for the role. Once the shortlist is completed, we share it directly with the company that owns the job opening. The final decision and next steps (such as interviews or additional assessments) are then made by their internal hiring team.

Thank you for your interest!