Data Analyst: Ranking Team

About us

Constructor is the next-generation platform for search and discovery in e-commerce, built to explicitly optimize for metrics like revenue, conversion rate, and profit. Our search engine is entirely invented in-house utilizing transformers and generative LLMs, and we use its core and personalization capabilities to power everything from search itself to recommendations to shopping agents. Engineering is by far our largest department, and we’ve built our proprietary engine to be the best on the market, having never lost an A/B test to a competitive technology. We’re passionate about maintaining this and work on the bleeding edge of AI to do so.

Out of necessity, our engine is built for extreme scale and powers over 1 billion queries every day across 150 languages and roughly 100 countries. It is used by some of the biggest e-commerce companies in the world like Sephora, Under Armour, and Petco.

We’re a passionate team who love solving problems and want to make our customers’ and coworkers’ lives better. We value empathy, openness, curiosity, continuous improvement, and are excited by metrics that matter. We believe that empowering everyone in a company to do what they do best can lead to great things.

Constructor is a U.S. based company that has been in the market since 2019. It was founded by Eli Finkelshteyn and Dan McCormick who still lead the company today.

Role Overview

The Ranking Team builds and optimizes machine learning models that determine what products shoppers discover through search and how effectively they find the items they're looking for in our Customers’ e-commerce platforms. We're looking for a Data Analyst to evaluate and improve the performance of ranking models, features, and configurations across our large-scale experimentation program—running 10s to 100s of A/B tests monthly and processing terabytes of data. Your work will help shape key metrics, interpret user behavior, and drive product and algorithm improvements.

As the first data analyst on our 6-person ranking team, you'll collaborate closely with engineers (who develop the platform and improve ML algorithms), data scientists, and product managers to define success, uncover insights, and influence roadmap decisions through data.

Challenges you will tackle

• Analyze Ranking Performance: Evaluate A/B tests across models, configurations, and customer segments to quantify business impact and guide ML development. Investigate training data quality, model performance trends, and feature effectiveness at scale.
• Understand Shopper Behavior: Investigate how ranking changes affect user behavior and conversion metrics. Use SQL, Python, and Spark to uncover usage patterns, anomalies, and opportunities for optimization.
• Design Validate Metrics: Define new metrics to measure search relevance, personalization, and model performance. Ensure metrics align with user experience and business goals through rigorous validation.
• Build Analytics Infrastructure: Create scalable dashboards and reporting tools for product, engineering, and leadership teams. Develop debugging tools to explain ranking decisions and identify performance issues.
• Drive Data-Informed Decisions: Partner cross-functionally to design experiments, validate hypotheses, and communicate insights that directly influence product roadmap and ML strategy.

Requirements

• Advanced Analytics A/B Testing: 3+ years analyzing complex experiments and extracting actionable insights from large, noisy datasets. Experience with statistical testing and practical experiment design.
• SQL Big Data Expertise: Write optimized SQL queries for terabyte-scale data extraction and transformation. Proficiency with distributed systems like Spark for large-scale data processing.
• Python Programming: Strong skills in exploratory analysis, custom metrics development, and building internal tools. Experience with data science libraries and automation.
• Machine Learning Search Systems: Understanding of ML pipelines, training data quality, and ranking/recommendation metrics. Familiarity with search relevance and personalization concepts.
• Metrics Design Validation: Design metrics that accurately reflect model and product performance. Ensure alignment between technical metrics and business outcomes.
• Data Visualization: Create compelling dashboards using Tableau, Looker, or custom dashboards in Python. Present complex findings clearly to both technical and executive audiences.
• Cross-Functional Leadership: Influence product and engineering decisions through data storytelling. Collaborate effectively across teams to drive ML and product improvements.
• Product-Minded Business-Focused: Deep curiosity about user behavior and business impact. Connect algorithm changes to real-world customer outcomes.

Benefits

• 🏝️ Unlimited vacation time - we strongly encourage all of our employees take at least 3 weeks per year

• 🌎 Fully remote team - choose where you live
• 🛋️ Work from home stipend! We want you to have the resources you need to set up your home office
• 💻 Apple laptops provided for new employees
• 🧑‍🎓 Training and development budget for every employee, refreshed each year
• 👪 Maternity Paternity leave for qualified employees
• 🧠 Work with smart people who will help you grow and make a meaningful impact
• 💵 This position has a base salary range between $80k and $120k USD. The offer varies on many factors including job related knowledge, skills, experience, and interview results.
• 🎉 Regular team offsites to connect and collaborate

Diversity, Equity, and Inclusion at Constructor

At Constructor.io we are committed to cultivating a work environment that is diverse, equitable, and inclusive. As an equal opportunity employer, we welcome individuals of all backgrounds and provide equal opportunities to all applicants regardless of their education, diversity of opinion, race, color, religion, gender, gender expression, sexual orientation, national origin, genetics, disability, age, veteran status or affiliation in any other protected group.

Studies have shown that women and people of color may be less likely to apply for jobs unless they meet every one of the qualifications listed. Our primary interest is in finding the best candidate for the job. We encourage you to apply even if you don’t meet all of our listed qualifications.