Principal Data Scientist - Agent Builder
hace 8 días
Madrid
ppElastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. /ph3What Is The Role /h3pAs a Principal Data Scientist on the Search Conversational Experiences team, you will help set the technical direction for evaluating, improving, and scaling chat quality across Elastic’s agentic platform. You will define the evaluation strategy that guides product decisions, including models, routing, tools, and tailoring agents to use cases. You will work closely with backend engineering, product, UX, and other data scientists to turn ambiguous, cutting‑edge problems into measurable product improvements. /ph3What You Will Be Doing /h3ulliDefine the evaluation strategy for conversational and agentic search, including offline and online evaluation, golden datasets, rubrics, LLM‑as‑judge calibration, groundedness and citation checks, and A/B testing. /liliLead the design of quality metrics and decision frameworks for RAG, agents, tools, model selection, agent routing, prompt behavior, and cost/latency trade‑offs. /liliBuild, compare, and guide improvements across retrieval and re‑ranking approaches, including sparse and dense retrieval, vector search, query understanding, semantic rewrites, and context enrichment. /liliTurn experimental results into product and business decisions: which models to use, how to route requests efficiently, which tools should be exposed, and how agents should be customized for different Elastic use cases. /liliPartner with engineering to productionize evaluation pipelines, telemetry, dashboards, CI guardrails, and regression detection for chat quality, helpfulness, dedication, latency, and cost. /liliInfluence the roadmap by identifying the highest‑leverage quality gaps, proposing practical solutions, and communicating trade‑offs clearly to product, engineering, and leadership. /liliMentor other data scientists and engineers in experiment design, evaluation methodology, statistical rigor, and practical approaches to improving LLM‑powered systems. /liliShare outcomes through clear docs, notebooks, PRs, dashboards, technical proposals, and cross‑functional reviews. /li /ulh3What You Bring /h3ulli8+ years of applied DS/ML experience, with deep expertise in IR, NLP, ranking, semantic search, RAG, or LLM‑powered product experiences. /liliStrong track record defining and leading evaluation for production AI/ML systems, including offline metrics, online experimentation, LLM‑as‑judge approaches, groundedness, citation quality, and model comparison. /liliExperience influencing product and technical strategy through data, especially in ambiguous or emerging domains where the “right” metric or approach is not obvious. /liliHands‑on ability with Python, PyTorch/Transformers, Pandas, notebooks, reproducible experiments, versioned datasets, and clean, reviewable code. /liliStrong understanding of retrieval systems, including dense and sparse retrieval, re‑ranking, vector search, query understanding, and evaluation metrics such as nDCG, MRR, , precision, and latency/cost trade‑offs. /liliExperience collaborating closely with engineering teams to move from prototype to production, including telemetry design, dashboards, CI guardrails, and quality regression tracking. /liliPractical Elasticsearch experience, or experience with similar search and distributed data systems. ES|QL familiarity is a plus. /liliExcellent written and verbal communication, with the ability to explain complex scientific and technical trade‑offs to engineering, product, design, and leadership audiences. /liliA collaborative, low‑ego style and a strong ability to mentor, raise standards, and develop transparency for others in a distributed team. /li /ulh3Compensation /h3pBase salary only. Typical starting salary range: €67,000–€106,000EUR. /ph3Equal Opportunity Employment /h3pElastic is an equal‑opportunity employer and is committed to creating an inclusive culture that celebrates different perspectives, experiences, and backgrounds. Qualified applicants will receive consideration for employment without regard to race, ethnicity, color, religion, sex, pregnancy, sexual orientation, gender identity or expression, national origin, age, marital status, protected veteran status, disability status, or any other basis protected by federal, state or local law, ordinance or regulation. /ppWe welcome individuals with disabilities and strive to create an accessible and inclusive experience for all individuals. To request an accommodation during the application or recruiting process, please email /p /p #J-18808-Ljbffr