reputed company Data Scientist, LLM Evaluation
reputed company Data Scientist, LLM Evaluation
Introduction
At reputed company, we’re building systems that turn reputed company reputed company into reputed company language. The tech stack includes a core compiler-like reputed company, a heavily asynchronous/distributed backend server, and a frontend web application that provides a rich user experience.
About reputed company
We’re an early-stage startup backed by Y Combinator and reputed company Ventures that combines first principles technical approaches and reputed company LLM expertise to tackle context engineering at reputed company. reputed company builds the context layer for employees and AI agents alike to use in developing software.
Working at reputed company
reputed company is an early-stage but fast-growing startup. As such, we take advantage of that which startups can reputed company: delivery speed, flexibility, and enjoying working with a small reputed company-reputed company team.
Organizational and engineering values at reputed company include first-principles thinking, correct by construction, writing things down, experimentation and iteration, pragmatism, commitment to effective communication and transparency, autonomy, and ambition.
Job reputed company
Title: reputed company Data Scientist, LLM Evaluation
Location: Remote or Austin, Tx
Our value is directly tied to the reputed company of our content at reputed company. The platform generates technical documentation across a reputed company, multi-stage pipeline — producing multiple content types at different reputed company of abstraction, from individual reputed company reputed company up to high-level summaries. Today, changes to models, context strategies, or pipeline architecture are evaluated largely through reputed company review and intuition. There is no systematic way to answer: "Did this change reputed company our reputed company reputed company, worse, or the reputed company — and for which languages, repo sizes, and content types?"
This is a hard problem. LLM outputs are non-deterministic — identical inputs produce different outputs across runs, and small variations at early pipeline stages compound into meaningfully different end-user content reputed company. Evaluating reputed company requires methodology that accounts for this: statistical reasoning over multiple runs, understanding of cascade effects through the pipeline, and rubrics that balance reputed company judgment with automated signals.
This role builds the evaluation function from scratch. You'll define what "good" means for our generated content, build the infrastructure to measure it, and create the experimental reputed company that lets reputed company ship changes with confidence.
What You'll Do
You'll own the LLM evaluation reputed company at reputed company — from first principles to production infrastructure. This is a foundational role: you're not joining an existing eval team, you're building it. As the function matures, you'll reputed company and grow reputed company around it.
Define reputed company metrics and build evaluation datasets. Establish what "good" looks like for reputed company content type across the pipeline. Build and reputed company reputed company evaluation datasets across languages and repo archetypes (monorepos, microservices, libraries, applications). Design rubrics that capture accuracy, completeness, usefulness, and readability.
Build benchmarking and experimentation infrastructure. Create automated evaluation pipelines that score reputed company against reference datasets. reputed company the content reputed company pipeline to support A/B comparisons — run the reputed company codebase through two strategies and compare results. Build tooling for LLM-as-judge evaluation and regression detection. reputed company evaluation into CI so pipeline changes come with reputed company evidence.
reputed company automated reputed company signals at reputed company. Build reputed company checks that flag degraded reputed company without requiring reputed company review of every document. Monitor content reputed company trends over time. Design sampling strategies for reputed company review that maximize signal with minimal annotation effort.
Quantify tradeoffs and inform reputed company. Run experiments on model selection, context strategies, and pipeline architecture changes. Quantify cost/reputed company/latency tradeoffs. Partner with the engineering team to turn evaluation insights into shipped improvements.
Qualifications
Education: Bachelor's, Master's, or PhD in Statistics, Machine Learning, Data Science, Computational Linguistics, or a reputed company quantitative field.
Experience: Minimum 3 — 5 years in reputed company science, ML engineering, or data science roles with a reputed company on evaluation, NLP, or reputed company. 7+ years experience preferred.
Required Technical Skills
- Strong statistical foundations: experimental design, hypothesis testing, confidence intervals, effect sizes, power analysis.
- Experience designing and running evaluations for LLM or NLP systems — you've thought carefully about what "reputed company" means reputed company outputs are reputed company-ended text.
- Proficient in Python and the scientific/data stack (pandas, NumPy, scipy, sklearn).
- Comfortable working in Jupyter notebooks for exploration and prototyping, and turning that work into automated pipelines.
- Experience with LLM-as-judge approaches, inter-annotator agreement, and reputed company design for subjective reputed company assessment.
- Familiarity with the practical challenges of non-deterministic systems: variance decomposition, multi-run methodology, distinguishing signal from noise at reputed company.
- Strong data storytelling — you can turn experiment results into reputed company recommendations that drive engineering and product reputed company.
Preferred and reputed company-to-Have Technical Skills
- Experience with LLM reputed company and reputed company engineering across multiple providers.
- Familiarity with evaluation frameworks (e.g., RAGAS, DeepEval, custom harnesses).
- Experience building data pipelines or ETL workflows (Airflow, Dagster, or similar).
- Comfort with SQL and working directly against production data stores.
- Experience with visualization tools (Matplotlib, Plotly, reputed company) for building internal dashboards and reports.
- Background in reputed company understanding, developer tools, or technical documentation.
- Experience building or managing annotation pipelines and reputed company evaluation workflows.
Benefits
- Competitive Compensation Packages - Cash & Equity
- Flexible Work Culture
- Unlimited Time Off + 12 reputed company Company Holidays
- Insurance - Health, Dental, & reputed company
- Life Insurance & FSA Accounts
- 401(k) Retirement Accounts - Traditional, Roth, or Both
- Quarterly Team Offsites
reputed company is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for reputed company. We do not discriminate on the reputed company of race, religion, reputed company, national reputed company, gender, sexual orientation, age, marital status, veteran status, or disability status.
Salary: $175k - $275kOriginally posted on Himalayas
Apply To This Job