‹ All jobs
AI Benchmark Engineer | Native Language Specialist - French (France) - Remote
<h2><strong>About The Opportunity</strong></h2><p style="min-height:1.5em">We are building a rigorous, verifiable evaluation suite of Terminal-Bench tasks designed to test the limits of large language models on multilingual software challenges. Our goal is to measure multilingual robustness across prompt language effects, non-English data processing, and complex locale/encoding edge cases in terminal workflows.</p><p style="min-height:1.5em">We are seeking experienced native-speaking software engineers to design, build, and validate these benchmarks. You will create high-signal, high-quality tasks that genuinely test a model's ability to handle multilingual environments without relying on English translation crutches.</p><p style="min-height:1.5em"><strong>Note this is a remote, freelance opportunity</strong></p><p style="min-height:1.5em"></p><h2><strong>What You’ll Deliver </strong></h2><ul style="min-height:1.5em"><li><p style="min-height:1.5em">Task Engineering: Evaluating Coding Agents.</p></li><li><p style="min-height:1.5em">Asset Creation: Build realistic task environments using datasets and files in your native language. <em>Crucially, these assets must remain in the target language to genuinely measure multilingual handling.</em></p></li><li><p style="min-height:1.5em">Prompting & Translation: finding failure points where AI does not work, in your native language</p></li><li><p style="min-height:1.5em"> Implementation & Verification: Support the development of robust solutions (reference implementations) and write highly reliable, deterministic verifier scripts (using rubric-based judging only when strictly necessary).</p></li><li><p style="min-height:1.5em">Calibration & Execution: Analyze execution logs and calibrate task difficulty (Easy to Very Hard) using standard Terminal-Bench run configurations against various model tiers (Haiku, Sonnet, Opus).</p></li><li><p style="min-height:1.5em">Quality Assurance: Participate in a rigorous, 4-layer human quality control process (creation, human review, calibration review, and audit) alongside automated LLM-based checks to ensure fairness, grammatical accuracy, and benchmark integrity.</p></li></ul><p style="min-height:1.5em"></p><h2><strong>Qualifications </strong></h2><ul style="min-height:1.5em"><li><p style="min-height:1.5em"><strong>Experience:</strong> 1+ years of industry experience in software engineering.</p></li><li><p style="min-height:1.5em"><strong>Background:</strong> Proven track record at leading technology companies and/or graduation from top-tier engineering universities.</p></li><li><p style="min-height:1.5em"><strong>Language:</strong> Native or near-native fluency, with a deep understanding of its grammar, register, and phrasing rules. High English proficiency.</p></li><li><p style="min-height:1.5em"><strong>Technical Stack:</strong> Strong proficiency in Python, standard shell scripting, and data processing.</p></li><li><p style="min-height:1.5em"><strong>Workflow:</strong> Extensive experience with Terminal/CLI-based development workflows and a working familiarity with coding agents.</p></li><li><p style="min-height:1.5em"><strong>Domain Expertise:</strong> Deep technical understanding of multilingual text processing pitfalls, i