Vacancy description

Senior Software Engineer - Ruby (LLM Evaluation & Repository Validation) in Australia

Turing

  • RemoteWork from anywhere
  • ContractFixed-term or freelance

Australia

Analyze GitHub issues and evaluate LLM performance in software projects.

Job Title: Senior Software Engineer – LLM Evaluation & Repository Validation

About the projects: we are building LLM evaluation and training datasets to train LLM to work on realistic software engineering problems. One of our approaches, in this project, is to build verifiable SWE tasks based on public repository histories in a synthetic approach with human-in-the-loop ; while expanding the dataset coverage to different types of tasks in terms of programming language, difficulty level, and etc.

About the Role: We are looking for experienced software engineers (tech lead level) who are familiar with high-quality public GitHub repositories and can contribute to this project. This role involves hands-on software engineering work, including development environment automation, issue triaging, and evaluating test coverage and quality

Why Join Us? Turing is one of the world’s fastest-growing AI companies accelerating the advancement and deployment of powerful AI systems. You’ll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research.

What does day-to-day look like :

  • Analyze and triage GitHub issues across trending open-source libraries.
  • Set up and configure code repositories, including Dockerization and environment setup.
  • Evaluating unit test coverage and quality.
  • Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
  • Collaborate with researchers to design and identify repositories and issues that are challenging for LLMs.
  • Opportunities to lead a team of junior engineers to collaborate on projects.

Required Skills:

  • Minimum 3+ years of overall experience
  • Strong experience with at least one of the following languages: Ruby
  • Proficiency with Git, Docker, and basic software pipeline setup.
  • Ability to understand and navigate complex codebases.
  • Comfortable running, modifying, and testing real-world projects locally.
  • Experience contributing to or evaluating open-source projects is a plus.

Nice to Have:

  • Previous participation in LLM research or evaluation projects.
  • Experience building or testing developer tools or automation agents.

Perks of Freelancing With Turing:

  • Work in a fully remote environment.
  • Opportunity to work on cutting-edge AI projects with leading LLM companies.

Offer Details:

  • Commitments Required : At least 4 hours per day and minimum 20 hours per week with overlap of 4 hours with PST. (We have 3 options of time commitment: 20 hrs/week, 30 hrs/week or 40 hrs/week)
  • Employment type : Contractor assignment (no medical/paid leave)
  • Location : India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico

Evaluation Process (approximately 75 mins) :

  • Two rounds of interviews (60 min technical + 30 min technical & cultural discussion)

Skills: Ruby

Similar vacancies

More like this role

SuperAnnotate

Ruby Engineer

  • Remote
  • Contract

$55 per hour

... - 1–2+ years of professional Ruby programming experience in production environments. - Strong understanding of Ruby on Rails, MVC architecture, and idiomatic Ruby patterns for structuring application code. - Hands-on software engineering experience building and maintaining Ruby or Rails applications, with exposure to real-world ...

Australia

  • Remote
  • Part-time
  • Contract
  • IT

$80 - $120 per hour

... data is the hardest to produce and most valuable for advanced model performance. Our pipeline performance system ensures both quality and velocity. Job Title: Senior Software Engineer Job Type: Contractor (~15 hrs a week) Location: Remote (Globally) Job Summary: In this role, you'll apply your expertise to help train next-generation ...

Australia

Insignia Financial

Senior Software Engineer

  • Full-time
  • IT
... supports better financial advice experiences - Work across the full software development lifecycle in an Agile environment - Join a senior role focused on modern engineering, automation and continuous improvement The opportunity to join our team Join Insignia Financial as a Senior Software Engineer full-stack Java, within our ...

Perth, Australia

  • Hybrid
  • Full-time
  • IT

A$137.6 AUD

... both to enhance our existing MongoDB cloud storage architecture and to power more of our customers' use cases more efficiently. We are looking for talented Senior Engineers to join the team and be founding members of the team, where you will play a crucial role in our multi-year roadmap. Our team champions a strong culture ...

Syd, Australia

Topcon Positioning

Senior Software Engineer

  • Full-time
  • IT
... applications in embedded and real-time environments - Build robust multi-threaded and event-driven software systems - Debug, troubleshoot and optimise complex software applications - Support software testing and validation activities - Provide technical guidance and mentoring to engineers within the team - Contribute to project ...

Adelaide, Australia

  • Hybrid
  • Full-time
  • IT
  • Legal
About CORTO We are CORTO, a cutting-edge software company dedicated to revolutionising the legal industry. Our mission is to empower legal practitioners with AI-driven solutions that streamline their workflow, boost productivity, and provide more efficient client service. Our team of AI experts and engineers collaborate ...

Syd, Australia

  • Remote
  • Hybrid
  • On-site
  • IT
... Communications System (DHFCS), a critical capability connecting Australian Defence Force command centres with deployed aircraft, ships and land forces. As a Senior Software Engineer, you will lead and deliver software maintenance activities and engineering changes, developing and maintaining software components and associated ...

Keswick, Australia

  • Hybrid
  • Full-time
  • IT
  • Legal
About CORTO We are CORTO, a cutting-edge software company dedicated to revolutionising the legal industry. Our mission is to empower legal practitioners with AI-driven solutions that streamline their workflow, boost productivity, and provide more efficient client service. Our team of AI experts and engineers collaborate ...

Syd, Australia

  • Full-time
  • IT
About the Role Our Senior Software Engineers work closely with other members of the engineering team to scope, design, document, develop, integrate and test software to address end-user needs for a large web-based application. Responsibilities - Designing, implementing and testing software components in Java - Design and ...

Adelaide, Australia

  • Remote
  • Hybrid
  • Full-time
  • IT
... lifecycle. - Contribute to Jira Coding Agent and Autodev. - Help drive increased agent invocations across the AI SDLC. - Work as a backend-leaning full-stack engineer with a strong product mindset. Requirements - Senior-level full-stack software engineering experience with a backend focus. - Strong technical proficiency. ...

Australia