Back to list

Vacancy description

Principal Data Engineer in Docklands, Australia

Professional Search Group

  • RemoteWork from anywhere
  • HybridMix of office and remote
  • Full-timeStandard weekly hours
  • ContractFixed-term or freelance

$180,000 - $210,000

Docklands, Australia

Job Description

Lead AI & Data Platform Engineer

  • Location: Melbourne (Hybrid Work Model)
  • Salary: $180,000 – $210,000 + Super
  • Industry: Financial Services
  • Contract Type: Permanent / Full-Time

About the Role

We are seeking a high-caliber Lead AI & Data Platform Engineer to spearhead the evolution of our next-generation data architecture. In this role, you will bridge the gap between advanced Data Engineering and production-grade MLOps, designing resilient, petabyte-scale systems within a highly regulated Financial Services environment.

You will be our internal Databricks Expert , driving the strategy and deployment of AI Agentic workloads, advanced analytics pipelines, and secure cloud infrastructure.

Key Responsibilities

1. Databricks Platform Excellence

  • Architect and maintain our enterprise Lakehouse infrastructure utilising Delta Lake and Medallion topology (Bronze/Silver/Gold layers).
  • Enforce robust data governance and secure discovery across the enterprise using Unity Catalog .
  • Build, monitor, and scale production pipelines via Delta Live Tables .

2. Production AI & MLOps Infrastructure

  • Deploy, monitor, and scale production-grade ML and AI Agentic workloads.
  • Operationalise LLM and Generative AI pipelines utilizing Databricks Mosaic AI , MLflow , Langchain , and Vector Databases .
  • Manage corporate feature stores and optimize model serving endpoints.

3. AWS Cloud Architecture & Infrastructure as Code (IaC)

  • Design secure, high-performance cloud environments using Terraform.
  • Manage intricate AWS configurations including S3 optimization, VPC peering, AWS PrivateLink, and granular IAM roles.
  • Integrate cloud-native AI services like AWS Bedrock seamlessly into our data ecosystem.

4. Performance Tuning & Streaming

  • Conduct advanced Spark optimization to handle high-concurrency, petabyte-scale data pipelines.
  • Tune GPU clusters specifically optimized for heavy AI/ML training and inference workloads.
  • Architect real-time streaming pipelines leveraging Kinesis or Kafka.

Your Profile

To be successful in this role, you will bring a blend of elite engineering skills and a deep understanding of modern data-ops.

Technical Requirements:

  • Core Languages: Mastery of PySpark , Python , SQL , and Scala .
  • AI Frameworks: Strong familiarity with PyTorch , TensorFlow , or Hugging Face .
  • DataOps & CI/CD: Proven experience embedding automated testing and CI/CD pipelines into data deployments.
  • Industry Experience: Prior experience within Financial Services or a similarly regulated environment is highly desirable (understanding of data privacy, compliance, and risk frameworks).

What’s on Offer

  • Competitive Compensation: $180K – $210K base salary, depending on experience.
  • Hybrid Flexibility: A healthy balance of remote work and collaborative time in our modern Melbourne CBD office.
  • Cutting-Edge Tech Stack: Total backing to utilize the absolute latest in Generative AI, Mosaic AI, and cloud-native tools.
Loading similar vacancies…