Vacancy description

Data Engineer in Syd, Australia

XPT Software Australia Pty Ltd

  • ContractFixed-term or freelance

Syd, Australia

 

Job Details:

Role Overview

We are looking for a Senior Data Engineer with strong hands-on experience in Azure Databricks and the Microsoft Azure data ecosystem. The role will design, build, optimise, and support scalable data pipelines and lakehouse solutions that enable analytics, reporting, data science, and enterprise decision-making.

What You'll Work On

• Design, build, and optimise scalable batch and streaming data pipelines using Azure Databricks, PySpark, Python, SQL, and Delta Lake.

• Develop reusable ingestion, transformation, validation, and publishing frameworks for large-scale enterprise datasets.

• Implement medallion or layered lakehouse patterns across raw, curated, and consumption-ready data zones.

• Integrate Azure Databricks with Azure Data Lake Storage Gen2, Azure Data Factory, Azure Key Vault, Azure DevOps, and related Azure services.

• Develop production-grade Databricks notebooks, jobs, workflows, libraries, and cluster configurations.

• Create complex SQL and PySpark transformations, data-quality checks, reconciliation logic, and error-handling routines.

• Optimise Spark workloads, storage layouts, partitioning, file sizes, caching, and query performance while controlling platform cost.

• Implement Delta Lake capabilities such as ACID transactions, schema enforcement, schema evolution, time travel, and merge operations.

• Build automated CI/CD pipelines for code promotion, configuration management, testing, and deployment across environments.

• Apply data security and governance controls using role-based access, secrets management, data masking, auditability, and controlled access patterns.

• Monitor data pipelines, troubleshoot failures, perform root-cause analysis, and deliver sustainable production fixes.

• Partner with Architects, Business Analysts, Data Analysts, Product Owners, and business stakeholders to convert requirements into reliable data products.

• Provide technical leadership, code reviews, engineering standards, and mentoring to other data engineers.

Key Skills and Experience

•8+ years of experience in data engineering, data integration, ETL/ELT, or large-scale data platform delivery.

• Strong hands-on experience with Azure Databricks, including notebooks, jobs or workflows, cluster configuration, debugging, and performance tuning.

• Advanced proficiency in Python, PySpark, and SQL for production-grade data engineering.

• Strong experience with Delta Lake and lakehouse architecture, including reliable incremental processing and data-quality controls.

• Practical experience with Azure Data Lake Storage Gen2 and Azure Data Factory.

• Experience implementing CI/CD using Azure DevOps, Git, branching strategies, automated testing, and environment-based deployment controls.

• Strong understanding of dimensional modelling, data warehousing, data lake design, metadata, lineage, partitioning, and file formats such as Parquet.

• Experience with orchestration, monitoring, alerting, operational support, incident analysis, and service-level expectations.

• Understanding of Azure identity, Key Vault, managed identities, networking, private endpoints, and secure data-platform patterns.

• Strong communication, documentation, problem-solving, code-review, and stakeholder collaboration skills.

Good to Have

• Experience with Microsoft Purview, Unity Catalog, Azure Synapse Analytics, Event Hubs, or Structured Streaming.

• Knowledge of infrastructure as code using Terraform or Bicep.

• Experience with dbt, Airflow, Kafka, Power BI, or machine-learning data pipelines.

• Experience in telecommunications or another high-volume enterprise environment.

• Relevant Microsoft Azure or Databricks certification.

Similar vacancies

More like this role

Insurance Australia Group (IAG)

Data Engineer

  • Full-time
  • Part-time
Responsibilities - Partner with business stakeholders to understand challenges, identify opportunities, and deliver measurable data solutions. - Design, build, and maintain scalable data pipelines using Python, SQL, and cloud technologies. - Develop reusable datasets, data models, and features for analytics, reporting, ...

Syd, Australia

Rio Tinto

Data Engineer

  • Remote
  • Hybrid
  • Full-time
... Leads What You Will Bring - A commitment to uphold and promote our values of Care, Courage, and Curiosity - SQL / Python proficiency - Experience with piping and data engineering is a must - Experience with large columnar datasets that require transformation (Proficiency in PySpark is a plus) - Proficiency with JavaScript/TypeScript ...

Perth, Australia

  • On-site
  • Contract
... technical stakeholders. - Mentor team members and promote engineering best practices. Skills & Experience - Strong experience in Data Engineering, Solution Design or Data Warehousing. - Advanced SQL and data transformation expertise. - Hands-on experience with Snowflake, Python, Spark, Azure Data Factory and Databricks. - Experience ...

Syd, Australia

  • Hybrid
  • Full-time
  • Contract
  • DevOps

$130,000 - $155,000

... Snowflake Location: Sydney/Hybrid Contract: 12-month fixed-term Salary: $130,000 - $155,000 + Super An established Australian organisation is looking for a hands-on Data Platform Engineer to support a significant uplift of its data engineering capabilities. The opportunity centres on improving how enterprise data is ingested, ...

Syd, Australia

  • Hybrid
  • On-site
  • Full-time
  • Contract
... architecture designs. - Strong stakeholder engagement skills and experience working across business and technical teams. Data Engineer - Demonstrated experience in a Data Engineer, Senior Data Engineer or similar hands-on engineering role. - Strong experience designing, building and maintaining data pipelines and data integration ...

Canberra, Australia

  • Full-time
... future-ready data products. You'll also have an opportunity to implement SageMaker! Key responsibilities - Lead end-to-end data platform architecture - Design scalable data lake and data mesh solutions - Build reusable data engineering frameworks and tools - Partner with stakeholders to deliver effective data solutions - Drive DataOps, ...

Syd, Australia

Zone IT Solutions

AWS Data Engineer

  • Full-time
We are seeking an experienced AWS Data Engineer to design, build, and maintain scalable data pipelines and curated datasets on AWS. The ideal candidate will have strong expertise in SQL, Python, AWS data services, and modern data engineering practices, with experience delivering data solutions for enterprise analytics and ...

Syd, Australia

  • Hybrid
  • Contract
... intelligence and enterprise reporting solutions . - Strong technical experience in data engineering, data integration, ETL processes, data transformation and data modelling . - Demonstrated experience with Microsoft data and analytics technologies, including Power BI. - Strong understanding of data governance, data quality, ...

Canberra, Australia

ACH Group

Data Engineer

  • Full-time
... Apply data governance standards through metadata and lineage management, access controls, and data validation, reconciliation and quality monitoring to safeguard data integrity - Partner with stakeholders to deliver practical data solutions that meet business needs - Contribute to other data engineering activities as required ...

Mile End, Australia

  • On-site
  • Full-time
  • Data Analytics
  • Business Intelligence
Data Analyst / BI Developer - Brisbane (5 days onsite) We are seeking an experienced Data Analyst / BI Developer to build reporting and analytics capabilities. Key responsibilities: - Develop Power BI dashboards, DAX measures and executive reports. - Build and maintain Databricks data pipelines integrating Salesforce and ...

Brisbane, Australia