Lead Data Engineer (Databricks)

Job Details

Our exclusive Client – a global financial services company based in Dublin city centre – have an opening for a Lead Data Engineer (Databricks). This person will lead and manage the data team, and take ownership for rollout of key Databricks solution on a core product for the business.
This is a hybrid working role (2 days in office / week) in Dublin city centre
 
Responsibilities

  • Lead and manage the Data team (local and dispersed) and collaborate with stakeholders to deliver robust, data-driven outcomes.
  • Design of end-to-end data pipelines using Databricks (Spark, Delta Lake, MLflow).
  • Integrate AWS services such as S3 / Glue / Lambda
  • Ensure compliance with industry standards (e.g., GDPR, SOC, ISO) and internal security policies.
  • Optimise Spark performance, automate infrastructure (Terraform/CloudFormation)
  • Champion CI/CD, observability, and agile data delivery.

 
Requirements

  • 8+ years relevant experience with 2+ years of Databricks (Spark)
  • Strong Python (PySpark) or Scala skills, plus solid SQL expertise.
  • Experience is agile data delivery
  • Proven leadership in building and managing high-scale, production-grade data environments.
  • Certifications in Databricks or AWS are a plus.

You may be interest in…