For our client, a global consulting company based in Dublin and Cork we are looking for a Data Test Engineer (Databricks)
In this role, you will design, develop, and implement automated testing frameworks for the Databricks-based data platforms. You’ll work closely with data engineering teams to ensure data pipelines, Delta Lake tables, and streaming workflows meet the highest standards of accuracy and reliability within the Lakehouse architecture.
This is a permanent position hybrid structure in Dublin or Cork
Responsibilities:
Develop automated tests for Databricks notebooks, workflows, and Delta Lake tables
Build test frameworks to validate Delta Live Tables and structured streaming pipelines
Automate testing of ETL/ELT data pipelines, Spark jobs, and data transformations
Implement quality assurance strategies, including data reconciliation and performance testing
Monitor pipeline health through automated test dashboards and reporting
Maintain thorough test documentation and contribute to continuous improvement
Requirements:
Degree in Computer Science, Data Science, or related field
2+ years hands-on experience with Databricks (Spark), Python (PySpark), and SQL
Familiarity with data testing frameworks and AWS services (S3, Glue, Lambda)
Strong understanding of Delta Lake, Lakehouse architecture, and Unity Catalog governance
Experience with version control (Git) and Agile methodologies
Excellent analytical, problem-solving, and communication skills