Senior Data Engineer
About This Job
Trách nhiệm chính
- Build and maintain data pipelines within Databricks.
- Extract, transform, and validate data from multiple sources.
- Develop reusable datasets and analytical tables for Data Science projects.
- Ensure data quality, consistency, and reliability across analytical workflows.
- Translate requirements from Data Scientists into efficient and maintainable code.
- Develop data processing solutions using PySpark, SQL, and Pandas.
- Support feature engineering activities for machine learning and advanced analytics projects.
- Investigate data anomalies and support root‑cause analysis.
- Optimize data queries and processing logic for performance and scalability.
- Collaborate with engineering stakeholders to understand operational data sources and business requirements.
- Contribute to code reviews and technical discussions.
- Document datasets, code, and analytical processes.
- Identify opportunities to improve existing data pipelines and analytical processes.
- Promote coding standards and best practices.
- Stay up to date with developments in Databricks, Spark, and modern data platforms.
Yêu cầu
- Experience developing analytical solutions using Databricks.
- Strong proficiency in PySpark.
- Strong SQL skills, including writing and optimising complex queries.
- Experience using Python and Pandas for data analysis and transformation.
- Familiar with Airflow and Kafka.
- Experience with Docker and Kubernetes is a plus.
- Experience with other stream processing like Flink is a plus.
- Experience working with large‑scale datasets.
- Ability to understand business requirements and translate them into technical solutions.
- Strong problem‑solving and debugging skills.
- Good communication and stakeholder engagement skills.
- Experience working in collaborative Agile teams.
Yêu cầu ưu tiên
- Experience within engineering, rail, transportation, manufacturing, industrial, or asset‑intensive industries.
- Exposure to machine learning workflows and feature engineering.
- Experience with Delta Lake and Spark optimisation.
- Familiarity with Azure Data Platform technologies.
- Knowledge of Power BI or Grafana.
- Experience using Git/version control.
Mức lương
- Thỏa thuận
Phúc lợi
- Industry‑leading benefits, support, and services that look after your holistic health and wellbeing.
- Flexible arrangements that work for you (role and location dependent).
Hình thức làm việc
- Flexible arrangements depending on role and location.
Thông tin khác
- We are a global company operating across a wide range of industries and regions.
- We are committed to building an inclusive culture based on mutual respect and merit‑based systems.
- We welcome all applicants for employment without attention to race, colour, religion, sex, sexual orientation, gender identity, national origin, veteran, age, disability status or any other protected characteristic.