Data Engineer

You are expert in Databricks.

You demonstrate and use your developer experience to develop and maintain complex data transformation pipelines under specific guidelines

You are familiar of working as data engineer in Databricks (on top of AWS) Your responsibilities will be to:

• Design, develop and maintain scalable data pipelines
• Manage data ingestion processes from various sources (xml ingestion, Kakfa, etc.).
• Implement and optimize ETL (Extract, Transform, Load) processes to ensure efficient data transformation and integration.
• Ensure data quality, integrity, and security across all data pipelines.
• Monitor and troubleshoot data pipeline performance, implementing necessary maintenance and updates.
• Document data pipeline processes, technical specifications and best practices.
• Realize knowledge transfer towards the Data engineer team which will take over the responsibility to run the platform in the future
• Expert in Databricks, Python, SQL, Energy Market Processes, DATAIKU and SAS