Responsibilities:

• Design and implement ETL pipelines using Databricks and Apache Spark.
• Collaborate with data scientists and analysts to understand data requirements and provide optimized solutions.
• Ensure the quality and integrity of data through data validation and testing processes.
• Monitor and troubleshoot performance issues within Databricks workflows and clusters.
• Develop and maintain documentation for data processing workflows and architectures.
• Perform data modeling and analysis to enhance data accessibility and usability.
• Stay updated on new features and best practices within the Databricks environment to leverage the latest advancements.

Requirements:

• Bachelor's degree in Computer Science, Information Technology, or related field.
• Proven experience working with Databricks and Apache Spark for data processing and analysis.
• Strong programming skills in languages such as Python, Scala, or SQL.
• Hands-on experience in data engineering workflows and ETL processes.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud.
• Knowledge of data warehousing concepts and architectures.
• Excellent problem-solving skills and ability to work collaboratively in a team environment.