Overview
Role Overview : As a Data Engineer on our Support Team, you will be responsible for maintaining robust data pipelines to support our products and services. You will work closely with cross-functional teams to understand data requirements and implement efficient solutions using PySpark. The ideal candidate will have a strong background in data engineering, exceptional problem-solving skills, and a passion for working with large-scale datasets.
Key Responsibilities
- Perform ongoing maintenance and optimization of existing data pipelines built with PySpark.
- Monitor pipeline performance and proactively identify areas for improvement to enhance efficiency and reliability.
- Maintain scalable data pipelines using PySpark to support various business needs.
- Ensure that all data is getting refreshed automatically daily as per SLA and data quality is being permanently monitored.
- Troubleshoot and resolve complex issues of Foundry users within SLAs.
- Optimize and tune existing data pipelines for improved performance and reliability.
- Troubleshoot and resolve data-related issues on time to ensure smooth operation of production systems.
- Document processes, procedures, and troubleshooting steps to facilitate knowledge sharing and maintain system integrity.
- Ensure immediate communications in case of Foundry outages and delays, but also for upcoming changes and other support information to end users.
- Implement continuous improvement of existing processes.
Requirements
3-4 years of experience working as a Data Engineer, strongly focusing on building data pipelines using PySpark.Proficiency in Python programming and experience with relevant libraries and frameworks.Solid understanding of distributed computing principles and experience with Apache Spark.Experience working with large-scale datasets and designing efficient data processing workflows.Strong analytical and problem-solving skills, with the ability to troubleshoot complex issues in production environments.Excellent communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.Intermediate+ English levelNice to have
Readiness to work with TypeScriptExperience with Palantir Foundry or similar data integration and analytics platforms is a plusWe offer
Flexible working format - remote, office-based or flexibleA competitive salary and good compensation packagePersonalized career growthProfessional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)Active tech communities with regular knowledge sharingEducation reimbursementMemorable anniversary presentsCorporate events and team buildingsOther location-specific benefitsNot applicable for freelancersSeniority level
Not ApplicableEmployment type
Full-timeJob function
Information Technology and EngineeringIndustries
Software Development#J-18808-Ljbffr