Julkaistu 3 viikkoa sitten
Data Engineer
- Etätyö
- Kokoaikainen
- AWS
- Azure
- Data Engineering
- Python
Työpaikan kuvaus
• Design, develop, and maintain scalable and reliable data platforms and pipelines for analytics and operational use cases • Build distributed data processing workflows using Apache Spark, pySpark, and Databricks • Develop and evolve cloud-native data solutions leveraging Azure and AWS services • Model and optimize data structures for analytics, reporting, and downstream consumption • Integrate data from multiple heterogeneous sources while ensuring data quality, consistency, and traceability • Implement data quality controls, schema validation, lineage tracking, and monitoring processes • Optimize performance, scalability, reliability, and cost-efficiency of data pipelines and infrastructure • Collaborate closely with scientists, analysts, and engineering teams to translate business and research requirements into technical solutions • Contribute to CI/CD processes, infrastructure automation, and engineering best practices • Research and introduce new technologies and approaches within the data engineering ecosystem
Qualifications
• 3+ years of commercial experience in Data Engineering • 2+ years of hands-on production experience with Databricks • Strong expertise with Delta Lake, including medallion architecture and table optimization • Advanced proficiency in Python, pySpark, and SQL • Deep practical knowledge of Databricks components including Unity Catalog, Lakeflow Jobs / Workflows, and job optimization • Hands-on experience with AWS services including S3, IAM, KMS, and at least one of AWS Batch, ECR, EventBridge, Lambda, or API Gateway • Experience designing event-driven and asynchronous data pipelines • Strong knowledge of Infrastructure as Code and CI/CD practices • Practical experience with Terraform and Databricks Asset Bundles • Experience working with Git workflows, pull request reviews, and automated testing • Understanding of secure authentication approaches such as OIDC and service principals • Practical experience with MLflow, experiment tracking, model registry, and reproducibility practices • Strong understanding of data quality, lineage, schema contracts, and provenance concepts • Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks DQX • Experience monitoring and optimizing data platform reliability and cloud costs • Upper-Intermediate or higher English level
WILL BE A PLUS:
• Experience packaging workloads using Docker and running batch workloads in AWS • Familiarity with ECR and containerized data processing solutions • Experience with experiment optimization libraries such as scikit-optimize, BoTorch, Ax, or scipy • Understanding of ML feature engineering and leakage prevention techniques • Experience working in regulated or high-security environments • Knowledge of data governance, compliance, and data residency requirements • Experience disabling or restricting AI-assisted tooling/features in secure environments
Additional Information
Lisää työpaikkoja: Sigma Software
Senior Data Engineer
Sigma Software Etätyö · Varsova, Mazovia, Puola🇵🇱
Data Engineer
Sigma Software Etätyö · Varsova, Mazovia, Puola🇵🇱
Senior Data Engineer (Cybersecurity)
Sigma Software Etätyö · Varsova, Mazovia, Puola🇵🇱
Data Engineer (DATABRICKS / PYSPARK)
Sigma Software Etätyö · Varsova, Mazovia, Puola🇵🇱
Muita työpaikkoja: Puola
Engineering Manager (Java / Kotlin) - Product Catalog
Allegro Sp. z o.o. Poznań, Ison-Puolan voivodikunta, Puola🇵🇱
QC Inspektor/ka - Goleniów, PL
Vestas Wind Systems A/S Goleniów, Länsi-Pommerin voivodikunta, Puola🇵🇱
People Operations Intern
Dolby Laboratories, Inc. Wrocław, Ala-Sleesian voivodikunta, Puola🇵🇱
Konsultant (K/M/X)_1/2 etatu_Galeria Mokotów
Sephora Varsova, Mazovia, Puola🇵🇱
Konsultant (K/M/X)_1/2 etatu_Reduta Warszawa
Sephora Varsova, Mazovia, Puola🇵🇱
Global Business Process Expert Invoice To Pay (all genders)
Susonity Wrocław, Ala-Sleesian voivodikunta, Puola🇵🇱
Freight Forwarder (M/F/D)
dsvas Toruń, Kujawsko-Pomorskie, Puola🇵🇱
Dyspozytor / Dyspozytorka (M/K/N)
dsvas Toruń, Kujawsko-Pomorskie, Puola🇵🇱
Konsultant - Sprzedawca (K/M)
Jerónimo Martins, S.G.P.S., S.A. Szczecin, Länsi-Pommerin voivodikunta, Puola🇵🇱
Sprzedawca-Lada Tradycyjna (m/k) I Nowy Sklep I Białołęka I Mile widziane osoby z niepełnosprawności
Jerónimo Martins, S.G.P.S., S.A. Useita sijainteja🇵🇱
Software Lifecycle Manager (IT Analyst)
Nordea Bank Abp Gdynia, Pomerania, Puola🇵🇱