Job Description Data Engineer
For our client, a young dynamic sourcing platform, we’re looking for a Data Engineer.
Our client is on a mission to transform the professional services industry. They’re starting first in IT where, with the support/endorsement of some of people like Microsoft, they’re helping an ecosystem of vendors, distributors, Partner work together more efficiently and effectively, to deliver better solutions for customers.
As a data engineer you will be a part of our data and working closely data scientists, analysts and the development team. The data engineer will be key in helping to define and shaping the data landscape within a fast-growing start-up
Your main responsibilities:
- You will be responsible for developing and monitoring the data pipelines feeding in and from our data lake
- Work with data science to productionize models into performant and reliable prediction services
What are we looking for?
- 2+ years of experience in a similar role
- Experience within a digital business
- Python development experience for data products, you love clean code and know how to structure your python packages
- You have developed solutions with (Py)Spark, and have use Spark Streaming
- Experience with real-time event processing (using EventHub, Kafka, Kinesis etc)
- Implemented a data platform on a cloud provider, preferably experience with developing on Azure
- Experience with orchestration tools such as Airflow, Azkaban, Azure Data Factory
- You have worked in lean startup and/or agile environments
- Accustomed to work in an international environment and speaks and writes English excellent;
- Understanding of lambda architecture
- Experience with Graph Databases (Neo4J)
- Experience with setting up CI/CD pipelines for data products