We are looking for a Data Engineer to join our team! In this role, you will work within an engineering team to design and develop data processing platforms and software frameworks for Data Science, Engineering and Analytics organizations.
Most of the things you’ll work on:
- Design and implement real-time and batch data processing frameworks
- Develop and manage scalable data processing platforms used by Data Science, Engineering and Analytics organizations
- Consult with Data Science and Analytics to provide best practices for data modeling
- Implement and maintain data pipelines through collection, storage, transformation and normalization of large data sets.
- Planning and estimating features the roadmap to determine the next sprint goals
- Fixing bugs and refactoring to improve Flipp internal and open source package
- Design and implement automated unit and integration test strategies
- Analyzing datasets and automating data quality checks to ensure top-notch data quality and consistency
- Develop proper data governance approaches by ensuring security, integrity and auditability standards are met
You’ll need to have:
- Bachelor’s degree Computer Science/Engineering or equivalent
- 2+ years of experience writing ETLs on big data platforms such as Hadoop, HDFS, Spark, Kafka
- 2+ years of designing and managing big data platforms such as Hadoop, Kafka or cloud based managed solutions.
- 2+ years of experience working with data processing languages such as pySpark, Hive, Scala & Spark
- Track record of shipping quality code and delivering high quality features
- High growth expectations for yourself and your team, and a willingness to push yourself and your team to achieve them