Senior Data Engineer

Responsibilities

• Data Pipelines development for online and batch processing from hundreds of data sources using the unified ETL framework
• ETL framework design and development for online and batch processing
• Optimization of ETL: unification, decrease of development TTM, increase of code readability
• Perform code reviews
• Physical and logical database design
• System/components/modules documentation
• Develop data feeds for external data consumers
• Adoption of the new technologies, used to improve the performance of the data platform, indexing engines, APIs
• Develop, construct, test and maintain architectures
• Align architecture with business requirements
• Identify ways to improve data reliability, efficiency and quality
• Use large data sets to address business issues
• Prepare data for predictive and prescriptive modeling
• Find hidden patterns using data
• Use data to discover tasks that can be automated
• Deliver updates to stakeholders based on analytics

Requirements

• BS in Information Management, Big Data, Computer Science or related field
• 1+ years or 2+ years experience in modern data lake and big data development
• Knowledge of technology best practices for building a modern data lake, data warehouses and data pipelines
• Good understanding of technologies and experience in building a highly scalable and fault tolerant cloud data platform
• Self-starter, capable of working without direction and able to deliver projects from scratch.
• Good practical experience and knowledge in building and maintaining Data Warehousing/Big Data Tools - Hadoop and MapReduce, Apache Spark and Spark SQL, HIVE.
• In-Depth Database Knowledge of RDBMS (PostgreSQL and MySQL) and NoSQL (Hbase).
• Strong experience in building and maintaining cloud Big Data and ETL tool.
• Strong knowledge and experience with Apache Spark in implementing batch and streaming data processing jobs, strong Development background in Scala ,Python or Java.
• Strong knowledge in messaging systems like Kafka
• Experience with Agile/Lean projects SCRUM, KANBAN etc.
• Practical knowledge with Git flow, Trunk and GitHub flow branching strategies.