Senior Data Engineer

PickMe
22 days ago
tie
3 Applied
Expires on: Jul 19 2024
tie

Ref.No 00005198

Description

Responsibilities

  • Data Pipelines development for online and batch processing from hundreds of data sources using the unified ETL framework
  • ETL framework design and development for online and batch processing
  • Optimization of ETL: unification, decrease of development TTM, increase of code readability
  • Perform code reviews
  • Physical and logical database design
  • System/components/modules documentation
  • Develop data feeds for external data consumers
  • Adoption of the new technologies, used to improve the performance of the data platform, indexing engines, APIs
  • Develop, construct, test and maintain architectures
  • Align architecture with business requirements
  • Identify ways to improve data reliability, efficiency and quality
  • Use large data sets to address business issues
  • Prepare data for predictive and prescriptive modeling
  • Find hidden patterns using data
  • Use data to discover tasks that can be automated
  • Deliver updates to stakeholders based on analytics


Requirements

  • BS in Information Management, Big Data, Computer Science or related field
  • 1+ years or 2+ years experience in modern data lake and big data development
  • Knowledge of technology best practices for building a modern data lake, data warehouses and data pipelines
  • Good understanding of technologies and experience in building a highly scalable and fault tolerant cloud data platform
  • Self-starter, capable of working without direction and able to deliver projects from scratch.
  • Good practical experience and knowledge in building and maintaining Data Warehousing/Big Data Tools - Hadoop and MapReduce, Apache Spark and Spark SQL, HIVE.
  • In-Depth Database Knowledge of RDBMS (PostgreSQL and MySQL) and NoSQL (Hbase).
  • Strong experience in building and maintaining cloud Big Data and ETL tool.
  • Strong knowledge and experience with Apache Spark in implementing batch and streaming data processing jobs, strong Development background in Scala ,Python or Java.
  • Strong knowledge in messaging systems like Kafka
  • Experience with Agile/Lean projects SCRUM, KANBAN etc.
  • Practical knowledge with Git flow, Trunk and GitHub flow branching strategies.
Skills
Data Science
Scala
Python
HiveQL
SQL
Statistics
problem solving
analytical
Industry Sector