Lead Data Engineer

Dialog Axiata PLC
9 days ago
0 Applied
Expires on: Feb 14 2023

Ref.No 00002369

Description
  • Possesses a degree in IT / Mathematics / Statistics or an equivalent relevant professional qualification.
  • Certifications in Linux Unix (Red hat certified Engineer or system administrator or equivalent) / Business intelligence / DWH / Database management / Hadoop are proffered.
  • Minimum 5+ years of combined experience in Data modelling, Mining and Validation, Data Engineering (ETL/BI/DWH/Hadoop), SQL Scripting / Database Administrating (or equivalent).
  • Knowledge in Big data management / distributed data processing systems and related environments (hive/impala/pig/scoop/flume/oozie).
  • Skills in Statistical /Mathematics data modelling / statistical algorithms and hypotheses testing.
  • Understanding of Business intelligence / Predictions models and business insights.
  • Sound knowledge on cloud technologies (AWS/Azure/GCP).


Responsibilities
  • Deciding and implementing relevant summary levels based on business requirements to execute business trends/growth analysis and actionable insights on predictions, and to increase performance of reports/analysis generating powers by minimizing processing time.
  • Assembling large complex data sets and data models by providing accurate merged DWH by merging Hybrid systems, from managing detail levels on top of Hadoop/EMR distributed system and summaries level from Snowflake/Oracle (Exadata) system. Need to manage and decide types of summaries, schedules, retentions and data transformation methods (optimal data pipeline architecture).
  • Data Engineering functionalities from Informatica Bigdata Management/Airflow/Glue, that working with wide range of Hadoop related systems, such as Spark, Hive, kudu and etc, with other related JOB Managers.
  • Handle cloud migration & cloud related data pipelines
  • New Data integrations (source system) Planning and data mart designing based on new reports and analysis requirements from internal customers, that relevant to new business areas and new report /analysis requirement from current business areas, and find out and applying DataMart design changes on existing models to increase accuracy of reports and analysis to provide strong business insights.
  • Responsible to Implement and finetuning ETL monitoring and failures(delays) alert systems by using email alerts, SMS alerts and reporting tools with possible reconciliation, and deciding proper actions based on alert, resulting to maximize ETL SLA performances.
  • Applying Machine learning algorithms on large data sets, that available on DWH (Hadoop/EMR) systems and find out gaps and structural problems
  • Designs experiments, test hypotheses, and build models with conducting advanced data analysis and complex designs algorithm for use cases that define by business management.
  • R&D on advanced statistical and predictive modelling techniques to build, maintain, and improve on multiple real-time decision systems
  • Responsible for Information protection to prevent competitor’s threats related to information.
  • SQL/PL SQL scripting on Oracle/snowflake RDBMS system to handle both ETLs and Database management (from table creations to advance administrating functionalities) and Hadoop (hive, impala, spark, presto and PIG) scripting on ETLs and Unix Schell scripting on OS level Administrating.
  • Provide Root Cause Analysis for any outage and apply corrective action to cure the cause.
  • Responsible for performing monthly/quarterly performance analysis to identify the progresses/ bottlenecks, issues and new resources requirements.


Skills
Data engineering
Industry Sector