[VNG]Senior Data Engineer
|Nơi làm việc:
||Tp.Hồ Chí Minh
Mô tả công việc
As a Hadoop big data engineer, you will develop, operate and drive scalable and resilient data platform based on Hadoop ecosystem to address the business requirements:
- Ensure industry best practices around data pipelines, metadata management, data quality, data governance and data privacy
- Design and implement business-specific large-scale data processing pipelines
- Work with complex data structures, manipulate, cleanse data, and perform transformations to make insights from data.
- Responsible to Ingest data from files, streams, and databases. Process the data with Spark Scala, Kafka…
- Develop efficient software code for multiple use cases leveraging Spark and Big Data Technologies for various use cases built on the platform
- Provide high operational excellence guaranteeing high availability and platform stability.
- 3+ years experience in professional software development (e.g. Data Engineering)
- Hands on and proven technical experience in Big Data environments and projects.
- Experience with integration of data from multiple data sources and of multiple formats/types.
Nice to have:
- Experience in Hadoop ecosystem including HDFS, MapReduce, YARN…
- Experience in building large-scale data processing (batch-processing, stream processing)
- Experiences in (most of) these technologies:
- AWS Cloud native development
- AWS S3, Lambda
- Apache Spark
- SQL, Scala, Python, Java
- Big Data tools, such as MongoDB / NoSQL technologies
- Understanding of SLA and meeting Timelines for support activities
- Experience in Apache Kafka
- Experience with Data warehouse
- Experience in ETL
- Experience in Data management: (Governance, Architecture, Modelling, Quality, Integration)
- Experience in SQL and NoSQL Database
- Experience in Backend development using Java
- Experience in Data API