Các Phúc Lợi Dành Cho Bạn
Mô Tả Công Việc
You will use various methods to transform raw data into useful data systems. For example, you’ll create algorithms and conduct statistical analysis. Overall, you’ll strive for efficiency by aligning data systems with business goals.
To succeed in this data engineering position, you should have strong analytical skills and the ability to combine data from different sources. Data engineer skills also include familiarity with several programming languages and knowledge of learning machine methods.
- Develop ETL/ELT solutions using Azure Bob Storage, Data Factory), Spark, and Python to load data across multiple sources into Azure Snowflake.
- Design, implement and maintain a robust, efficient and compliant data pipeline architecture.
- Create and maintain documentation on data pipelines
- Support various components of the data pipelines, including ingestion, validation, cleansing and curation.
- Manage and ensure the success of ongoing data pipeline routines.
- Help the team to maintain an overall position of strong data governance with great data observability (data monitoring & alerting, CI/CD pipelines).
- Build the infrastructure required for optimal acquisition, transformation, and provisioning of data from a variety of data sources.
- Design enhancements, updates, and programming changes for portions and subsystems of data pipelines, repositories, and models for structured/unstructured data.
- Engage with our software engineering team to ensure precise data points per application specifications
- Work with customer service, devops, and quality assurance teams to move software through the pipeline to the customer
Yêu Cầu Công Việc
- Bachelor's or master's degree in computer science, information systems, engineering, or equivalent
- Experience with relational SQL and NoSQL databases, including Postgres and MSSQL Server.
- Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
- Experience working within hybrid cloud environment; Azure experience is a plus
- Expertise with ETL/ELT and the development of automated validation and data pipelines
- Experience with object-oriented/object function scripting languages: .Net Core, C#, Python, Scala,etc
- Expert Understanding of CI/CD and DevOps methodologies
- Experience with Linux, Docker and Bash scripting.
- Experience with basics of search engines/indexing (e.g. Elasticsearch, Lucene)
- Experience with common distributed data tools in a production setting (Spark, Kafka, etc)
- Ability to work on high performing teams supporting multiple projects and personnel.
- Good English verbal communication is a big plus