Data Engineer - ETL/Python/Big Data
- Huquo Consulting
- 3 to 7 Yrs
- Bengaluru
- 12 months ago
- Email to a friend
- Report this job
Job Description
Must Have :
- Software Engineer with experience (3+ years) in designing / implementing ETL / ELT processes for on-prem databases / cloud warehouses using combination of Python scripting (Spark), SQL scripting & Data Engineering Tools (Cloud services / Open-Source Tools)
- Solid background in database concepts, ETL architectures / strategies / administration to be able to design compute-efficient data pipelines
- Prior experience working with large scale data (big data) processing pipelines / engines using PySpark
Good to Have but not necessary :
- Ability to design pipelines for analytics / ML / AI workflows and deploy models / solutions to production
- Familiarity / Experience in SWE best practices for unit-testing, code modularization, QA
- Coursework / Past Projects / Github Repos illustrating familiarity with data warehousing best practices
- Experience working with open-source pipeline orchestration frameworks like Airflow, Prefect, Luigi
- Familiarity with application containerization tools like Docker, Kubernetes
Job Responsibilities :
- Responsible for the design, development, and implementation of data integration processes using SQL / Python Scripting / Azure Data Flow / Other Cloud Platforms & Tools
- Responsible for setting up end to end data pipeline which includes importing, cleaning, transforming, validating and analyzing data with the purpose of understanding or making conclusions from the data for data modeling, data integration and decision making purposes.
- Collaborate with business users, analyze user requirements, translate & apply business rules to data transformations
- Lead and mentor junior developers in information integration standards, best practices, etc.
- Create functional & technical documentation - e.g. data integration architecture flows, source to target mappings, ETL specification documents, run books, test plans
- Performs data collection, profiling, validation, cleansing, analysis, and reporting. Tests, debugs, and documents ETL processes, SQL queries, and stored procedures
- Initiates analysis to identify data inconsistencies and resolve in order to raise the level of data integrity
- Analyzes data volumes, data types and content to support the design of data architecture solutions
- Work with Data Warehouse Architects performing source system analysis, identification of key data issues, data profiling and development of normalized and star-snowflake physical schemas
Employement Category:
Employement Type: Full timeIndustry: Recruitment Services
Role Category: General / Other SoftwareIT Operations / EDP / MIS
Functional Area: Not Applicable
Role/Responsibilies: Data Engineer - ETL/Python/Big Data
Contact Details:
Company: HuquoLocation(s): Bengaluru
Keyskills: sql java data warehousing informatica python source system analysis data flow sql queries music making data modeling sql scripting business rules data profiling data collection system analysis data integration
Similar positions
Huquo Consulting
Huquo Consulting Pvt. Ltd HuQuo is an HR Analytics and Consulting organization to provide path-breaking HR services by deeply understanding the human quotient of organizations we work with, and to evolve the industry and make it future ready through HR Analytics.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in
