Job Description
Strong experience in ADF( Azure data factory), Azure SQL, Synapse, Spark/Databricks= 5+ Yrs Excellent written and verbal communication, intellectual curiosity, a passion to understand and solve problems, consulting & customer service Structured and conceptual mindset coupled with strong quantitative and analytical problem-solving aptitude Exceptional interpersonal and collaboration skills within a team environment PySpark Parquet files Big data and modern data lake Demonstrable experience in enterprise level data platforms involving implementation of end-to-end data pipelines Hands-on experience with Azure Experience with column-oriented database technologies (e.g., Synapse), NoSQL database technologies (e.g., DynamoDB, Cosmos DB, etc.) and traditional database systems (e.g., SQL Server, Oracle, MySQL) Experience in data pipelines and solutions for both streaming and batch integrations using tools/frameworks like Azure Data Factory, Azure functions and Stream. analytics Metadata definition and management via data catalogs, service catalogs, and stewardship tools such as OpenMetadata, and Azure Purview Test, plan, creation and test programming using automated testing frameworks, data validation and quality frameworks, and data lineage frameworks Data modeling, querying, and optimization for relational, NoSQL, timeseries, graph databases, data warehouses and data lakes Data processing programming using SQL, Python, and similar tools Logical programming in Python, Spark, PySpark, Java, Javascript, and/or Scala Cloud-native data platform design with a focus on streaming and event-driven architectures Participate in integrated validation and analysis sessions of components and subsystems on production servers Data ingest, validation, and enrichment pipeline design and implementation SDLC optimization across workstreams within a solution Bachelors degree in Computer Science, Engineering, or related field
Employement Category:
Employement Type: Full time
Industry: IT Services & Consulting
Role Category: Application Programming / Maintenance
Functional Area: Not Applicable
Role/Responsibilies: Sr Data Engineer-ADF
Keyskills:
ADF
Spark
Big data
Azure
SQL Server
Oracle
MySQL
Data validation
Data modeling
Optimization
Relational databases
Graph databases
SQL
Python
Java
Javascript
Scala
Validation
Azure data factory
Azure SQL
Synapse
Databricks
PySpark
Parquet files
Data lake
Columnoriented database
NoSQL database
Azure Data Factory
Azure functions
Stream analytics
Metadata definition
Data catalogs
Service catalogs
Stewardship tools
OpenMetadata
Azure Purview
Automated testing frameworks
Quality frameworks
Data lineage frameworks
Querying
NoSQL databases
Timeseries databases
Data warehouses
Data lakes
Logical programming
Cloudnative data platform design
Streaming architectures
Eventdriven architectures
Data ingest
Enrichment pipeline design
SDLC optimizat