As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines for ingestion, transformation, and loading of data into Azure Synapse Analytics. You will work in a fast-paced environment, adapt to evolving technologies, and contribute to the success of data-driven initiatives within the organization. This role requires a proactive and resourceful individual with a solid understanding of data engineering, data analytics, and cloud architecture.

Essential Functions

• Data Integration and ELT Development: Design, develop, and maintain data pipelines for ingestion, transformation, and loading of data into Azure Synapse Analytics. This includes understanding functional and non-functional requirements, performing source data analysis, data profiling, and implementing efficient ELT processes.
  
  
• Azure Synapse Development: Work with Azure Synapse Analytics to build and optimize data models, SQL queries, stored procedures, and other artifacts necessary for data processing and analysis.
  
  
• Data Lake File Handling: Understand the characteristics of various file formats, optimizing data storage, and implementing efficient data reading and writing mechanisms for incremental updates within Azure Synapse Analytics.
  
  
• Data Governance and Security: Ensure compliance with data governance policies and implement security measures to protect sensitive data stored in Azure. This involves encryption, masking, and access control mechanisms.
  
  
• Performance Optimization: Continuously optimize data pipelines and storage configurations to improve performance, scalability, and reliability. This includes identifying bottlenecks, query tuning, and leveraging Azure Synapse Analytics features for parallel processing.
  
  
• Monitoring and Troubleshooting: Implement monitoring solutions to track data pipeline performance, data quality, and system health. Troubleshoot issues related to data ingestion, transformation, or storage, and provide timely resolutions.

Skills Needed to be Successful

• Education: Bachelor’s degree in computer science, engineering
• Minimum of 10 years of experience in a relevant data engineering role.
• Relational Database Experience: Proficiency with one or more of the following database platforms; e.g. Oracle, Microsoft SQL Server, PostgreSQL, MySQL/MariaDB
  
  
• Proficiency in SQL: Strong SQL skills, including experience with complex SQL queries, stored procedures, and performance optimization techniques. Familiarity with T-SQL for Azure Synapse Analytics is a plus.
  
  
• ELT and Data Integration Skills: Proven experience in building ELT pipelines and data integration solutions using tools like Azure Data Factory, Oracle Golden Gate, or similar platforms. Ability to handle a variety of legacy data sources and file formats efficiently.
  
  
• Data Modeling and Warehousing Concepts: Familiarity with dimensional modeling, star schemas, and data warehousing principles. Experience in designing and implementing data models for analytical workloads.
  
  
• Analytical and Problem-Solving Abilities: Strong analytical skills with the ability to understand complex data requirements, troubleshoot technical issues, and propose effective solutions to meet business needs.
  
  
• Communication and Collaboration: Excellent communication skills with the ability to collaborate effectively with cross-functional teams, including Data Scientists, Reporting Analysts, and DevOps professionals.

Preferred Experience & Education

• Education: Master’s degree in computer science, engineering
• Experience with Azure Synapse Analytics: Hands-on experience in designing, developing, and deploying solutions using Azure Synapse Analytics, including familiarity with its various components such as SQL pools, Spark pools, and Integration Runtimes.
  
  
• Expertise in Azure Data Lake Storage: In-depth understanding of Azure Data Lake Storage, including its architecture, features, and best practices for managing a large-scale Data Lake or Lakehouse in an Azure environment.
  
  
• Experience with AI Tools: Experience with AI Tools and LLMs (e.g. GitHub Copilot, Copilot, ChatGPT) in automating many of the responsibilities outlined for this role.
  
  
• Knowledge of Avro and Parquet: Experience working with Avro and Parquet file formats, including data serialization, compression techniques, and schema evolution. Understanding of their advantages and use cases in a big data environment.
  
  
• Healthcare: Prior experience working with data in a healthcare or clinical laboratory environment and a strong understanding of PHI, GDPR & HIPPA/HITRUST is highly desirable.
  
  
• Certifications: Relevant certifications such as Azure Data Engineer Associate or Azure Synapse Analytics Developer Associate are highly desirable.

Supervisory Responsibilities

• No direct reports