A Senior Data Engineer responsible for designing, developing, and maintaining scalable data platforms and pipelines using cloud technologies like AWS, to support analytics, machine learning, and data governance initiatives within a growing energy solutions company.
Key Responsibilities
Build, automate, and manage real-time scalable data ingestion pipelines for master data management, deep learning, and predictive analytics
Develop and maintain cloud-native big data environments on AWS ensuring security, scalability, and performance
Lead data governance and profiling efforts to ensure data quality and proper metadata documentation
Design and implement data models, schemas, and processing logic in collaboration with data scientists, BI developers, and product managers
Create and optimize ETL processes for data validation, transformation, and feature modeling using Spark, Python, SQL, and related technologies
Establish and monitor SLAs for data availability and accuracy, automating data quality and availability monitoring
Communicate progress and results across organizational levels, facilitating decision-making and issue resolution
Requirements
A bachelor’s degree in computer science or information technology.
Minimum of 8 years of relevant experience in data engineering or related fields.
High proficiency in programming languages commonly used in ETL development, such as PLSQL, SQL, and Python.
Ability to write efficient SQL queries, SQL stored procedures, develop scripts for data transformations, and utilize programming frameworks and libraries to create and enhance ETL mappings and workflows.
Expertise in utilizing AWS services, including but not limited to Amazon S3, AWS Glue, AWS Data Catalog, Amazon Redshift, Redshift Spectrum, and Amazon Athena, to build scalable, reliable, and performant data pipelines and analytics solutions.
Proficiency in working with relational databases such as Postgres, Oracle, MySQL, or SQL Server, including knowledge of database design, optimization techniques, and advanced querying capabilities.
Experience in performance tuning and optimizing database operations.
Ability to lead data governance and data profiling efforts to ensure data quality and proper metadata documentation for data lineage.
Knowledge of data security best practices and familiarity with data governance frameworks.
Ability to build, automate, and manage near-real-time scalable data ingestion pipelines for master data management, deep-learning, and predictive analytics.
Experience designing and developing ETL processes to validate and transform data, calculate metrics, model features, and populate data models using Spark, Python, SQL, and other technologies in the AWS environment.
Demonstrated ability to lead by example, including best practices for code development, optimization, unit testing, CI/CD, performance testing, capacity planning, documentation, monitoring, alerting, and incident response to ensure data availability, quality, and usability.
Ability to define SLAs for data availability and correctness, automate data availability and quality monitoring, and respond to alerts when data delivery SLAs are not being met.
Excellent communication skills to clearly communicate progress and results across organizations and levels, from individual contributors to executives.
Benefits & Perks
Salary range of $140,000 - $165,000 annually, with a target of $156,750 based on experience and qualifications
Hybrid work opportunity with flexible work arrangements
Generous retirement package
Medical, dental, and vision insurance
Pre-tax contribution plans
Employee Stock Ownership Plan (ESOP)
Ready to Apply?
Join Energy Solutions and make an impact in renewable energy