Principle Data Engineer

Required Skills:

Python, Pyspark/Spark, SQL, Data Lake (ON Prem or cloud) AWS, Snowflake.

Job Summary:

The Data Engineer’s role is to play a lead role in developing a high-performance data platform, integrating data from a variety of internal and external sources, in order to support data and analytics activities. This is a technical role that involves defining changes to the warehouse data model and building scalable and efficient processes to populate or modify warehouse data. The successful candidate will have hands-on data processing and data modeling experience in cloud and on-prem environments.

Responsibilities:

• Be a technical lead in the development of high-volume platforms to drive decisions and have a tangible beneficial impact on our clients and on business results.
• Design and implement efficient data pipelines (ETLs) in order to integrate data from a variety of sources into Data Warehouse.
• Design and implement data model changes that align with warehouse standards.
• Design and implement backfill or other warehouse data management processes.
• Develop and execute testing strategies to ensure high quality warehouse data.
• Provide documentation, training, and consulting for data warehouse users.
• Perform requirement and data analysis in order to support warehouse project definition.
• Provide input and feedback to support continuous improvement in team processes.
• Experience in working both, On-Prem and Cloud (AWS preferred).
• Responsible for leading the team onsite and off shore with technical leadership and guidance. 

Qualifications:

• 5+ years in a Data Engineering role
• 7+ years hands on experience with SQL:

• Ability to write/ interpret SQL and Complex Joins/ Queries
• Execution plan and SQL optimization (Oracle SQL Profiler)

15. o 3+ years coding experience (Python and/or PySpark).
16. o 3+ years hands on experience with big data and cloud technologies (Snowflake, EMR, Redshift, or similar technologies) is highly preferred

• Schema design and architecture on snowflake
• Architecture and design experience with AWS cloud
• AWS services expertise: S3, RDS, EC2, ETL services (Glue, Kinesis)
• Consumption layer design experience for reporting and dashboard

• Expert level understanding and experience of ETL fundamentals and building efficient data pipelines.
• 3+ years at least - Enterprise GitHub – branch, release, DevOps, CI/CD pipeline.
• Team player, Strong communication and collaboration skills.
• Experience with Agile methodologies.
• Master’s Degree (or a B.S. degree with relevant industry experience) in math, statistics, computer science, or equivalent technical field.