Senior Data Engineer
Primary Responsibilities: ETL Development:
Design, implement, and sustain high-performance, scalable, and dependable ETL processes within the Snowflake platform.
Collaborate with cross-functional teams to identify business requirements and convert them into detailed ETL specifications.
Analyze source systems, data models, and structures to ensure data quality and uniformity across systems.
Data Workflow Management:
Utilise GCP services, including Cloud Functions and Cloud Composer/Airflow, to oversee data workflows.
Ensure robust version control using Git for extensive projects, managing frequent code updates, merges, and branches to uphold code integrity and facilitate collaboration.
Data Transformation and Validation:
Conduct data profiling, cleansing, and validation for accurate data transformation and integration.
Develop and maintain comprehensive ETL process documentation, encompassing data flow diagrams and data mapping documents.
Performance Optimization:
Enhance ETL performance through the implementation of best practices, continuous performance monitoring, and effective troubleshooting.
Collaborate closely with data warehouse and database administrators to achieve optimal data storage and retrieval.
Collaboration and Feedback:
Participate actively in code reviews, peer feedback sessions, and initiatives focused on continuous improvement.
Provide technical support and guidance on ETL processes and data integration.
Knowledge and Trend Monitoring:
Stay abreast of industry trends and best practices in ETL development, data integration, and cloud-based data warehousing.
Requirements:
Education and Experience:
Possess a Bachelor's degree in Computer Science, Information Systems, or a related field.
Accumulate a minimum of 5 years of experience in ETL development, data integration, or data warehousing.
Technical Skills:
Demonstrate strong proficiency in cloud-based data warehouse platforms, specifically Snowflake.
Familiarity with DBT for data transformation is preferred but not mandatory.
Showcase expertise in SQL for data querying and transformation.
Experience with relational databases such as Postgres and SQL Server.
Proficiency in Python or other relevant programming languages.
Understanding of data modeling concepts and hands-on experience in integrating database objects with applications.
Experience with GCP services, especially Cloud Functions and Cloud Composer/Airflow.
Familiarity with version control, CI/CD tools, and best practices.
Soft Skills:
Exhibit strong analytical and problem-solving skills.
Possess excellent communication and collaboration skills.
Ability to manage multiple tasks efficiently in a fast-paced environment.
Security and Compliance:
Familiarity with data privacy and security best practices in cloud environments.