Data Engineer

Data Engineer is responsible for designing, building, and maintaining the infrastructure and systems required for collecting, storing, and processing large datasets efficiently.

**Education**:Bachelor's degree in computer science with 8+ years of experience

**Experience**:

- Technical Skills
- Programming Languages: Proficiency in Python, SQL, Java, or Scala for data manipulation and pipeline development.
- Data Processing Frameworks: Experience with tools like Apache Spark, Hadoop, or Apache Kafka for large-scale data processing.
- Data Systems and Platforms
- Databases: Knowledge of both relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).
- Data Warehousing: Experience with platforms like Snowflake, Amazon Redshift and Azure Synapse.
- Cloud Platforms: Familiarity with AWS, Azure Cloud for deploying and managing data pipelines. Having Good experience in Fabric is advantageous
- Experience working with distributed computing systems such as Hadoop HDFS, Hive, or Spark.
- Managing and optimizing data lakes and delta lakes for structured and unstructured data.
- Data Modeling and Architecture
- Expertise in designing efficient data models (e.g., star schema, snowflake schema) and maintaining data integrity.
- Understanding of modern data architectures like Data Mesh or Lambda Architecture.
- Data Pipeline Development
- Building and automating ETL/ELT pipelines for extracting data from diverse sources, transforming it, and loading it into target systems.
- Monitoring and troubleshooting pipeline performance and failures.
- Workflow Orchestration
- Hands-on experience with orchestration tools such as Azure Data Factory, AWS Glue jobs, DMS or Prefect to schedule and manage workflows.
- Version Control and CI/CD
- Utilizing Git for version control and implementing CI/CD practices for data pipeline deployments.

**Key Skills**:

- Proficiency in programming languages such as Python, SQL, and optionally Scala or Java.
- Proficiency in data processing frameworks like Apache Spark and Hadoop is crucial for handling large-scale and real-time data.
- Expertise in ETL/ELT tools such as Azure ADF and Fabric Data Pipeline is important for creating efficient and scalable data pipelines.
- A solid understanding of database systems, including relational databases like MySQL and PostgreSQL, as well as NoSQL solutions such as MongoDB and Cassandra, is fundamental.
- Experience with cloud platforms, including AWS, Azure and their data-specific services like S3, BigQuery, and Azure Data Factory, is highly valuable.
- Data modeling skills, including designing star or snowflake schema, and knowledge of modern architectures like Lambda and Data Mesh, are critical for building scalable solutions.

**Role and Responsibilities**:

- Responsible for designing, developing, and maintaining data pipelines and infrastructure to support our data-driven decision-making processes.
- Design, build, and maintain data pipelines to extract, transform, and load data from various sources into our data warehouse and data lake.
- Proficient in creating data bricks creating notebooks, working with catalogs, native SQL, creating clusters, Parameterizing notebooks, and administrating data bricks. Define security models and assign roles as per requirement.
- Responsible for creating data flow in Synapse analytics integrating external source systems, creating external tables, data flows and create data models. Schedule the pipelines using various jobs, creating trigger
- Design and develop data pipelines using Fabric pipelines, spark notebooks accessing multiple data sources. Proficient in developing Data bricks notebooks and data optimization
- Develop and implement data models to ensure data integrity and consistency. Manage and optimize data storage solutions, including databases and data warehouses.
- Develop and implement data quality checks and validation procedures to ensure data accuracy and reliability.
- Design and implement data infrastructure components, including data pipelines, data lakes, and data warehouses.
- Collaborate with data scientists, analysts, and other stakeholders to understand business requirements and translate them into technical solutions.
- Monitoring Azure and Fabric data pipelines, spark jobs and work on fixes based on the request priority.
- Responsible for data monitoring activities, having good knowledge on reporting tools like Power Bi and Tableau is required.
- Responsible for understanding the client requirements and architect solutions in both Azure and AWS cloud platforms.
- Monitor and optimize data pipeline performance and scalability to ensure efficient data processing.