Data Engineer

About the job Data Engineer Job Title: Data Engineer Experience: 8 to 10 Years ?Time Zone: IST Time?? Job Type: Remote Work Location: - ?Domain: - ?? Responsibilities: • Data Pipeline Architecture: Design, develop, and optimize end-to-end data pipelines to extract, transform, and load (ETL) data from various sources into our data warehouse. Ensure data quality, reliability, and performance throughout the pipeline. • Data Modeling and Schema Design: Work with data scientists, analysts, and stakeholders to understand data requirements and create scalable and efficient data models. Implement and maintain database schemas that facilitate easy data access and querying. • Data Integration: Integrate data from diverse internal and external sources, including databases, APIs, and third-party systems. Build connectors and adaptors to ensure seamless data flow between systems. • Performance Optimization: Continuously monitor and fine-tune the performance of data pipelines and databases... Identify bottlenecks and implement optimizations to enhance processing speed and resource utilization. • Data Security and Governance: Implement robust security measures to safeguard sensitive data. Ensure compliance with data protection regulations and industry best practices for data governance and privacy. • Data Transformation and Enrichment: Develop data transformation routines to enrich raw data and make it suitable for analytical processing. Apply data cleansing, aggregation, and normalization techniques as needed. • Data Monitoring and Error Handling: Establish monitoring systems to detect data inconsistencies, anomalies, and errors. Develop automated alerts and error handling processes to ensure data integrity. • Technology Evaluation and Implementation: Stay up-to-date with the latest data engineering technologies and best practices. Evaluate new tools and frameworks, and lead the implementation of suitable technologies to improve data processing efficiency and scalability. • Documentation and Collaboration: Maintain comprehensive documentation for data engineering processes, data dictionaries, and workflows. Collaborate with cross-functional teams to understand their data needs and deliver effective solutions. • Mentoring and Leadership: Mentor and provide guidance to junior data engineering team members. Act as a technical leader, driving innovation and best practices within the data engineering team. Requirements: • Bachelor's or Master's degree in Computer Science, Engineering, or a related field. • Proven track record of at least 8 years of experience in data engineering roles. • Expertise in building and maintaining data pipelines using ETL tools like Apache Spark, Apache Airflow, or similar. • Strong proficiency in SQL and database technologies (e.g., PostgreSQL, MySQL, NoSQL databases). • Extensive experience with cloud-based data platforms, such as AWS, Azure, or Google Cloud Platform. • Solid understanding of data modeling concepts and data warehousing principles. • Proficiency in at least one programming language (e.g., Python, Java) for data manipulation and automation. • Experience with data streaming technologies (e.g., Kafka, Kinesis) is a plus. • Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes) is beneficial. • Excellent problem-solving skills and the ability to work independently and as part of a team. • Strong communication and interpersonal skills to collaborate effectively with stakeholders across the organization Apply Job!