Manager/Associate Manager [Data Engineer (Big Data & Real-Time Streaming)]

Job Description

Title: Manager/Associate Manager [Data Engineer (Big Data & Real-Time Streaming)]

Company Name: BRAC Bank PLC

Vacancy: --

Age: Na

Job Location: Dhaka

Salary: --

Experience:

• 4 to 8 years
• The applicants should have experience in the following business area(s): Banks

• 4 to 8 years
• The applicants should have experience in the following business area(s): Banks

Minimum 4–8 years of professional experience in relevant fields.

Big Data Core & Storage

• Deep understanding of the Hadoop Ecosystem, specifically HDFS for storage and YARN for resource management.

• Proven experience with Apache Hive and Impala for data warehousing and interactive SQL queries on big data.

Ingestion & ETL

• Expertise in Apache NiFi for automating data flow between systems.

• Proficiency with Apache Sqoop for efficiently transferring bulk data between Apache Hadoop and structured (RDBMS), Unstructured, Semi Structured (MongoDB, Cassandra, HBase) datastores.

Processing & Computation

• Strong programming skills in Python and PySpark for writing complex transformations and distributed processing logic.

• Advanced knowledge of SQL for data manipulation and analysis.

Real-Time Streaming

• Hands-on experience with Apache Kafka for building real-time data pipelines and streaming apps.

• Experience with Apache Flink for complex event processing and streaming analytics.

API Integration & Development

• Strong experience designing, consuming, and maintaining RESTful APIs.

• Proficiency in handling JSON/XML data formats and parsing complex API responses.

• Experience with API authentication methods (OAuth2, API Keys, JWT).

• Experience with Python API frameworks (FastAPI, Flask) to build micro-services for data access.

Proficiency in SQL and relational databases.

Strong communication and interpersonal skills.

Customer-centric, team-oriented, self-motivated, and capable of meeting deadlines.

BRAC Bank PLC is a full-service private commercial bank in Bangladesh, aiming to become the most trusted, innovative and inclusive bank, driven by purpose and powered by people. It plays a pivotal role in shaping the country`s financial future, with a clear focus on sustainability. With the highest market capitalization, the highest international shareholding and the country’s best credit rating, the bank leads the Bangladesh banking industry in key financial metrics. BRAC Bank is considered the standard bearer of good governance, ethics and values-based banking. BRAC Bank today serves as a trusted partner for individuals and businesses across every segment of the economy - Small & Medium Enterprises (SME), Corporate & Institutional and Retail Banking.

BRAC Bank seeks an ambitious, intelligent, purpose-driven, and enthusiastic individual for the following position in its Technology Division:

Manager/Associate Manager [Data Engineer (Big Data & Real-Time Streaming)]

Employment Type: Full-time

Job Grade: SO-SPO

We are looking for skilled and passionate Data Engineer to join our data infrastructure team. In this role, you will design, build, and maintain scalable data pipelines that handle massive volumes of data. You will play a pivotal role in bridging the gap between traditional batch processing and modern real-time data streaming, ensuring our data is accurate, accessible, and timely for analytics.

KEY RESPONSIBILITIES:

• Pipeline Development: Design and implement robust ETL/ELT pipelines using Apache NiFi and Sqoop to ingest data from various sources (RDBMS, Logs, APIs) into our Big Data ecosystem.

• Big Data Processing: Develop high-performance batch processing applications using PySpark on Hadoop/YARN clusters.

• Real-Time Architecture: Architect and maintain real-time streaming solutions using Apache Kafka for message queuing and Apache Flink for stateful stream processing.

• Data Storage & Management: Manage and optimize data storage within HDFS and ensure efficient query performance using complex SQL queries.

• Data Warehousing & Querying: Specific high-performance data models and execute complex queries using Hive, Impala, and advanced SQL to support analytics teams.

• Cluster Management: Monitor and troubleshoot jobs running on YARN to ensure optimal resource allocation and system stability.

• API-Based Data Ingestion: Design and maintain automated workflows to fetch data from third-party REST APIs using Python or Apache NiFi processors.

• Performance & Security: Optimize API calls (pagination, rate limiting, authentication/OAuth) and tune Hadoop/Spark jobs for performance.