SKANDHA IT SERVICES PRIVATE LIMITED
This listing was posted on hirist.
Lead Data Engineer - Scala/PySpark (7-10 yrs) Hyderabad/Bangalore/Pune
Location:
Hyderabad/Bangalore/Pune
Description:
Lead Data Engineer Locations : Hyderabad, Bengaluru, Pune. Skills :- Programming : Python , Pyspark/Scala, Flask.- API Microservices , ECS Fargate.- Bigdata AWS : EMR, Glue, Athena, Redshift, Lambda.- Analytics : AWS Quicksight.- Database : Postgres RDS,Mongo (SQL/NoSQL).- Orchestration : preferably Airflow.About the Role : We are searching for a talented and versatile Full-Stack Data Engineer to join our growing team. You will play a crucial role in designing, developing, and deploying data pipelines and solutions. You will leverage your expertise in Python, Big Data technologies on AWS, and orchestration tools to build and maintain scalable data infrastructure for advanced analytics. Responsibilities : - Design, develop, and implement data pipelines for data ingestion, transformation, and loading using Python and frameworks like PySpark/Scala. - Build and deploy microservices using Flask for data processing tasks. - Utilize containerization technologies like AWS ECS Fargate for running data processing workloads. - Work with Big Data tools on AWS, including EMR, Glue, Athena, Redshift, and Lambda, to manage and analyze large datasets. - Develop and maintain data pipelines for streaming and batch data processing. - Utilize AWS Quicksight to create interactive data visualizations and dashboards. - Design and manage data models for both relational databases (Postgres RDS) and NoSQL databases (MongoDB). - Implement Apache Airflow (preferred) or other orchestration tools to automate data pipelines and workflows. - Write clean, maintainable, and well-documented code. - Collaborate with data analysts, data scientists, and software engineers to understand data requirements and deliver effective solutions. - Perform unit testing and integration testing for data pipelines. - Monitor and maintain data pipelines for performance and stability. - Stay up-to-date with the latest advancements in data engineering technologies and best practices. Qualifications : - Proven experience in data engineering principles and practices. - Proficiency in Python programming with experience in data manipulation libraries (e.g., Pandas, NumPy). - Working knowledge of PySpark/Scala for distributed data processing. - Familiarity with building and deploying microservices using Flask. - Experience with containerization technologies like Docker and ECS Fargate. - Experience with AWS Big Data services, including EMR, Glue, Athena, Redshift, and Lambda. - Experience with data visualization tools like AWS Quicksight (a plus). - Experience with relational databases (Postgres RDS) and NoSQL databases (MongoDB). - Experience with data orchestration tools, preferably Apache Airflow. - Strong understanding of data modeling concepts for both SQL and NoSQL databases. - Excellent analytical and problem-solving skills. - Effective communication and collaboration skills. - Ability to work independently and as part of a team (ref:hirist.tech)
Visit Our Partner Website
This listing was posted on another website. Click here to open: Go to hirist
Important Safety Tips
- Always meet the employer in person.
- Avoid sharing sensitive personal and financial information.
- Avoid employment offers that require a deposit or investment.
To learn more, visit the Safety Center or click here to report this listing.
More About this Listing: Lead Data Engineer - Scala/PySpark (7-10 yrs) Hyderabad/Bangalore/Pune
Lead Data Engineer - Scala/PySpark (7-10 yrs) Hyderabad/Bangalore/Pune is a Engineering Engineer Job at Skandha IT Services located in India. Find other listings like Lead Data Engineer - Scala/PySpark (7-10 yrs) Hyderabad/Bangalore/Pune by searching Oodle for Engineering Engineer Jobs.
Lead Data Engineer - Scala/PySpark (7-10 yrs) Hyderabad/Bangalore/Pune is a Engineering Engineer Job at Skandha IT Services located in India. Find other listings like Lead Data Engineer - Scala/PySpark (7-10 yrs) Hyderabad/Bangalore/Pune by searching Oodle for Engineering Engineer Jobs.