Hadoop Data Lake Automation Engineer Job at Cloudious LLC, Charlotte, NC

Ulg2elo1M1l4RWZJMVp4SHlNejd3V2l1
  • Cloudious LLC
  • Charlotte, NC

Job Description

Job Summary:

We are looking for a skilled and motivated Hadoop Data Lake Automation Engineer with 4 5 years of experience in automating data workflows and processes within Hadoop-based data lake environments. The ideal candidate will be responsible for building scalable automation solutions, optimizing data pipelines, and ensuring efficient data movement and transformation across platforms.

Key Responsibilities:

  • Design and implement automation solutions for data ingestion, transformation, and processing in Hadoop data lake environments.
  • Develop and maintain scalable data pipelines using tools such as Apache NiFi, Spark, Hive, and Sqoop.
  • Collaborate with data engineers, analysts, and business stakeholders to understand data requirements and deliver automation solutions.
  • Monitor and troubleshoot data workflows, ensuring reliability and performance.
  • Implement best practices for data governance, security, and metadata management.
  • Maintain documentation for data flows, automation scripts, and operational procedures.
  • Support production environments and participate in on-call rotations as needed.

Required Skills & Qualifications:

  • 3 5 years of hands-on experience in Hadoop ecosystem (HDFS, Hive, Spark, Sqoop, Oozie, etc.).
  • Strong experience in automating data lake workflows and ETL processes.
  • Proficiency in scripting languages such as Python, Shell, or Scala.
  • Experience with scheduling and orchestration tools (e.g., Apache Airflow, Control-M, AutoSys).
  • Solid understanding of data modelling, data quality, and performance optimization.
  • Familiarity with cloud platforms (AWS, Azure, GCP) and big data services.
  • Excellent problem-solving and communication skills.

Preferred Qualifications:

  • Experience with Apache NiFi or similar data flow tools.
  • Exposure to CI/CD pipelines and DevOps practices.
  • Knowledge of data cataloguing and lineage tools.

Job Tags

Similar Jobs

Cross Border Talents

Italian speaking reservations agent in Malta Job at Cross Border Talents

 ...As an Italian-Speaking Reservations Agent, you will be responsible for managing reservations and providing outstanding customer service to our...  ...systems and Microsoft Office Suite. Ability to work efficiently under pressure in a fast-paced environment.... 

Amazon.com Services LLC

Aviation Security Manager Job at Amazon.com Services LLC

 ...DESCRIPTION The Amazon Aviation Security (AvSec) team is seeking an enthusiastic AvSec Manager to provide leadership, support, and...  ...required by the Federal Aviation Administration (FAA), Department of Homeland Security (DHS), and TSA (as applicable). - This position may... 

Pure Tonic Dispensary

Dispensary Lead Job at Pure Tonic Dispensary

Dispensary Lead Location Sparks : Dispensary Lead **MUST HAVE OPEN AVAILABILITY OVERNIGHTS & WEEKENDS** Position Description Location: PURE TONIC - STOREY COUNTY Status: FULL-TIME Pay TYPE: HOURLY Reports to: Inventory Control Manager, Assistant General... 

Logo Daddy Graphics

Screen Print Production Employee Job at Logo Daddy Graphics

 ...We are seeking a Screen Print Production Employee to join our team! Responsibilities: Screen Print eye-catching graphics on a variety of apparel. Learn and perform all aspects of the screen print process. Help clean and maintain equipment and production... 

MH Energy

Class A Crude Oil Driver- Excellent Pay Job Job at MH Energy

 ...A Crude Oil Driver- Excellent Pay JobHauling crude oil out of the oil fields. Pay: $350-$400 Per Day$1500 Quarterly Bonuses, Annual Raises, Day and Night Shifts Available, Paid Vacation, Health Insurance, Uniforms and Boots, Spotless Trucks. How to apply for this...