Sr Big Data Engineer Airflow and Oozie

hace 6 días


Desde casa, Colombia Rackspace A tiempo completo

**About the Role**:
**What you will be doing**:

- Develop scalable and robust code for batch processing systems. This includes working with technologies like Hadoop, Oozie, Pig, Hive, Map Reduce, Spark (Java), Python, Hbase
- Develop, Manage and optimize data workflows using Oozie and Airflow within the Apache Hadoop ecosystem
- Leverage GCP for scalable big data processing and storage solutions
- Implementing automation/DevOps best practices for CI/CD, IaC, etc.

**Requirements**:

- Experience with GCP managed services and understanding of cloud-based batch processing systems are critical.
- Proficiency in Oozie, Airflow, Map Reduce, Java
- Strong programming skills with Java (specifically Spark), Python, Pig, and SQL
- Expertise in public cloud services, particularly in GCP.
- Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, Map Reduce
- Familiarity with BigTable and Redis
- Experienced in Infrastructure and Applied DevOps principles in daily work. Utilize tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform to automate and improve development and release processes.
- Ability to tackle complex challenges and devise effective solutions. Use critical thinking to approach problems from various angles and propose innovative solutions.
- Worked effectively in a remote setting, maintaining strong written and verbal communication skills. Collaborate with team members and stakeholders, ensuring clear understanding of technical requirements and project goals.
- Proven experience in engineering batch processing systems at scale.
- Hands-on experience in public cloud platforms, particularly GCP. Additional experience with other cloud technologies is advantageous.

**Must Have**:

- Google Associate Cloud Engineer Certification or other Google Cloud Professional level certification
- 10+ years of experience in customer-facing software/technology or consulting
- 5+ years of experience with “on-premises to cloud” migrations or IT transformations
- 5+ years of experience building, and operating solutions built on GCP (ideally) or AWS/Azure
- Technical degree: Computer Science, software engineering or related

**About Rackspace Technology**

**More on Rackspace Technology


  • Senior Data Engineer

    hace 2 semanas


    Desde casa, Colombia Fusemachines A tiempo completo

    **About Fusemachines** Fusemachines is an artificial intelligence talent and education solutions company dedicated to democratizing AI. Headquartered in New York,US with operations in Dominican Republic, Canada, and Nepal, it is a 300+ company. Fusemachines' AI educational program has made world-class AI education available, accessible and affordable to...

  • Senior Data Engineer

    hace 2 semanas


    Desde casa, Colombia Fusemachines A tiempo completo

    About FusemachinesFusemachines is an artificial intelligence talent and education solutions company dedicated to democratizing AI. Headquartered in New York,US with operations in Dominican Republic, Canada, and Nepal, it is a 300+ company. Fusemachines' AI educational program has made world-class AI education available, accessible and affordable to students...

  • Data Engineer

    hace 2 semanas


    Desde casa, Colombia Intellias A tiempo completo

    Project Overview: Our customer is a US based international company on the Fortune 500 list. They provide energy, logistics, and technology solutions to aviation, marine, and land customers and suppliers around the world. Their solutions include oil and energy procurement, distribution and storage, operations support, financial, consulting, and technology...

  • Data Engineer

    hace 2 semanas


    Desde casa, Colombia Intellias A tiempo completo

    Project Overview:Our customer is a US based international company on the Fortune 500 list. They provide energy, logistics, and technology solutions to aviation, marine, and land customers and suppliers around the world. Their solutions include oil and energy procurement, distribution and storage, operations support, financial, consulting, and technology...

  • Data Engineer

    hace 1 semana


    Desde casa, Colombia Merkle A tiempo completo

    **Information Technology**: **Columbia, MD** **About the job**: We are looking for a talented and enthusiastic Data Engineer with excellent communication skills and big data processing knowledge to join our engineering team. You will build and maintain world class products to enable Merkle's people-based marketing vision using cutting-edge cloud-based...

  • Sr Data Engineer

    hace 4 semanas


    Desde casa, Colombia DEUNA A tiempo completo

    About us: **DEUNA,** a rapidly growing SaaS startup, is revolutionizing the e-commerce landscape in LATAM by solving the most pressing challenges faced by businesses today:customer conversion, payment acceptance, and fraud prevention with a single integration. With a recently announced $30M USD Series A funding, we operate in 5 LATAM countries and are on...

  • Senior Data Engineer

    hace 3 semanas


    Desde casa, Colombia Growth Acceleration Partners A tiempo completo

    **Description**: We are seeking a Senior Data Engineer with a proven record of accomplishment in the design, development, maintenance, and operation of scalable, high-performance Data solutions primarily using PostgreSQL and SQL Server secondarily. This is a hands-on position that includes developing data technologies with Technology Team Leadership and...


  • Desde casa, Colombia Soft Dev Team A tiempo completo

    As an AI Data Engineer at Agile Dream Team, you will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. You will support our data scientists on data initiatives and ensure optimal data delivery architecture is consistent throughout ongoing...

  • Senior Data Engineer

    hace 3 semanas


    Desde casa, Colombia Gorilla Logic A tiempo completo

    **Senior Data Engineer** Gorilla Logic is looking for a Sr. Data Engineer, who will assist in organizing our client's system data into dashboards and on demand reports. This is a unique and highly technical role, requiring strong database and reporting experience in delivering leading-edge solutions. Our environment will require you to work effectively with...

  • Data Architect

    hace 4 semanas


    Desde casa, Colombia EPAM Systems A tiempo completo

    EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multicultural teams, contribute to a myriad of innovative projects that deliver the most...

  • Senior Big Data Engineer

    hace 3 semanas


    Desde casa, Colombia EPAM Systems A tiempo completo

    RESPONSIBILITIES - Work with product and engineering teams to understand requirement and help drive decisions - Maintain collaboration with architects, technical leads and key individuals within other functional groups - Perform analysis of business problems and technical environments and use this in implementation of quality technical solutions -...

  • Senior Data Engineer

    hace 2 semanas


    Desde casa, Colombia Gorilla Logic A tiempo completo

    Senior Data Engineer (Java/GCP)Responsibilities Design, build and maintain scalable and efficient data architectures, including databases, data warehouses, data lakes, data marts, and other storage solutions. Design, build, and maintain robust data pipelines for the extraction, transformation, and loading (ETL) of data from various sources to storage and...

  • Data Engineer

    hace 4 semanas


    Desde casa, Colombia Perficient, Inc A tiempo completo

    **Location**: Colombia, México, Uruguay, Chile y Argentina "="" ,="" sans-serif;font-size:="" 16.0px;font-style:="" normal;font-weight:="" 400;letter-spacing:="" normal;text-indent:="" 0.0px;text-transform:="" none;word-spacing:="" 0.0px;"="">As a leader in data-driven insights, we understand the transformative power of Snowflake expertise. If you're a...

  • Senior Data Engineer

    hace 4 semanas


    Desde casa, Colombia Provectus A tiempo completo

    We are seeking a talented and experienced** Senior Data Engineer** to join our team at Provectus. You will encounter numerous technical challenges and have the opportunity to contribute to exciting open-source projects (e.g., ODD, UI for Kafka), build internal solutions, and engage in R&D activities, providing an excellent environment for professional...

  • Data Engineer

    hace 3 semanas


    Desde casa, Colombia Formula.Monks A tiempo completo

    We are lookin for a strategy-obsessed, problem-solving, and detail-loving** Data Engineer** to join the team! With solid experience pulling data from various databases into database objects. The specific tittle is **Senior Marketing Infrastructure & Automation Engineer.** This role will act as the senior technical lead on the team supporting the continuous...

  • Data Engineer

    hace 4 semanas


    Desde casa, Colombia TEAM International Services A tiempo completo

    Our client is a global professional services firm committed to delivering quality in assurance, advisory, and tax services. Financial Crime team works at the forefront of risk management, leveraging advanced technologies and analytics to combat financial crimes and safeguard our clients' interests. Data Pipeline Development: - Design and implement robust,...

  • Data Engineer

    hace 2 semanas


    Desde casa, Colombia Perficient, Inc A tiempo completo

    Location: Colombia, México, Uruguay, Chile y Argentina"="" ,="" sans-serif;font-size:="" 16.0px;font-style:="" normal;font-weight:="" 400;letter-spacing:="" normal;text-indent:="" 0.0px;text-transform:="" none;word-spacing:="" 0.0px;"="">As a leader in data-driven insights, we understand the transformative power of Snowflake expertise. If you're a Data...

  • Data Engineer

    hace 2 semanas


    Desde casa, Colombia TEAM International Services A tiempo completo

    Our client is a global professional services firm committed to delivering quality in assurance, advisory, and tax services. Financial Crime team works at the forefront of risk management, leveraging advanced technologies and analytics to combat financial crimes and safeguard our clients' interests.Data Pipeline Development: Design and implement robust,...


  • Desde casa, Colombia Convergenz A tiempo completo

    **Role**: Data Architect/Data Security Engineer Tech SETA **Duties**: - Provide technical guidance to external teams working on large-scale data projects - Ensure robust and scalable data solutions are implemented, with a strong focus on security and compliance - Analyze scientific and technical data with respect to novelty, rigor, and robustness -...

  • Senior Data Engineer

    hace 2 semanas


    Desde casa, Colombia Provectus A tiempo completo

    We are seeking a talented and experienced Senior Data Engineer to join our team at Provectus.You will encounter numerous technical challenges and have the opportunity to contribute to exciting open-source projects (e.g., ODD, UI for Kafka), build internal solutions, and engage in R&D activities, providing an excellent environment for professional...