Sr Big Data Engineer Airflow and Oozie

hace 5 meses


Desde casa, Colombia Rackspace A tiempo completo

**About the Role**:
**What you will be doing**:

- Develop scalable and robust code for batch processing systems. This includes working with technologies like Hadoop, Oozie, Pig, Hive, Map Reduce, Spark (Java), Python, Hbase
- Develop, Manage and optimize data workflows using Oozie and Airflow within the Apache Hadoop ecosystem
- Leverage GCP for scalable big data processing and storage solutions
- Implementing automation/DevOps best practices for CI/CD, IaC, etc.

**Requirements**:

- Experience with GCP managed services and understanding of cloud-based batch processing systems are critical.
- Proficiency in Oozie, Airflow, Map Reduce, Java
- Strong programming skills with Java (specifically Spark), Python, Pig, and SQL
- Expertise in public cloud services, particularly in GCP.
- Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, Map Reduce
- Familiarity with BigTable and Redis
- Experienced in Infrastructure and Applied DevOps principles in daily work. Utilize tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform to automate and improve development and release processes.
- Ability to tackle complex challenges and devise effective solutions. Use critical thinking to approach problems from various angles and propose innovative solutions.
- Worked effectively in a remote setting, maintaining strong written and verbal communication skills. Collaborate with team members and stakeholders, ensuring clear understanding of technical requirements and project goals.
- Proven experience in engineering batch processing systems at scale.
- Hands-on experience in public cloud platforms, particularly GCP. Additional experience with other cloud technologies is advantageous.

**Must Have**:

- Google Associate Cloud Engineer Certification or other Google Cloud Professional level certification
- 10+ years of experience in customer-facing software/technology or consulting
- 5+ years of experience with “on-premises to cloud” migrations or IT transformations
- 5+ years of experience building, and operating solutions built on GCP (ideally) or AWS/Azure
- Technical degree: Computer Science, software engineering or related

**About Rackspace Technology**

**More on Rackspace Technology


  • Sr Data Engineer

    hace 6 meses


    Desde casa, Colombia DEUNA A tiempo completo

    About us: **DEUNA,** a rapidly growing SaaS startup, is revolutionizing the e-commerce landscape in LATAM by solving the most pressing challenges faced by businesses today:customer conversion, payment acceptance, and fraud prevention with a single integration. With a recently announced $30M USD Series A funding, we operate in 5 LATAM countries and are on...

  • Senior Data Engineer

    hace 6 meses


    Desde casa, Colombia Growth Acceleration Partners A tiempo completo

    **Description**: We are seeking a Senior Data Engineer with a proven record of accomplishment in the design, development, maintenance, and operation of scalable, high-performance Data solutions primarily using PostgreSQL and SQL Server secondarily. This is a hands-on position that includes developing data technologies with Technology Team Leadership and...


  • Desde casa, Colombia EPAM Systems A tiempo completo

    RESPONSIBILITIES - Work with product and engineering teams to understand requirement and help drive decisions - Maintain collaboration with architects, technical leads and key individuals within other functional groups - Perform analysis of business problems and technical environments and use this in implementation of quality technical solutions -...

  • Senior Data Engineer

    hace 6 meses


    Desde casa, Colombia Gorilla Logic A tiempo completo

    **Senior Data Engineer** Gorilla Logic is looking for a Sr. Data Engineer, who will assist in organizing our client's system data into dashboards and on demand reports. This is a unique and highly technical role, requiring strong database and reporting experience in delivering leading-edge solutions. Our environment will require you to work effectively with...

  • Data Engineer

    hace 6 meses


    Desde casa, Colombia Perficient, Inc A tiempo completo

    **Location**: Colombia, México, Uruguay, Chile y Argentina "="" ,="" sans-serif;font-size:="" 16.0px;font-style:="" normal;font-weight:="" 400;letter-spacing:="" normal;text-indent:="" 0.0px;text-transform:="" none;word-spacing:="" 0.0px;"="">As a leader in data-driven insights, we understand the transformative power of Snowflake expertise. If you're a...

  • Senior Data Engineer

    hace 6 meses


    Desde casa, Colombia Provectus A tiempo completo

    We are seeking a talented and experienced** Senior Data Engineer** to join our team at Provectus. You will encounter numerous technical challenges and have the opportunity to contribute to exciting open-source projects (e.g., ODD, UI for Kafka), build internal solutions, and engage in R&D activities, providing an excellent environment for professional...

  • Data Engineer

    hace 6 meses


    Desde casa, Colombia Formula.Monks A tiempo completo

    We are lookin for a strategy-obsessed, problem-solving, and detail-loving** Data Engineer** to join the team! With solid experience pulling data from various databases into database objects. The specific tittle is **Senior Marketing Infrastructure & Automation Engineer.** This role will act as the senior technical lead on the team supporting the continuous...

  • Data Engineer

    hace 6 meses


    Desde casa, Colombia TEAM International Services A tiempo completo

    Our client is a global professional services firm committed to delivering quality in assurance, advisory, and tax services. Financial Crime team works at the forefront of risk management, leveraging advanced technologies and analytics to combat financial crimes and safeguard our clients' interests. Data Pipeline Development: - Design and implement robust,...

  • Data Engineer Semi Sr

    hace 6 meses


    Desde casa, Colombia STEFANINI LATAM A tiempo completo

    ¡Sé parte de Stefanini! En Stefanini somos más de 30.000 genios, conectados desde 41 países, haciendo lo que les apasiona y co-creando un futuro mejor. ¡Seguro no te quieres quedar fuera! ¡Data Engineer Semi Sr.! ¿Por qué te elegiremos? ¡Porque los desafíos que asumirás reflejan tus ambiciones! **Responsabilidades y atribuciones** -...


  • Desde casa, Colombia Rackspace A tiempo completo

    **About the Role**: **What you will be doing**: - Implementing automation/DevOps best practices for CI/CD, IaC, Containerization, etc to Build a reusable infra structure for stream and batch processing systems at scale. - Create automation, whether that is building DevOps pipelines, scripting or creating Infrastructure as Code in Terraform - Participating...


  • Desde casa, Colombia Convergenz A tiempo completo

    **Role**: Data Architect/Data Security Engineer Tech SETA **Duties**: - Provide technical guidance to external teams working on large-scale data projects - Ensure robust and scalable data solutions are implemented, with a strong focus on security and compliance - Analyze scientific and technical data with respect to novelty, rigor, and robustness -...

  • Sr Big Data Engineer

    hace 5 meses


    Desde casa, Colombia Rackspace A tiempo completo

    **About the Role**: **What you will be doing** - Build a reusable, and reliable code for stream and batch processing systems at scale. This includes working with technologies like Pub/Sub, Kafka, Kinesis, DataFlow, Flink, Hadoop, Pig, Hive, and Spark. Implementing automation/DevOps best practices for CI/CD, IaC, Containerization, etc. **Requirements**: -...

  • Senior Data Engineer

    hace 6 meses


    Desde casa, Colombia Gorilla Logic A tiempo completo

    **Senior Data Engineer (Java/GCP)** **Responsibilities** - Design, build and maintain scalable and efficient data architectures, including databases, data warehouses, data lakes, data marts, and other storage solutions. - Design, build, and maintain robust data pipelines for the extraction, transformation, and loading (ETL) of data from various sources to...

  • Senior Data Engineer

    hace 6 meses


    Desde casa, Colombia Gorilla Logic A tiempo completo

    **Senior Data Engineer (Java/GCP)** **Responsibilities** - Design, build and maintain scalable and efficient data architectures, including databases, data warehouses, data lakes, data marts, and other storage solutions. - Design, build, and maintain robust data pipelines for the extraction, transformation, and loading (ETL) of data from various sources to...

  • Senior Data Engineer

    hace 6 meses


    Desde casa, Colombia Gorilla Logic A tiempo completo

    **Senior Data Engineer** **Responsibilities** - Design, build and maintain scalable and efficient data architectures, including databases, data warehouses, data lakes, data marts, and other storage solutions. - Design, build, and maintain robust data pipelines for the extraction, transformation, and loading (ETL) of data from various sources to storage and...


  • Desde casa, Colombia Perficient, Inc A tiempo completo

    **Location**: Colombia, México, Uruguay, Chile y Argentina Are you ready to jump on board and join our crew? This is a no-brainer if you are curious enough to push yourself to the next level while working with top talented team members. As a**Senior Informática Data Engineer**, you will be responsible for designing, developing, and implementing data...

  • Data Engineer

    hace 6 meses


    Desde casa, Colombia Dev.Pro A tiempo completo

    Are you in Colombia, Argentina, Brazil, Chile, or Costa Rica? Join us as we actively recruit in these locations, offering a comfortable remote environment. Submit your CV in English, and we’ll get back to you! We invite a skilled Data Engineer to join a US/UAE-based Fintech project — a digital asset management platform integrated with both fiat and...

  • Senior Data Engineer

    hace 6 meses


    Desde casa, Colombia Provectus A tiempo completo

    **Responsibilities**: - Collect and manage large volumes of varied data sets; - Work directly with Data Scientists and ML Engineers to create robust and resilient data pipelines that feed Data Products; - Define data models that integrate disparate data across the organization; - Design, implement, and maintain ETL pipelines; - Develop and continuously test...

  • Data Engineer

    hace 6 meses


    Desde casa, Colombia Athena A tiempo completo

    Our clients are ambitious, high-impact CEOs, founders, and execs. They have started 100+ companies, raised billions in venture capital, taken companies public, worked in the White House, invested in Facebook / Uber / Airbnb, run professional sports teams, are Governors of US states, and compete in Ironman races and the World Series of Poker. They have...


  • Desde casa, Colombia EPAM Systems A tiempo completo

    RESPONSIBILITIES - Design, deploy, and manage data infrastructure in the cloud, primarily using AWS, Azure, or GCP - Develop and implement automation scripts and workflows for infrastructure provisioning, deployment, and monitoring using tools like Terraform or similar Infrastructure as Code (IaC) tools - Ensure efficient data pipelines and processes,...