Sr Big Data Engineer Airflow and Oozie
hace 2 semanas
**About the Role**:
**What you will be doing**:
- Develop scalable and robust code for batch processing systems. This includes working with technologies like Hadoop, Oozie, Pig, Hive, Map Reduce, Spark (Java), Python, Hbase
- Develop, Manage and optimize data workflows using Oozie and Airflow within the Apache Hadoop ecosystem
- Leverage GCP for scalable big data processing and storage solutions
- Implementing automation/DevOps best practices for CI/CD, IaC, etc.
**Requirements**:
- Experience with GCP managed services and understanding of cloud-based batch processing systems are critical.
- Proficiency in Oozie, Airflow, Map Reduce, Java
- Strong programming skills with Java (specifically Spark), Python, Pig, and SQL
- Expertise in public cloud services, particularly in GCP.
- Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, Map Reduce
- Familiarity with BigTable and Redis
- Experienced in Infrastructure and Applied DevOps principles in daily work. Utilize tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform to automate and improve development and release processes.
- Ability to tackle complex challenges and devise effective solutions. Use critical thinking to approach problems from various angles and propose innovative solutions.
- Worked effectively in a remote setting, maintaining strong written and verbal communication skills. Collaborate with team members and stakeholders, ensuring clear understanding of technical requirements and project goals.
- Proven experience in engineering batch processing systems at scale.
- Hands-on experience in public cloud platforms, particularly GCP. Additional experience with other cloud technologies is advantageous.
**Must Have**:
- Google Associate Cloud Engineer Certification or other Google Cloud Professional level certification
- 10+ years of experience in customer-facing software/technology or consulting
- 5+ years of experience with “on-premises to cloud” migrations or IT transformations
- 5+ years of experience building, and operating solutions built on GCP (ideally) or AWS/Azure
- Technical degree: Computer Science, software engineering or related
**About Rackspace Technology**
**More on Rackspace Technology
-
Senior Data Engineer
hace 4 semanas
Desde casa, Colombia Growth Acceleration Partners A tiempo completo**Description**: We are seeking a Senior Data Engineer with a proven record of accomplishment in the design, development, maintenance, and operation of scalable, high-performance Data solutions primarily using PostgreSQL and SQL Server secondarily. This is a hands-on position that includes developing data technologies with Technology Team Leadership and...
-
Data Engineering Specialist for Ai Applications
hace 3 días
Desde casa, Colombia Soft Dev Team A tiempo completoAs an AI Data Engineer at Agile Dream Team, you will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. You will support our data scientists on data initiatives and ensure optimal data delivery architecture is consistent throughout ongoing...
-
Data Engineer
hace 4 días
Desde casa, Colombia TEAM International Services A tiempo completoOur client is a global professional services firm committed to delivering quality in assurance, advisory, and tax services. Financial Crime team works at the forefront of risk management, leveraging advanced technologies and analytics to combat financial crimes and safeguard our clients' interests. Data Pipeline Development: - Design and implement robust,...
-
Data Engineer
hace 4 semanas
Desde casa, Colombia Formula.Monks A tiempo completoWe are lookin for a strategy-obsessed, problem-solving, and detail-loving** Data Engineer** to join the team! With solid experience pulling data from various databases into database objects. The specific tittle is **Senior Marketing Infrastructure & Automation Engineer.** This role will act as the senior technical lead on the team supporting the continuous...
-
Senior Data Engineer
hace 3 días
Desde casa, Colombia Gorilla Logic A tiempo completo**Senior Data Engineer** **Responsibilities** - Design, build and maintain scalable and efficient data architectures, including databases, data warehouses, data lakes, data marts, and other storage solutions. - Design, build, and maintain robust data pipelines for the extraction, transformation, and loading (ETL) of data from various sources to storage and...
-
Senior Data Delivery Engineer
hace 3 días
Desde casa, Colombia Merkle A tiempo completo**Engineering**: **Remote | Columbia, MD** **About the job**: As a Manager of Data Engineer, you will lead a data engineering team reporting to the Director, Data Engineering: developing new or enhancing existing data products as we build a meta data driven, big data solution, processing and transforming data to produce high quality data assets for our...
-
Data Engineer
hace 1 semana
Desde casa, Colombia Athena A tiempo completoOur clients are ambitious, high-impact CEOs, founders, and execs. They have started 100+ companies, raised billions in venture capital, taken companies public, worked in the White House, invested in Facebook / Uber / Airbnb, run professional sports teams, are Governors of US states, and compete in Ironman races and the World Series of Poker. They have...
-
Senior Data Engineer
hace 4 semanas
Desde casa, Colombia Growth Acceleration Partners A tiempo completo**Description**: **Education** - Bachelor’s Degree in Computer Science or related field. **Experience** - Minimum 3 years of experience as a Data Engineer or related field. **Required Skills**: - Data engineering and data modeling architecture knowledge of analytics platforms; ETL/ELT, denormalization and schema knowledge, tuning of data ingest...
-
Senior Data Engineer
hace 3 días
Desde casa, Colombia Growth Acceleration Partners A tiempo completo**Description**: **Education** - Bachelor’s Degree in Computer Science or related field. **Experience** - Minimum 3 years of experience as a Data Engineer or related field. **Required Skills**: - Data engineering and data modeling architecture knowledge of analytics platforms; ETL/ELT, denormalization and schema knowledge, tuning of data ingest...
-
Senior Data Engineer
hace 3 días
Desde casa, Colombia Gorilla Logic A tiempo completo**Senior Data Engineer** **Responsibilities** - Ability to understand ETL requirements and implement them - Work with DevOps and QA to build a quality focused CI/CD pipeline - Responsible for code reviews, testing and delivery of high-quality maintainable code - Work in an Agile environment to define, estimate and work user stories - Highly self-motivated,...
-
Data Engineer
hace 4 semanas
Desde casa, Colombia Cafeto Software A tiempo completo**Description**: **Duties and Responsibilities** - ** **Design, develop and implement data management environments like databases or processing systems. - Ensuring architecture is planned in such a way that it meets all the business requirements. - Discover opportunities for data acquisition. - Taking care of the complete ETL (Extract, Transform & Load)...
-
Director, Data Engineering- Remote
hace 4 semanas
Desde casa, Colombia Merkle A tiempo completo**Information Technology**: **Columbia, MD** **About the job**: We are seeking a Director, Data Engineering who excels at creating, enhancing, maintaining, and managing meta data-driven, big data processing platforms. You will work with passionate Data engineers and QA engineers. You will collaborate with Data Management, Operations, and Delivery teams to...
-
Semi Senior/senior Dba/data Engineer
hace 7 días
Desde casa, Colombia Devsu A tiempo completoWe are looking for an experienced DBA/Data Engineer to work remotely from anywhere in Latin America. You will be working in an agile environment with an awesome team on the implementation of world-class software products. **Responsibilities** - Design, and develop data-products, with adjoining data-pipelines, engineered to high standards of performance,...
-
Senior Data Engineer
hace 4 semanas
Desde casa, Colombia Gorilla Logic A tiempo completoGorilla Logic is looking for a Senior Data Engineer using the AWS Cloud Platform. This is a unique and highly technical role responsible for data analysis within the enterprise. This role works with multiple data sources, data aggregation into data lakes, and data transformation into value. Our environment will require you to work effectively with your...
-
Senior Data Engineer
hace 4 semanas
Desde casa, Colombia AgileEngine A tiempo completo**What you will do** - Build data pipelines(ETL flows) to connect different sources of data together - Designing data structures for analytics - Identification of performance bottlenecks - Serve as a go-to Data Engineering consultant for other team members - Optimize SQL queries - Communication with Engineering & Product management - Alignment with existing...
-
Senior Data Analyst
hace 3 días
Desde casa, Colombia Growth Acceleration Partners A tiempo completo**Description**: We are looking for a Senior Data Analyst. You must have experience using a variety of data wrangling and analysis methods, with a variety of tools. To do so, problem-solving and strong analytical skills are a must. You have to be curious and enthusiastic about data and have the necessary communication skills to explain your...
-
Senior Data Engineer
hace 4 semanas
Desde casa, Colombia AgileEngine A tiempo completo**What you will do** - Build data pipelines using NiFi (ETL flows) to connect different sources of data together; - Build data reports and optimize SQL queries; - Produce high-quality code; - Identify performance bottlenecks; - Communicate with Engineering & Product management; - Align with existing development teams. **Must haves** - 3+ years of experience...
-
Alteryx Specialist
hace 11 horas
Desde casa, Colombia Data Meaning A tiempo completo**Alteryx Specialist** **Location**: Brazil **Position Type**:Full time **Position Summary** Data Meaning is a front-runner in Business Intelligence and Data Analytics consulting, renowned for our high-quality consulting services throughout the US and LATAM. Our expertise lies in delivering tailored solutions in Business Intelligence, Data Warehousing,...
-
Advanced Python Software Engineer
hace 3 días
Desde casa, Colombia Growth Acceleration Partners A tiempo completoThe position is for a Back-end engineer with a high level of experience either in Python / node.js and Web Scraping. This person must have REST API, CSS, and HTML experience since it will require heavy DOM manipulation. It also requires familiarity with techniques and tools for crawling, extracting, and processing data (like Selenium, Scrapy, Puppeteer, or...
-
Data Scientist and Ai Specialist
hace 3 días
Desde casa, Colombia Soft Dev Team A tiempo completo**Responsibilities**: - Use predictive modeling to increase and optimize customer experiences, revenue generation, ad targeting, and other business outcomes. - Coordinate with different functional teams to implement models and monitor outcomes. - Develop processes and tools to monitor and analyze model performance and data accuracy. **Requirements**: -...