Data Integration Engineer

hace 1 semana

Colombia Viaduct A tiempo completo

At Viaduct, we use patented AI to discover hidden patterns in complex time series data – so manufacturers and operators of connected equipment can deliver transformative business results from their data, fast.  On our platform we deliver solutions across the equipment lifecycle, from manufacturing productivity, manufacturing quality, service operations and fleet management. 

Who You Are

You are a thoughtful engineer. You understand the complexities of distributed systems and how to triage and solve issues that arise with them. Scalability is top of mind when designing any system or writing code. You believe building a better ETL system requires close collaboration with the machine learning and data science teams.

About the Role

As an data engineer at Viaduct, an analytics and ML platform company, your work is critical to our success. You are responsible for ensuring data quality and reliability in every part of our ETL pipeline, from ingestion to client integrations. 

Responsibilities
  • Creating and supporting batch, incremental, and real-time  ETL pipelines
  • Standardizing ingestion, validation, and cleaning processes across clients
  • Automating data validation to increase data quality
  • Managing and evolving schemas in all parts of our pipeline
  • Monitoring, tuning, and optimizing Spark jobs
  • Becoming a domain-expert in connected vehicle data
About You
  • 4+ years as a data engineer
  • Experience as a tech lead or mentor
  • Proficiency in Python/Scala/Java/C++ and SQL
  • 3+ years of experience with Spark or equivalent technologies
  • 2+ years of experience with a workflow scheduler (Airflow, Prefect, Argo, etc) 
  • 2+ years of experience with distributed file-systems (HDFS, S3, etc)
  • Familiar with the tools in open-source data ecosystem (Apache, CNCF, etc)
  • Experience with incremental or real-time processing (Delta Lake, Apache Hudi, Kafka Stream, Spark Streaming, etc) 
Security and Privacy Responsibilities
  • Follow our policy and procedure documents related to security and privacy
  • Follow the security and privacy guidelines in the Employee Handbook
  • Participate in new hire and annual training for security and privacy
  • Treat data security and privacy as one of your primary job responsibilities
  • Report Security Incidents you discover as bugs
  • Get approval from the Security Team before adding new 3rd party software to our codebase
  • Explicitly consider security implications when doing PR reviews
Bonus
  • Experience with Kubernetes
  • Experience working with ML teams
  • Contributor to open source projects
  • Experience in the Automotive industry or a love of cars
  • Prior work in small, agile teams

  • Actimize Integration Engineer

    hace 6 días

    Colombia Sutherland Global A tiempo completo

    A leading global provider of business services is seeking a Software Integration Engineer in Colombia. The ideal candidate will manage Actimize-related operations and resolve complex technical issues while demonstrating expertise in Java/J2EE and database management. Candidates should have strong problem-solving skills and effective communication abilities...

  • Integration Engineer

    hace 7 días

    Colombia Rush Street Interactive A tiempo completo

    If you are passionate about iGaming and want to create amazing user experience for our players, join one of our dynamic and enthusiastic teams as an Integration Engineer. We are on a mission to design sports betting and online casino experience for millions of players in Americas and through the most innovative solutions our aim is to give sports and casino...

  • Sr. Data Engineer

    hace 7 días

    Colombia Kualty A tiempo completo

    **What were looking for** Looking for Sr. Data Engineer With +5 years of experience. **Requirements**: - Design, develop, and maintain data storage solutions and infrastructure. - Build and optimize ETL (Extract, Transform, Load) pipelines to ensure efficient data processing. - Handle data quality and data governance aspects, including data cleaning and...

  • Data Engineer

    hace 8 horas

    Colombia Neoris A tiempo completo

    NEORIS is a Digital accelerator that helps companies enter the future, having 20 years of experience as Digital Partners of some of the largest companies in the world. We have more than 4,000 professionals in 11 countries, with our multicultural startup culture where we cultivate innovation, continuous learning to create high-value solutions for our...

  • Data Engineer

    hace 2 días

    Capgemini Colombia Capgemini A tiempo completo

    Job Description Data EngineerChoosing Capgemini means choosing a company where you will be empowered to shape your career in the way you'd like, where you'll be supported and inspired by a collaborative community of colleagues around the world, and where you'll be able to reimagine what's possible. Join us and help the world's leading organizations...

  • Data Engineer

    hace 2 semanas

    Colombia Dev-Pro A tiempo completo

    **Description**: Are you in Colombia, Argentina, Brazil, Chile, or Costa Rica? Join us as we actively recruit in these locations, offering a comfortable remote environment. Submit your CV in English, and we’ll get back to you! We invite a skilled Data Engineer to join a US/UAE-based Fintech project — a digital asset management platform integrated with...

  • Epicor ERP Integration Specialist

    hace 4 días

    Colombia Eddy Pump A tiempo completo

    EDDY Pump Corporation is a leading manufacturer of heavy-duty pumps and dredge equipment, serving various industries worldwide. With cutting-edge technology and a commitment to innovation, we pride ourselves on delivering high-quality products that exceed customer expectations. **Responsibilities**: - Design, develop, and implement data integration...

  • Senior Data Engineer

    hace 2 semanas

    Colombia CI&T A tiempo completo

    We are tech transformation specialists, uniting human expertise with AI to create scalable tech solutions. With over 8,000 CI&Ters around the world, we've built partnerships with more than 1,000 clients during our 30 years of history. Artificial Intelligence is our reality.  Hey everyone This is Thai Viapiana from CI&T's Talent Attraction Team :) We are...

  • QA Automation Engineer

    hace 6 días

    Colombia Hire Horatio CX A tiempo completo

    A tech company in Colombia is seeking a Software Quality Assurance Engineer focused on automation and quality. This role requires 5+ years of experience in QA Automation, advanced skills in Python, and expertise in frameworks like Playwright. You will lead automation initiatives, ensuring data integrity and seamless integration in complex ecosystems using...

  • Sr Data Engineer

    hace 1 semana

    Colombia Xmartlabs Uruguay A tiempo completo

    We are looking for an experienced **Data Engineer** to join our team. In this role, you will be responsible for streamlining the intake of raw data, driving core data engineering and business intelligence frameworks, and ensuring the scalability and reliability of our data pipelines. You will collaborate with cross-functional teams to analyze business needs,...