Senior Data Engineer Autonomous Vehicle Data Solutions

Are you passionate about building robust, secure, and adaptable data environments for the future of autonomous vehicles? We are looking for a Senior Data Engineer to join our innovative team. In this role, you will establish and maintain high-quality data environments, ensuring consistent and secure access for all authorized users. You will work closely with multiple project and product delivery teams, providing data engineering leadership and end-to-end support throughout the data lifecycle. As a key interface with stakeholders, you will deliver scalable data engineering solutions that drive our mission forward. The role In this role, you will be responsible for building and maintaining the backbone of our data infrastructure. Your primary focus will be on data pipeline development, where you will design, develop, and maintain robust pipelines for uploading, ingesting, extracting, transforming, and loading (ETL/ELT) data from a variety of sources into our data lake. This includes implementing advanced ingestion techniques and managing multiple file formats, with a particular emphasis on time series data such as Mcap and MDF4. You will also take ownership of database and data lakehouse management, architecting and maintaining both relational and non-relational databases to ensure scalable, high-performance data storage and retrieval. A key part of your work will involve designing and optimizing our Data Lakehouse solution to enable unified analytics, efficient governance, and seamless integration of structured and unstructured data. Ensuring robust security, access controls, and compliance across all data platforms will be a critical responsibility. Maintaining data quality is essential, and you will participate in rigorous testing and monitoring to guarantee accuracy and integrity throughout all pipelines. Additionally, you will create and maintain comprehensive documentation for data pipelines and architecture, actively updating team resources in Confluence and contributing to roadmap discussions. Knowledge sharing and supporting ongoing maintenance within the team will be part of your day-to-day activities. Collaboration is key in this role. You will work closely with cross-functional teams to troubleshoot and resolve data-related issues, review colleagues’ code and deliveries, and uphold clean coding practices and high standards. Your problem-solving skills and ability to foster teamwork will ensure the success of our data initiatives. Key Technical Competencies: - 5+ years of hands-on Data Engineering experience - Deep knowledge of Data Engineering methodologies and tools - Experience with ETL/ELT, data mesh architecture, and containerization (Docker) - Proficiency in Airflow, Foxglove, and advanced data ingestion - Strong skills in Python, PostgreSQL, and working with measurement data (MDF4, mcap) - Experience with Databricks, hybrid storage solutions, and Data Lake architectures - API development using FastAPI, Apache APISIX - Data modeling expertise: schema optimization, schema evolution, and enabling data visualization (API, Power BI) - Big data systems: PySpark, Spark, Hadoop (or equivalent) - Image data engineering and/or LiDAR data processing - Data ingestion for vehicle data and experience in autonomous vehicle development Qualifications: - Master’s degree in Computer Science, Data Science, or a related field (or Bachelor’s with relevant experience) Who you are You are a collaborative and detail-oriented professional who thrives in team environments and knows how to lead when needed. Sharing knowledge and fostering collaboration come naturally to you, and you have a genuine willingness to teach, mentor, and support others. Your meticulous attention to detail ensures high-quality work, and your strong documentation skills help keep processes clear and accessible for everyone. You bring experience working with on-premises, cloud, and hybrid solutions, giving you the flexibility to navigate diverse technical landscapes. A solid understanding of vehicle data and best practices in data security rounds out your profile, making you a trusted partner in safeguarding sensitive information while driving innovation. Required Software & Platform Knowledge You’re comfortable working in Linux environments and have solid experience with Python. You understand data processing platforms and tools such as Spark or Hadoop (or similar), and you’re familiar with production-grade software development methodologies like SPICE or equivalent. You know how to handle both batch and streaming data effectively, and you have experience with SQL and NoSQL tools. Exposure to platforms and tools such as Foxglove, Jama, Baseliner, and Vast is a plus. Additionally, you have worked with cloud technologies, particularly Azure and ADLS, and you’re familiar with Databricks for advanced data analytics. Why Join Us Joining our team means being part of something truly exciting—developing cutting-edge data solutions that power the future of autonomous vehicles. You’ll collaborate with talented engineers and data scientists, working in an environment that values innovation and teamwork. Here, you’ll have the opportunity to lead projects, bring new ideas to life, and make a real impact in a supportive and forward-thinking setting. We believe in continuous growth, so you’ll find plenty of opportunities to expand your skills, share knowledge, and learn from others. If you’re passionate about shaping the future of mobility and want to work where your contributions truly matter, this is the place for you. Ready to Take the Lead? Join us in transforming the way the world moves. Apply today and be a part of the future.