Distributed Systems Engineer

Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach is to combine frontier-scale pre-training, domain-specific RL, ultra-long context, and test-time compute to achieve this goal.

About the role: 

As a Distributed Systems Engineer, you write efficient and robust pipelines for giant, multimodal datasets. You will develop and optimize web scraping techniques to harvest and maintain data at internet-scale.

What you might work on:

  • Design & independently implement multimodal (video, audio, text etc.) web crawlers for scraping and indexing petabytes of data
  • Develop scalable fuzzy deduplication techniques for different modalities
  • Develop heuristic and model-based techniques for parsing and filtering crawled data
  • Identify new data sources for inclusion in pre/post-training datasets

What we’re looking for:

  • Strong proficiency in distributed computing and parallel processing techniques
  • Obsession with details, reliability, and good testing to ensure data quality and integrity
  • Experience with designing and maintaining high-performance, scalable data architectures
  • Ability to design, develop and operate an LLM data pipeline from web scraping to data loading

Magic strives to be the place where high-potential individuals can do their best work. We value quick learning and grit just as much as skill and experience.

Our culture:

  • Integrity. Words and actions should be aligned
  • Hands-on. At Magic, everyone is building 
  • Teamwork. We move as one team, not N individuals
  • Focus. Safely deploy AGI. Everything else is noise
  • Quality. Magic should feel like magic

Compensation, benefits and perks (US):

  • Annual salary range: $150K - $1M
  • Equity is a significant part of total compensation, in addition to salary
  • 401(k) plan with 6% salary matching
  • Generous health, dental and vision insurance for you and your dependants
  • Unlimited paid time off
  • Option to work in-person in SF or remotely
  • Visa sponsorship and relocation stipend to bring you to SF
  • A small, fast-paced, highly focused team
Apply for this job
logo Magic Distributed Systems Engineer FullTime 🌎 Remote 📍 San Francisco Apply Now
Your subscription could not be saved. Please try again.
Your subscription has been successful.

Newsletter

Subscribe and stay updated.

Your subscription could not be saved. Please try again.
Your subscription has been successful.

Join our newsletter