Research Engineer Large Language Model (LLM) Pretraining

Key Responsibilities

1. Data Engineering for Pretraining

• Build and maintain scalable pipelines for text collection, cleaning, deduplication, filtering, and quality scoring.

• Process large-scale Vietnamese and multilingual datasets.

• Implement tokenization workflows, corpus sharding, mixture sampling, and dataset balancing.

• Develop automated dataset validation and quality assurance tools.

2. Model Training & Optimization

• Support distributed training of LLMs using DeepSpeed, Megatron-LM, FSDP, or similar.

• Optimize throughput, memory efficiency, and multi-node GPU performance.

• Run full-scale LLM experiments and troubleshoot training issues.

• Conduct model fine-tuning, instruction tuning, and alignment if needed.

3. Infrastructure & Engineering

• Work with multi-GPU/multi-node clusters using Slurm, Docker/Singularity.

• Maintain experiment tracking pipelines.

• Develop reusable tools for logging, checkpointing, and evaluations.

4. Evaluation & Benchmarking

• Prepare and maintain Vietnamese and multilingual benchmark suites.

• Implement automated evaluation pipelines.

• Analyze results to guide improvements.

Minimum Requirements

• Bachelor’s/Master’s/PhD’s degree in CS/AI/ML or related fields.

• Strong Python programming and PyTorch experience.

• Understanding of transformer architectures and tokenization.

• Experience with GPU clusters, Linux, Bash.

• Familiarity with distributed training frameworks.

Preferred Qualifications

• Experience with large-scale datasets.

• Knowledge of Vietnamese NLP.

• Experience with MoE, long-context models, deduplication.

• Open-source contributions.

• Experience with quantization, distillation, compression.