Software Performance Optimization

My next tutorial on pretraining an LLM from scratch is now out. It starts with a step-by-step walkthrough of understanding, calculating, and optimizing the loss. After training, we update the text generation function with temperature scaling and top-k sampling. And finally, we also load openly available pretrained weights into our scratch-built model architecture. Along with this pretraining tutorial, I also have bonus material on speeding up the LLM training. These apply not just to LLMs but also to other transformer-based models like vision transformers: 1. Instead of saving the causal mask, this creates the causal mask on the fly to reduce memory usage (here it has minimal effect, but it can add up in long-context size models like Llama 3.2 with 131k-input-tokens support) 2. Use tensor cores (only works for Ampere GPUs like A100 and newer) 3. Use the fused CUDA kernels for `AdamW` by setting 4. Pre-allocate and re-use GPU memory via the pinned memory setting in the data loader 5. Switch from 32-bit float to 16-bit brain float (bfloat16) precision 6. Replace from-scratch implementations of attention mechanisms, layer normalizations, and activation functions with PyTorch counterparts that have optimized CUDA kernels 7. Use FlashAttention for more efficient memory read and write operations 8. Compile the model 9. Optimize the vocabulary size 10. After saving memory with the steps above, increase the batch size Video tutorial: https://lnkd.in/gDRycWea PyTorch speed-ups: https://lnkd.in/gChvGCJH

Coding agents are accelerating different types of software work to different degrees. When we architect teams, understanding these distinctions helps us to have realistic expectations. Listing functions from most accelerated to least, my order is: frontend development, backend, infrastructure, and research. Frontend development — say, building a web page to serve descriptions of products for an ecommerce site — is dramatically sped up because coding agents are fluent in popular frontend languages like TypeScript and JavaScript and frameworks like React and Angular. Additionally, by examining what they have built by operating a web browser, coding agents are now very good at closing the loop and iterating on their own implementations. Granted, LLMs today are still weak at visual design, but given a design (or if a polished design isn’t important), the implementation is fast! Backend development — say, building APIs to respond to queries requesting product data — is harder. It takes more work by human developers to steer modern models to think through corner cases that might lead to subtle bugs or security flaws. Further, a backend bug can lead to non-intuitive downstream effects like a corrupted database that occasionally returns incorrect results, which can be harder to debug than a typical frontend bug. Finally, although database migrations can be easier with coding agents, they’re still hard and need to be handled carefully to prevent data loss. While backend development is much faster with coding agents, they accelerate it less, and skilled developers still design and implement far better backends than inexperienced ones who use coding agents. Infrastructure. Agents are even less effective in tasks like scaling an ecommerce site to 10K active uses while maintaining 99.99% reliability. LLMs' knowledge is still relatively limited with respect to infrastructure and the complex tradeoffs good engineers must make, so I rarely trust them for critical infra decisions. Building good infrastructure often requires a period of testing and experimentation, and coding agents can help with that, but ultimately that’s a significant bottleneck where fast AI coding does not help much. Lastly, finding infrastructure bugs — say, a subtle network misconfiguration — can be incredibly difficult and requires deep engineering expertise. Thus, I’ve found that coding agents accelerate critical infrastructure even less than backend development. Research. Coding agents accelerate research work even less. Research involves thinking through new ideas, formulating hypotheses, running experiments, interpreting them to potentially modify the hypotheses, and iterating until we reach conclusions. Coding agents can speed up the pace at which we can write research code. (I also use coding agents to help me orchestrate and keep track of experiments.) [Truncated for length; full text: https://lnkd.in/gCnqy_4e ]

𝗘𝘅𝗽𝗹𝗮𝗶𝗻 𝗧𝗵𝗶𝘀: 𝗟𝗹𝗮𝗺𝗮 𝟯 𝗡𝗲𝗲𝗱𝘀 𝟮.𝟰𝗧𝗕. 𝗬𝗼𝘂𝗿 𝗚𝗣𝗨 𝗛𝗮𝘀 𝟴𝟬𝗚𝗕. 𝗜𝘁 𝗦𝘁𝗶𝗹𝗹 𝗧𝗿𝗮𝗶𝗻𝘀. Training Llama-3 405B needs ~2.4TB with BF16 + 8-bit Adam: • Weights: 810GB • Gradients: 810GB • Optimizer: 810GB (vs 3.24TB with standard Adam!) • Total: ~2.4TB (Illustrative budget—config-dependent; FP32 masters, ZeRO stage, and offload change totals) Your H100? 80GB. You'd need 30+ GPUs just to hold everything. 𝗧𝗵𝗿𝗲𝗲 𝗧𝗿𝗶𝗰𝗸𝘀 𝗧𝗵𝗮𝘁 𝗠𝗮𝗸𝗲 𝗜𝘁 𝗪𝗼𝗿𝗸 𝟭. 𝗗𝗮𝘁𝗮 𝗣𝗮𝗿𝗮𝗹𝗹𝗲𝗹: Split batch. Problem: Each GPU needs 2.4TB. Fix: ZeRO splits it across N GPUs. 𝟮. 𝗠𝗼𝗱𝗲𝗹 𝗣𝗮𝗿𝗮𝗹𝗹𝗲𝗹: Split layers. Problem: Sequential bottleneck. Fix: Pipeline batches. 𝟯. 𝗦𝗲𝗾𝘂𝗲𝗻𝗰𝗲 𝗣𝗮𝗿𝗮𝗹𝗹𝗲𝗹: Split tokens. This is the game changer. 8K tokens → 8 GPUs → 1K each. But attention needs every token to see all others. 𝗧𝗵𝗲 𝗠𝗮𝗴𝗶𝗰 𝗠𝗼𝗺𝗲𝗻𝘁: Instead of moving the 2.4TB model, GPUs only exchange attention keys/values (K,V). Each GPU: • Computes K,V for its 1K tokens (32MB) • Sends to others via all-to-all • Receives 7×32MB = 224MB total • Computes attention, deletes copies 𝟮𝟮𝟰𝗠𝗕 𝗺𝗼𝘃𝗲𝗱 𝗶𝗻𝘀𝘁𝗲𝗮𝗱 𝗼𝗳 𝟮.𝟰𝗧𝗕. That's 10,000x less. 𝗧𝗵𝗲 𝗥𝗲𝘀𝘂𝗹𝘁: Combine all three (ZeRO + tensor + pipeline + sequence parallel). Each GPU holds ~75GB instead of 2.4TB. This exact choreography powers ChatGPT, Claude, and every frontier model. Without it? 10K token limits. With it? Entire books in one context. Not magic. Just brilliant engineering making the impossible routine.

Exciting New Research: Injecting Domain-Specific Knowledge into Large Language Models I just came across a fascinating comprehensive survey on enhancing Large Language Models (LLMs) with domain-specific knowledge. While LLMs like GPT-4 have shown remarkable general capabilities, they often struggle with specialized domains such as healthcare, chemistry, and legal analysis that require deep expertise. The researchers (Song, Yan, Liu, and colleagues) have systematically categorized knowledge injection methods into four key paradigms: 1. Dynamic Knowledge Injection - This approach retrieves information from external knowledge bases in real-time during inference, combining it with the input for enhanced reasoning. It offers flexibility and easy updates without retraining, though it depends heavily on retrieval quality and can slow inference. 2. Static Knowledge Embedding - This method embeds domain knowledge directly into model parameters through fine-tuning. PMC-LLaMA, for instance, extends LLaMA 7B by pretraining on 4.9 million PubMed Central articles. While offering faster inference without retrieval steps, it requires costly updates when knowledge changes. 3. Modular Knowledge Adapters - These introduce small, trainable modules that plug into the base model while keeping original parameters frozen. This parameter-efficient approach preserves general capabilities while adding domain expertise, striking a balance between flexibility and computational efficiency. 4. Prompt Optimization - Rather than retrieving external knowledge, this technique focuses on crafting prompts that guide LLMs to leverage their internal knowledge more effectively. It requires no training but depends on careful prompt engineering. The survey also highlights impressive domain-specific applications across biomedicine, finance, materials science, and human-centered domains. For example, in biomedicine, domain-specific models like PMC-LLaMA-13B significantly outperform general models like LLaMA2-70B by over 10 points on the MedQA dataset, despite having far fewer parameters. Looking ahead, the researchers identify key challenges including maintaining knowledge consistency when integrating multiple sources and enabling cross-domain knowledge transfer between distinct fields with different terminologies and reasoning patterns. This research provides a valuable roadmap for developing more specialized AI systems that combine the broad capabilities of LLMs with the precision and depth required for expert domains. As we continue to advance AI systems, this balance between generality and specialization will be crucial.

🚀 Announcing our new research on data efficiency for language model pre-training, it reduces data required to train Llama-1B by 22x: 🌟 CLIMB (Clustering-based Iterative Data Mixture Bootstrapping) 🌟Fresh on arXiv: https://lnkd.in/gfTXqd-n 📌 Challenge: Constructing optimal data mixtures for pre-training large language models (LLMs) is hard, given the enormous unlabeled web corpora with no domain indication. 📌 Our Solution: CLIMB introduces a scalable, iterative approach leveraging semantic clustering to identify the most impactful subsets of data: ➤ Embeds massive web-scale datasets. ➤ Uses k-means clustering to semantically partition the data (we analyze data on the web page). ➤ Trains a set of 100 proxy models (300M) with different cluster weighting. ➤ Iteratively refines data mixtures guided by a lightweight performance predictor. First it fits a model to predict proxy model performance, then samples mixures that maximize predictor output. 📈 Results: ➤ A 1B parameter model trained on our optimized 400B-token dataset (ClimbMix) surpasses LLaMA-3.2-1B accuracy by +2.0%, this is 22x reduction! ➤ Significant domain-specific boosts—training on our social-science optimized subset yields a +5% gain over random sampling. ➤ We introduce ClimbLab, a rich 1.2T-token, semantically clustered corpus across 20 distinct domains, available publicly. Both with CC license! 🛠 Practical Impact: ➤ Reduces unnecessary computation by focusing training on the highest-quality data. We observed that noisy data, and domains such as “advertisements” confuse the model.  ➤ Enables domain-specific fine-tuning with fewer resources and higher accuracy. This is helpful if you know the domain. ➤ ClimbMix (400B tokens) is a balanced dataset for ablation studies that results in high benchmarks numbers. 🔗 Read our paper: https://lnkd.in/gfTXqd-n 📂 Datasets available on Hugging Face with free license: https://lnkd.in/garzY6VF 🌐 Project page: https://lnkd.in/gx4p_BtK (check cluster visualizations) 🗨️ Discussion: https://lnkd.in/gY2A3dn5 👏 Huge thanks to the talented NVIDIA Research team behind this work: SHIZHE DIAO, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu (Danny) Y., Mostofa Patwary Yingyan (Celine) Lin, Jan Kautz, and Pavlo Molchanov. NVIDIA AI , NVIDIA

🚀 Just Launched: A Deep Dive into Fine-Tuning LLaMA 3.2 (3B) for Q&A! 🦙💡 Are you ready to unlock the full potential of LLaMA 3.2? I’ve just released a step-by-step YouTube video where I walk you through the entire process of fine-tuning the LLaMA 3.2 (3B) instruct model using the mlabonne/finetome 100k Q&A dataset. https://lnkd.in/gFq2XefC This isn’t just another tutorial—it’s a comprehensive guide designed to solve 99% of your fine-tuning challenges. Whether you're a beginner or an experienced ML practitioner, this video has something for everyone. What’s Inside? ✅ Dataset Preparation: How to leverage the finetome 100k dataset for optimal results. ✅ Fine-Tuning Process: Every single step explained in detail—no shortcuts, no assumptions. ✅ Model Optimization: Tips and tricks to get the best performance out of your LLaMA model. ✅ Real-World Applications: How this fine-tuned model can be used to solve complex Q&A problems. This video is packed with actionable insights and practical examples to help you replicate the process and achieve state-of-the-art results. 💡 If you’ve ever struggled with fine-tuning large language models or wanted to dive deeper into the world of Q&A systems, this is the video for you. 🎥 Watch it now: https://lnkd.in/gFq2XefC Let me know your thoughts, questions, or feedback in the comments! I’d love to hear how this helps you in your ML journey. #MachineLearning #LLMs #FineTuning #LLaMA #AI #QnA #YouTube

🤩 What if you could use just 17k fine-tuning samples and change only 5% of the model to make a small LLM reason like the o1-preview model? DeepSeek-R1’s famous trick to make cheaper/smaller LLMs behave more like reasoning models seems to be working well—another paper reproduces similar results more efficiently! The DeepSeek-R1 paper introduced an experiment where they fine-tuned smaller Qwen and Llama models to improve their reasoning abilities by using outputs from the larger DeepSeek-R1 671B model. Some have called this soft distillation, while others say it's fine-tuning, but you get the point! Another recent paper has done something similar: ⛳ The paper focuses on improving LLMs' reasoning ability by getting them to generate Long Chain-of-Thought (Long CoT) responses for complex problems. ⛳It uses DeepSeek-R1's results to fine-tune smaller models like the Qwen2.5-32B-Instruct . ⛳They use only supervised fine-tuning (SFT) and low-rank adaptation (LoRA) with just 17k samples, meaning they didn't even modify the entire model (only 5% of it as per the authors). ⛳ The paper highlights that the structure of Long CoTs is far more critical than the content of individual reasoning steps. Errors in the content (e.g., mistakes in reasoning steps) have little impact, while disrupting the structure (e.g., deleting or shuffling reasoning steps) significantly hurts performance. ⛳They demonstrate that this approach works across different models and tasks! If this approach works well, many smaller models could be adapted to perform reasoning tasks! 💡 It's super interesting, every breakthrough with large models seems to push smaller models to become much more powerful simply by using these big models as teachers. Link: https://lnkd.in/e5rzRWqd

The interview is for an Applied Scientist role at Oracle, focused on optimizing LLMs for domain-specific use. Interviewer: "Let's say your LLM struggles with financial QA - for example, understanding IFRS accounting rules. Would you fine-tune or prompt-engineer?" You: "Depends on three variables: data sensitivity, response variability, and inference cost." Interviewer: "Explain." You: "If the model frequently misinterprets domain-specific terms - say, impairment loss vs. write-off - prompt engineering may not fix it, because the model hasn’t internalized that domain semantics. Fine-tuning helps the model internalize these relationships." "But if the gap is contextual adaptation (not factual misunderstanding), then structured prompting + retrieval augmentation does the job." Interviewer: "So where’s the boundary?" You: Use prompt engineering when: - Domain knowledge is externalizable via retrieval (e.g., product manuals, docs). - Model performance depends more on contextual clarity than internal weights. - You need quick iteration and transparency. Use fine-tuning when: - The model repeatedly fails to interpret domain-specific logic. - There's abundant domain QA or conversation data. - You need deterministic behavior (e.g., financial or legal assistants). Interviewer: "What about hybrid approaches?" You: "That’s often the sweet spot: - Start with RAG for quick wins. - Collect high-quality interactions over time. - Then fine-tune a smaller model (like Llama 3 8B) with those examples for latency + cost efficiency." In short: don’t fine-tune for context gaps; fine-tune for conceptual gaps. #LLMs #FineTuning #PromptEngineering #AIResearch

This paper examines the adaptation and performance of Transformer-based LLMs in the biomedical domain, focusing on their use in Natural Language Inference (NLI) and Named Entity Recognition (NER) tasks. 1️⃣ Pre-trained models significantly outperform randomly initialized ones, highlighting the critical role of pre-training in learning contextualized representations applicable to downstream tasks. 2️⃣ Domain-specific pre-training provides substantial benefits, particularly for tasks like NER that rely on specialized terminologies, with models such as BioBERT and BioGPT outperforming their general-purpose counterparts. 3️⃣ Encoder-based models (e.g., BERT) generally outperform decoder-based models (e.g., GPT-2) due to their bidirectional structure, which captures contextual information more effectively. 4️⃣ Fine-tuning redistributes task-relevant information across layers, with later layers encoding the most specialized knowledge after tuning, especially in domain-adapted models. 5️⃣ Domain-specific LLMs show greater stability during fine-tuning, requiring fewer changes to their internal mechanisms, which aligns with their pre-training on specialized corpora. 6️⃣ Fine-tuning efficiency varies across architectures; encoder-based models excel with smaller datasets, whereas decoder-based models demonstrate substantial improvements only with larger datasets. 7️⃣ Probing tasks reveal distinct patterns of knowledge encoding in LLM layers, with encoder models concentrating task-specific information in intermediate and later layers. 8️⃣ Attention mechanisms dynamically adapt during fine-tuning, with significant shifts reflecting alignment with task-specific requirements, especially in non-domain-adapted models. 9️⃣ Dynamic Time Warping analysis highlights the resilience of domain-specific LLMs, showing less dramatic shifts in attention patterns, particularly for larger datasets. 🔟 Strategic preliminary analysis of models’ internal dynamics can guide decisions about further tuning or data annotation, optimizing resource allocation in data-scarce biomedical domains. ✍🏻 Agnese Bonfigli, Luca Bacco, Mario Merone, Felice Dell'Orletta. From pre-training to fine-tuning: An in-depth analysis of Large Language Models in the biomedical domain. Artificial Intelligence In Medicine. 2024. DOI: 10.1016/j.artmed.2024.103003

TL;DR 🧠 Smaller LLMs outperform giants: A 1B LLM can surpass a 405B LLM on reasoning tasks like MATH-500 using compute-optimal Test-Time Scaling (TTS). 🚀 Efficiency boost: Smaller models achieve higher accuracy with 14.1× faster inference and 256× fewer FLOPS compared to larger models. 🔍 Key insight: TTS strategies depend on policy model size, Process Reward Models (PRMs), and problem difficulty. Problems & Solutions 🛑 Problem 1: Lack of systematic analysis of how policy models, PRMs, and problem difficulty affect TTS. ✅ Solution: Introduced reward-aware compute-optimal TTS to dynamically adapt strategies. 🛑 Problem 2: PRMs struggled with out-of-distribution (OOD) responses and token-length bias. ✅ Solution: Implemented absolute difficulty thresholds and PRM-Vote aggregation to improve robustness. Experiments & Setup 📚 Tasks: MATH-500 (500 problems) and AIME24 (advanced math challenges). 🤖 Models: Llama 3 (1B-405B), Qwen2.5 (0.5B-72B), and DeepSeek-R1 variants. ⚖️ Metrics: Pass@k, token efficiency, FLOPS comparison. 🔧 Ablations: PRM scoring methods (Min/Last/Avg) and voting strategies (Majority/PRM-Max/PRM-Vote). 💻 Hardware: 8×A100 GPU clusters for TTS experiments with beam width=4 and max tokens=8192. Novel Insights 🧩 Policy model size matters: Best-of-N (BoN) works well for large models, while Beam Search and DVTS excel for smaller ones. 📉 PRM limitations: Observed over-criticism, error neglect, and token-length bias in PRMs, impacting TTS performance. ⚖️ Trade-off: TTS gains diminish as policy model size increases (e.g., 154.6% gain for 1B vs. 9.5% for 72B). Improvements Over Prior Work 🚀 135× size gap: A 3B model outperforms a 405B model, improving the prior benchmark of 23×. 🔬 Enhanced PRMs: Qwen2.5-Math-PRM-72B enables 7B models to surpass o1 and DeepSeek-R1. ⏱️ Efficiency: 1B model + TTS achieves 256× fewer FLOPS compared to 405B CoT models. Key Implementation Details 🔄 Reward-aware TTS: Integrated PRM scores into a Markov Decision Process (MDP) framework for dynamic scaling. 🌳 DVTS: Parallel subtree exploration for diverse reasoning paths. 📉 Absolute difficulty bins: Replaced quantile-based thresholds with fixed Pass@1 ranges (easy: 50%-100%, medium: 10%-50%, hard: 0%-10%). Resources Paper: Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling (https://lnkd.in/g55ybikb) 🤖 Models: Llama-3.2-3B-Instruct (https://lnkd.in/gnQ3d87S), Qwen2.5-Math-PRM (https://lnkd.in/gk6gMqMw). 🔧 Framework: OpenR (https://lnkd.in/gCPxPR4H) for TTS pipelines. 📊 Datasets: MATH-500 (https://lnkd.in/g4jvAzsp), PRM800K (https://lnkd.in/gEb6XE3A). 🌐 Project Page: Compute-Optimal TTS (https://lnkd.in/gVutpamZ).