Understanding Model Frameworks

Training LLMs for spam classification: I added 14 experiments comparing different approaches: https://lnkd.in/gTNVvGcj - which token to train - which layers to train - different model sizes - LoRA - unmasking - and more! Any additional experiments you'd like to see? And here are the take aways for the table shown in the picture: 1. Training the Last vs. First Output Token (Row 1 vs. 2): Training the last output token results in substantially better performance compared to the first. This improvement is expected due to the causal self-attention mask. 2. Training the Last Transformer Block vs. Last Layer (Row 1 vs. 3): Training the entire last transformer block is also results in substantially better results than training only the last layer. 3. Training All Layers vs. Last Transformer Block (Row 1 vs. 4): Training all layers shows a modest improvement of ~2% over just training the last transformer block, but it requires almost three times longer in terms of training duration. 4. Using Larger Pretrained Models (Row 1 vs 5, and Row 1 vs. 6 and 7): Employing a 3x larger pretrained model leads to worse results. However, using a 5x larger model improves performance compared to the initial model, as was anticipated. Similarly, the 12x larger model improves the predictive performance even further. (The medium model was perhaps not well pretrained or the particular finetuning configuration works not as well for this model.) 5. Using a Model with Random Weights vs. Pretrained Weights (Row 1 vs. 8): Utilizing a model with random weights yields results that are only slightly worse by 1.3% compared to using pretrained weights. 6. Using LoRA (Low-Rank Adaptation) vs Training All Layers (Row 9 vs. 4): Keeping the model frozen and adding trainable LoRA layers (see Appendix E for details) is a viable alternative to training all model parameters and even improves the performance by 1% point. As it can be seen by the 1% lower gap between the training and validation accuracy when using LoRA, this is likely due to less overfitting. 7. Padding Input to Full Context Length vs. Longest Training Example (Row 1 vs. 10): Padding the input to the full supported context length results is significantly worse. 8. Padding vs no padding (Row 1 vs. 11 and 12): The `--no_padding` option disables the padding in the dataset, which requires training the model with a batch size of 1 since the inputs have variable lengths. This results in a better test accuracy but takes longer to train. In row 12, we additionally enable gradient accumulation with 8 steps to achieve the same batch size as in the other experiments. 9. Disabling the causal attention mask (Row 1 vs. 13): Disables the causal attention mask used in the multi-head attention module. This means all tokens can attend all other tokens. The model accuracy is slightly improved compared to the GPT model with causal mask.

Most people use AI tools like ChatGPT, Claude, or Gemini, but few understand how they actually work under the hood. So I put together this visual to explain the 3 main types of LLM architectures, in the simplest way possible. 𝟭. 𝗘𝗻𝗰𝗼𝗱𝗲𝗿-𝗢𝗻𝗹𝘆 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗹𝗶𝗸𝗲 𝗕𝗘𝗥𝗧, 𝗥𝗼𝗕𝗘𝗥𝗧𝗮) These models mask some words in a sentence and try to guess them using context from both sides. 𝘌𝘹𝘢𝘮𝘱𝘭𝘦: Input: "New York is a [MASK] city." → Output: "𝗯𝗶𝗴" Great for understanding language. Used in: Sentiment analysis, entity recognition, document classification. 𝟮. 𝗗𝗲𝗰𝗼𝗱𝗲𝗿-𝗢𝗻𝗹𝘆 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗹𝗶𝗸𝗲 𝗚𝗣𝗧, 𝗖𝗹𝗮𝘂𝗱𝗲) These models read text from left to right and predict the next word. 𝘌𝘹𝘢𝘮𝘱𝘭𝘦: Input: "Once upon a time, there was a dragon who" → Output: "𝗹𝗶𝘃𝗲𝗱 𝗶𝗻 𝗮 𝗰𝗮𝘃𝗲" Great for generating long text. Used in: Chatbots, writing, coding, storytelling. 𝟯. 𝗘𝗻𝗰𝗼𝗱𝗲𝗿-𝗗𝗲𝗰𝗼𝗱𝗲𝗿 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗹𝗶𝗸𝗲 𝗧𝟱, 𝗕𝗮𝗿𝗱) These models read the input, understand it, and then rewrite it in a new form. 𝘌𝘹𝘢𝘮𝘱𝘭𝘦: Input: "Summarize: The stock market saw a major drop due to inflation fears and rising interest rates." → Output: "𝗠𝗮𝗿𝗸𝗲𝘁𝘀 𝗳𝗲𝗹𝗹 𝗼𝘃𝗲𝗿 𝗶𝗻𝗳𝗹𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗿𝗮𝘁𝗲 𝗰𝗼𝗻𝗰𝗲𝗿𝗻𝘀." Great at transforming text. Used in: Translation, summarization, question answering. Each architecture is built for a different purpose. Understanding these basics can help you make better decisions, whether you're building with AI or simply trying to understand what powers your tools.

Most people still think of LLMs as “just a model.” But if you’ve ever shipped one in production, you know it’s not that simple. Behind every performant LLM system, there’s a stack of decisions, about pretraining, fine-tuning, inference, evaluation, and application-specific tradeoffs. This diagram captures it well: LLMs aren’t one-dimensional. They’re systems. And each dimension introduces new failure points or optimization levers. Let’s break it down: 🧠 Pre-Training Start with modality. → Text-only models like LLaMA, UL2, PaLM have predictable inductive biases. → Multimodal ones like GPT-4, Gemini, and LaVIN introduce more complex token fusion, grounding challenges, and cross-modal alignment issues. Understanding the data diet matters just as much as parameter count. 🛠 Fine-Tuning This is where most teams underestimate complexity: → PEFT strategies like LoRA and Prefix Tuning help with parameter efficiency, but can behave differently under distribution shift. → Alignment techniques- RLHF, DPO, RAFT, aren’t interchangeable. They encode different human preference priors. → Quantization and pruning decisions will directly impact latency, memory usage, and downstream behavior. ⚡️ Efficiency Inference optimization is still underexplored. Techniques like dynamic prompt caching, paged attention, speculative decoding, and batch streaming make the difference between real-time and unusable. The infra layer is where GenAI products often break. 📏 Evaluation One benchmark doesn’t cut it. You need a full matrix: → NLG (summarization, completion), NLU (classification, reasoning), → alignment tests (honesty, helpfulness, safety), → dataset quality, and → cost breakdowns across training + inference + memory. Evaluation isn’t just a model task, it’s a systems-level concern. 🧾 Inference & Prompting Multi-turn prompts, CoT, ToT, ICL, all behave differently under different sampling strategies and context lengths. Prompting isn’t trivial anymore. It’s an orchestration layer in itself. Whether you’re building for legal, education, robotics, or finance, the “general-purpose” tag doesn’t hold. Every domain has its own retrieval, grounding, and reasoning constraints. ------- Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

Not every AI problem needs a giant LLM. In fact, picking the biggest model is often the most expensive way to ship a worse product. The teams winning in 2026 know exactly when to use an LLM, when to switch to an SLM, and when only a multimodal model will do. Here's how the three actually compare: 🔹 𝐋𝐋𝐌 (𝐋𝐚𝐫𝐠𝐞 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬) Built for: human-like text at scale Powers: chatbots, copilots, content tools Compute: very high (GPUs/TPUs) Examples: GPT-4, Llama 3, Claude, PaLM 🔹 𝐒𝐋𝐌 (𝐒𝐦𝐚𝐥𝐥 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬) Built for: speed, efficiency, privacy Powers: edge AI, on-device apps, embedded systems Compute: low to moderate Examples: Phi-3, Gemma, Mistral 7B, TinyLlama 🔹 𝐌𝐮𝐥𝐭𝐢𝐦𝐨𝐝𝐚𝐥 𝐌𝐨𝐝𝐞𝐥𝐬 Built for: text + image + audio + video understanding Powers: image captioning, video analysis, voice assistants Compute: high (multi-modal processing) Examples: GPT-4o, Gemini 1.5, Claude 3, LLaVA Quick rule: → Need depth and reasoning → LLM → Need speed, edge, or privacy → SLM → Need to understand more than text → Multimodal The best AI isn't the biggest. It's the one that fits the job. Save this before your next AI decision. #AI #LLM #SLM #MultimodalAI #GenAI

When a public LLM benchmark says one model is better than another, it is testing broad, difficult reasoning and, more importantly, opinionated tasks. Your use case might not need that. For example, if you are summarizing tickets, classifying intent, or extracting fields from structured text, a smaller and cheaper model can match the output of a frontier model. General benchmarks tell us which model wins on average across a wide mix of tasks. They do not tell us which model wins on your specific, narrow, repeatable task. The only way to know for sure is to build your own benchmark. One simple way is to take real examples from your workload, run them across your candidate models, and score the outputs against what you actually need. It is laborious work, but yes, your favorite LLM can help you build that :) A model ranked lower on a general benchmark can be the better, cheaper choice for your use case, simply because your task was never complex enough to need the extra capability in the first place. In a gist, always try to answer: Which model is better for your task? Not which model is better in general. Hope this helps.

The Evolution of RAG: From Simple Retrieval to Deep Reasoning Agents A fascinating new survey from leading universities including Tsinghua University, UIC, and HKUST reveals how Retrieval-Augmented Generation is transforming from basic fact-lookup to sophisticated reasoning systems. The Technical Journey: Traditional RAG systems follow a static "Retrieval-Then-Reasoning" approach - retrieve once, then generate. But researchers have identified critical limitations: retrieved knowledge often misaligns with actual reasoning needs, and errors propagate through reasoning chains. Three Evolutionary Stages: 1 Reasoning-Enhanced RAG: Uses multi-step reasoning to optimize retrieval queries, assess relevance, and synthesize information across the entire RAG pipeline. Think smarter query reformulation and reasoning-aware document filtering. 2 RAG-Enhanced Reasoning: Leverages external knowledge bases, web search, and tools to ground reasoning in factual evidence, preventing hallucinations during complex inference. 3 Synergized RAG-Reasoning: The breakthrough paradigm where retrieval and reasoning iteratively enhance each other. Systems dynamically decide what to search, when to reason, and how to integrate new evidence. Under the Hood: These advanced systems employ diverse reasoning workflows - from chain-based approaches for efficiency to tree-based exploration (Tree-of-Thought, MCTS) for complex scenarios, and graph-based methods for structured knowledge navigation. Agent orchestration ranges from single LLMs with ReAct loops to multi-agent systems with specialized roles. The Deep Research Revolution: Modern implementations like OpenAI's Deep Research and similar systems showcase agentic capabilities - they autonomously plan multi-step queries, coordinate specialized tools, and synthesize findings across diverse sources. What's Next: The survey identifies key challenges: reasoning efficiency, multimodal retrieval, trustworthiness of sources, and human-AI collaboration. The future points toward systems that adapt reasoning strategies based on user expertise and context. This represents a fundamental shift from passive information retrieval to active research assistance - truly intelligent systems that think, search, and reason in integrated loops.

How to Choose the Right LLM Architecture A conversation I keep having with teams stuck debugging AI systems that should be working - the issue almost never turns out to be the model. It is usually that they picked prompt-only when the job needed retrieval or reached for fine-tuning when what they actually needed was grounded RAG with citations. The distinction that matters most: RAG updates what a system knows, fine-tuning changes how it behaves. Confusing those two is probably the single most common architecture mistake happening right now. Teams fine-tune a model hoping it will pick up fresh company data, when what they needed was retrieval. Or they lean entirely on RAG expecting consistent tone and behaviour, when that requires fine-tuning instead. Beyond that base layer, the pattern selection gets specific fast. Connected reasoning across entities points to GraphRAG. Reliable JSON and deterministic schemas point to structured output. Actions in external systems point to tool-connected or MCP-connected agents. Autonomous multi-step work points to agentic workflows. Several expert roles collaborating points to multi-agent systems. The rule of thumb: almost no serious production system runs on a single pattern. Retrieval, tools, memory, and routing get combined and knowing which combination the job actually needs is the real skill here, well before any conversation about which model to use. #LLMArchitecture #AIEngineering

A new paper from Technical University of Munich and Universitat Politècnica de Catalunya Barcelona explores the architecture of autonomous LLM agents, emphasizing that these systems are more than just large language models integrated into workflows. Here are the key insights:- 1. Agents ≠ Workflows Most current systems simply chain prompts or call tools. True agents plan, perceive, remember, and act, dynamically re-planning when challenges arise. 2. Perception Vision-language models (VLMs) and multimodal LLMs (MM-LLMs) act as the 'eyes and ears', merging images, text, and structured data to interpret environments such as GUIs or robotics spaces. 3. Reasoning Techniques like Chain-of-Thought (CoT), Tree-of-Thought (ToT), ReAct, and  Decompose, Plan in Parallel, and Merge (DPPM) allow agents to decompose tasks, reflect, and even engage in self-argumentation before taking action. 4. Memory Retrieval-Augmented Generation (RAG) supports long-term recall, while context-aware short-term memory maintains task coherence, akin to cognitive persistence, essential for genuine autonomy. 5. Execution This final step connects thought to action through multimodal control of tools, APIs, GUIs, and robotic interfaces. The takeaway? LLM agents represent cognitive architectures rather than mere chatbots. Each subsystem, perception, reasoning, memory, and action, must function together to achieve closed-loop autonomy. For those working in this field, this paper titled 'Fundamentals of Building Autonomous LLM Agents' is an interesting reading:- https://lnkd.in/dmBaXz9u #AI #AgenticAI #LLMAgents #CognitiveArchitecture #GenerativeAI #ArtificialIntelligence

Most PE firms don’t have an operating model. They have a PowerPoint. The “Value Creation Plan” (VCP) has become a kind of corporate theatre.....a glossy internal pitch deck designed to impress LPs, not to actually change how portfolio companies run. It’s full of big promises: digital transformation, pricing excellence, sales enablement. Then it disappears into the ether the moment the deal closes. In reality, most firms are still managing value creation through ad hoc emails, Excel trackers, and quarterly updates. They talk about playbooks, but they don’t operate from one. The “ops model” is a myth built to sell credibility, not capability. A real operating model isn’t a presentation. It’s infrastructure. It’s what happens every Monday morning, not every QBR. Here’s what that would actually look like: 1. Live dashboards, not static reports. Every portfolio company should feed performance data into a single platform, updated weekly. You don’t need 40 KPIs per company, you need 6 that matter. Revenue, CAC, LTV, conversion rate, retention, and cash. Real-time visibility drives action. 2. Cross-portfolio benchmarks. Stop pretending every company is unique. The point of owning multiple businesses is to spot patterns and act on them. If one business converts at 8% and another at 3%, that’s an immediate playbook opportunity. You don’t need consultants to find that; you need standardised data. 3. Capability pods. Hire functional experts.....in digital marketing, pricing, M&A integration....and let them operate across the portfolio. These aren’t advisors or slide-makers. They’re doers who drop into companies to fix, build, and leave systems behind. Think of it as shared services for execution. 4. Cadence and accountability. Weekly stand-ups. Monthly performance reviews. Clear owners. If the only time your ops team meets management is before board meetings, you don’t have a system, you have a reaction function. 5. Shared language of performance. Right now, every portfolio company defines “success” differently. Some measure top-line growth, others focus on EBITDA, others on margin expansion. A real model creates a unified framework: one vocabulary, one scorecard, one rhythm. When you see the few firms that do this well, the difference is obvious. Their portfolio dashboards look like trading floors. Their operators sound like GMs, not consultants. And their CEOs know where they stand every week, not every quarter. Until then, “value creation” will stay exactly where it began: inside a PowerPoint, somewhere between the “synergy pipeline” slide and the “next steps” placeholder. #PrivateEquity #OperatingModel #ValueCreation #ClaymorePartners #notveryprivateequity

We benchmarked 15 LLM models on finding real vulnerabilities. The results surprised us. Everyone talks about AI for security. We wanted numbers. We built an automated pentesting harness and pointed 15 different LLMs at OWASP Juice Shop — same target, same tools, same sandbox. Each model got source code access and a running instance. No hand-holding, no prompt engineering tricks. Just: "find vulnerabilities." Here's what we found after 38 runs: The leaderboard isn't what you'd expect. - Kimi K3 (Moonshot) found 62 vulnerabilities — more than any Claude or GPT model - Grok 4.5 (xAI) hit 59 findings in just 26 minutes — the best speed/quality ratio - Claude Opus 4.5 took 3rd place (57 findings), but the current flagship Opus 4.8 ranked 11th with only 26 - GLM 5.2 (Zhipu) achieved the lowest cost per finding: $0.007 — yes, less than a penny per vulnerability The cost story is wild: GLM 5.2 spent $0.32 total and found MORE vulnerabilities than Claude Opus 4.8 at $2.28. The most expensive model per finding (Opus 4.1 at $0.14/finding) found 3x fewer issues than the cheapest (GLM at $0.007/finding). What actually matters isn't the model — it's the harness. Our minimal-prompt harness (just "find vulnerabilities" + sandboxed tools) consistently outperformed the 900-line methodology prompt we spent weeks engineering. The tooling and verification loop carry quality. The model provides the intelligence, but the framework determines whether that intelligence is directed effectively. Three takeaways for anyone building AI security tools: 1. Don't assume the most expensive model wins. Benchmark your actual task. 2. Non-Western AI labs are producing surprisingly strong security-capable models. 3. The orchestration layer matters more than the model layer. Invest in tools, sandboxing, and verification — not just prompt engineering. Full benchmark data in the attached image. #AISecurity #Cybersecurity #LLM #PenetrationTesting #AIBenchmark #InfoSec #Aurascape #GLM #KIMI #KIMIK3 #Deepseek #Grok #Anthropic #OpenAI #Gemini