Large Language Models Insights

Don't just blindly use LLMs, evaluate them to see if they fit into your criteria. Not all LLMs are created equal. Here’s how to measure whether they’re right for your use case👇 Evaluating LLMs is critical to assess their performance, reliability, and suitability for specific tasks. Without evaluation, it would be impossible to determine whether a model generates coherent, relevant, or factually correct outputs, particularly in applications like translation, summarization, or question-answering. Evaluation ensures models align with human expectations, avoid biases, and improve iteratively. Different metrics cater to distinct aspects of model performance:  Perplexity quantifies how well a model predicts a sequence (lower scores indicate better familiarity with the data), making it useful for gauging fluency. ROUGE-1 measures unigram (single-word) overlap between model outputs and references, ideal for tasks like summarization where content overlap matters. BLEU focuses on n-gram precision (e.g., exact phrase matches), commonly used in machine translation to assess accuracy. METEOR extends this by incorporating synonyms, paraphrases, and stemming, offering a more flexible semantic evaluation. Exact Match (EM) is the strictest metric, requiring verbatim alignment with the reference, often used in closed-domain tasks like factual QA where precision is paramount. Each metric reflects a trade-off: EM prioritizes literal correctness, while ROUGE and BLEU balance precision with recall. METEOR and Perplexity accommodate linguistic diversity, rewarding semantic coherence over exact replication. Choosing the right metric depends on the task—e.g., EM for factual accuracy in trivia, ROUGE for summarization breadth, and Perplexity for generative fluency. Collectively, these metrics provide a multifaceted view of LLM capabilities, enabling developers to refine models, mitigate errors, and align outputs with user needs. The table’s examples, such as EM scoring 0 for paraphrased answers, highlight how minor phrasing changes impact scores, underscoring the importance of context-aware metric selection. Know more about how to evaluate LLMs: https://lnkd.in/gfPBxrWc Here is my complete in-depth guide on evaluating LLMs: https://lnkd.in/gjWt9jRu Follow me on my YouTube channel so you don't miss any AI topic: https://lnkd.in/gMCpfMKh

Do you rely on one large generalist model to power multiple use cases, or do you build a suite of specialized models fine-tuned for specific tasks? Large Language Models (LLMs) act as the generalists. One model can handle many functions across financial services: -Fraud Detection -Automated Investing -Customer Service Chatbots -Personalized Banking -Consumer Loan Underwriting -This flexibility makes them ideal for exploration, rapid prototyping, and -scenarios where breadth of understanding matters more than hyper-optimization. Small Language Models (SLMs) act as the specialists. Each is optimized for a single task, such as: -Loan Qualification -Consumer Loan Underwriting -Fraud Detection -The benefit? Efficiency, accuracy, and cost control. By narrowing the scope, SLMs can outperform generalist models in production environments where precision is non-negotiable. The Hybrid Future The reality isn’t LLM or SLM — it’s both. LLMs will serve as the reasoning engines, orchestrating complex workflows and bridging gaps across domains. SLMs will deliver deep expertise in critical tasks, ensuring enterprise-grade performance. This hybrid approach mirrors how organizations operate: broad leadership supported by domain experts. As AI adoption accelerates, companies that can strike the right balance between generalist adaptability and specialist efficiency will set the standard for the next wave of digital transformation. Question for you: In your industry, are you leaning more toward the power of generalist LLMs, the precision of SLMs, or a blended strategy?

The AI landscape is undergoing a fundamental shift, and it’s not the one you think. The competitive frontier isn’t only about building the largest proprietary models. There are two other major trends emerging that haven’t had enough discussion: Open source models have made tremendous strides, especially on cost relative to performance. Compact, fit-for-purpose models can meaningfully out-perform general purpose LLMs on specific tasks and do so at dramatically lower cost. Our Chief Technology Officer and AI team share how we are using open source AI models at Pinterest to achieve similar performance at less than 10% of the cost of leading, proprietary AI models. They also share how Pinterest has built in-house, fit-for-purpose models that are able to significantly outperform leading, proprietary general purpose models. The race to build the largest, most powerful models is profound and meaningful. If you want to see a thriving ecosystem of innovation in an AI-driven world, you should also want to see a thriving open source AI community that creates democratization and transparency. It’s a good thing for us all that open source is in the race. For our part, we’ll continue to share our findings in leveraging open source AI so that more companies and builders can benefit from the democratizing effect of open source AI. https://lnkd.in/gmT6UNXs

For a long time, many companies built AI systems around a simple idea: choose the most powerful large language model available and use it across the entire workflow. One large model handling classification, summarization, routing, reasoning, and generation. What I am seeing now, especially going into 2026, is a clear architectural shift. Teams are moving away from the “one giant model does everything” approach. Instead, they are decomposing workflows and assigning different models to different layers of the system. Smaller, more specialized models are being used for well-defined tasks, while larger models are reserved for complex reasoning where their breadth actually matters. For those who are newer to this space, a SLMs typically refers to a model in the 1B to 12B parameter range. These models are optimized for efficiency, lower latency, and narrower domains. They are not designed to replace frontier-scale models, but to handle specific tasks extremely well. There are two practical reasons why I believe 2026 will be a high-adoption year for SLMs: ✦ Cheaper, faster, and more customizable For tasks like classification, structured extraction, lightweight reasoning, or domain-specific summarization, a smaller model is often more than sufficient. It runs with lower latency, costs less to scale, and if it is open source, it can be fine-tuned and adapted to your internal data and workflows. That level of customization gives teams real control over performance and differentiation. ✦ On-device and edge intelligence As more AI moves closer to the user, on-device and edge inference become critical. Mobile assistants, IoT systems, and privacy-sensitive enterprise applications cannot always rely on sending every request to a large cloud model. Small models make local inference feasible, improving both responsiveness and privacy. Large models are still essential for open-ended reasoning and complex generation. But the most mature systems will not rely on a single model. They will be orchestrated systems, where each model is chosen based on what it is best at. Model size is no longer the strategy, architecture is.

Generative AI cannot write what hasn’t been written. This is a subtle but profound truth the tech industry is just fully coming to understand. Ask a question of any large language model – from ChatGPT to Grok go Gemini – and you notice that LLMs cannot deliver reliable answers for topics that haven’t been covered. Gen AI programs work well when you are researching big companies like Nvidia and famous people like Tim Cook but not at all for the obscure or unsung. Since there is no way humans can keep up, the somewhat counterintuitive solution to this problem is to tap Gen AI to create a vast library of digital content. Effectively, we need machines to write articles so other machines can read them. To illustrate the challenge, consider Perplexity, an AI search engine, which is building a finance vertical. The biggest challenge won’t be the speed or depth of the LLMs that they leverage, but the lack of historical news about companies. In many cases, these are small or mid-sized companies that were never covered by reporters at Bloomberg or Reuters or the New York Times. Remember: Generative AI cannot write what hasn’t been written. The importance of specialized content was driven home by the recent announcement by DeepSeek, a Chinese LLM, when it said it had unveiled a high-performing open source large language model at a fraction of the cost of the version created by OpenAI. The DeepSeek announcement signaled that model enhancements will continue to come fast and furious, each leap-frogging the previous one and driving down inference costs.  Developers seeking to build moats around their applications will rely less on model performance and more and more on the quality, reliability and comprehensiveness of content. 

Most production systems that do retrieval or classification still run on BERT, a model released in 2018, even though the techniques used to build language models have changed substantially since then; the encoder family (models that read text and produce representations, rather than generating new text the way GPT-style models do) was largely left behind while attention went to large generative models. The authors take the accumulated improvements from recent generative models—rotary positional embeddings (a way of encoding word order that extends cleanly to longer texts), gated activation functions, alternating local and global attention so most layers only look at nearby words while a few look at everything, and removal of padding tokens that waste computation—and apply them to encoder models, then train on 2 trillion tokens including code at a native context length of 8192 tokens (sixteen times BERT's 512-word limit). The resulting models, ModernBERT-base and -large, do better than prior encoders across classification, single- and multi-vector retrieval, and code search, while processing long inputs roughly twice as fast as the next fastest competitor. Since its December 2024 release, ModernBERT has become a widely adopted encoder backbone, with the approach extended into multilingual variants and broadly treated as the current default starting point for encoder-based retrieval and classification work. Read and learn with an AI tutor: https://lnkd.in/eQWaDRCu PDF: https://lnkd.in/e9F_uSCm

One of OpenAI's top repositories is not a well-known one. The goal of this library, called simple-evals, is to provide a lightweight framework to evaluate the accuracy of language models using zero-shot and chain-of-thought prompts. Evals are sensitive to prompting, and these two techniques tend to better reflect models’ performance. The covered benchmarks include MMLU (reasoning), MATH (math), HumanEval (coding), and the recently released SimpleQA, checking for LLMs' tendency to hallucinate. This release also provides the benchmark results of known commercial models such as Claude and Gemini. GitHub repo https://lnkd.in/gqZQh8XZ

A child gathers more data in their first four years than all the text ever published online. That’s not just a fun stat. It highlights a core limitation in how modern AI is built. Most AI systems are trained on natural language data. They learn by extracting statistical patterns from language, not through embodied experience or real-world interaction. Compare that to how humans learn: → Multimodal sensory input processed in parallel → Continuous physical interaction with dynamic environments → Emotional and contextual feedback shaping understanding in real time Natural language is a compressed abstraction of experience. It encodes meaning, but strips away direct context, causality, and sensory nuance. That’s why language models excel at: Summarizing information at scale Extracting patterns from structured data Generating coherent, fluent responses …but often fail at: Grounding responses in real-world causality Navigating ambiguity or incomplete information Adapting to evolving, unstructured scenarios Even state-of-the-art models can: Confidently output factually incorrect information Misinterpret intent in natural instructions Break down when context isn’t explicitly encoded We’re training systems to imitate comprehension, using only the shadows of real experience. So what’s the next frontier? True progress in AI will require a leap beyond language: → Multisensory data (audio, video, spatial signals) → Embodied interaction → Context-aware models Language is an entry point. But if the goal is adaptive, human-like intelligence, grounded experience is essential.

This is the best and most current view of the market for LLMs, laid out as a series of succinct data visualizations on what matters. Well worth flipping through. It is accompanied by a paper that dives deeper into market evolution. Key insights include: 📈 Open-source supply is exploding but trails in adoption despite far lower cost. By late 2025, the cumulative number of open-source models approached 400, significantly outpacing the ~200 closed-source models available. Despite this variety and being roughly 90% cheaper than comparable proprietary models, open-source options account for less than 30% of the market share on average. 📉 The price of intelligence has crashed. The cost to access high-level intelligence plummeted between 2023 and 2025, with prompt prices for top-tier models falling from $30.00 per million tokens (GPT-4) to under $0.10 for capable newer models. When adjusting for the quality of intelligence provided, the relative price of tokens dropped by a factor of nearly 100 over this two-year period. 🧠 Consumers trade peak intelligence for value. Although the "frontier" of intelligence continues to rise with models like GPT-5 and Gemini 3 Pro, the average intelligence actually consumed by users consistently trails the highest available capability. Users effectively trade off maximum smarts for lower costs. 🛠️ Different industries demand different levels of "smarts." Demand is highly segmented by use case, with users requiring significantly higher intelligence levels for programming (index 0.51) and science (0.47) compared to tasks like translation (0.32) or trivia (0.37). Programming and technology tasks command higher premiums than creative writing or SEO. 🏆 Closed-source models retain dominance despite competition. Despite the proliferation of free-entry open-source providers, closed-source models continued to command the vast majority of token market share throughout 2025. However, the leaderboard is volatile; strictly closed-source incumbents face sudden disruption, such as when xAI surged from negligible usage to a significant market share within months in late 2025. 💻 Technical workflows exhibit extreme model turnover. In specialized applications like coding assistants, users abandon incumbent models rapidly; for instance, the "Cline" app saw usage shift from "Claude 4 Sonnet" in mid-2025 to being dominated by "Grok Code Fast 1" by October 2025. This sector demonstrates a "winner-takes-all" dynamic where a new, faster, or more specialized model can capture the majority of traffic in a remarkably short window. ⚓ Corporate demand is sticky and inelastic. Despite the rapid release of cheaper and faster models, business demand remains relatively inelastic in the short run, meaning firms do not immediately switch providers in response to price drops.

There’s been a lot of discussion about how Large Language Models (LLMs) power customer-facing features like chatbots. But their impact goes beyond that—LLMs can also enhance the backend of machine learning systems in significant ways. In this tech blog, Coupang’s machine learning engineers share how the team leverages LLMs to advance existing ML products. They first categorized Coupang’s ML models into three key areas: recommendation models that personalize shopping experiences and optimize recommendation surfaces, content understanding models that enhance product, customer, and merchant representation to improve shopping interactions, and forecasting models that support pricing, logistics, and delivery operations. With these existing ML models in place, the team integrates LLMs and multimodal models to develop Foundation Models, which can handle multiple tasks rather than being trained for specific use cases. These models improve customer experience in several ways. Vision-language models enhance product embeddings by jointly modeling image and text data; weak labels generated by LLMs serve as weak supervision signals to train other models. Additionally, LLMs also enable a deeper understanding of product data, including titles, descriptions, reviews, and seller information, resulting in a single LLM-powered categorizer that classifies all product categories with greater precision. The blog also dives into best practices for integrating LLMs, covering technical challenges, development patterns, and optimization strategies. For those looking to elevate ML performance with LLMs, this serves as a valuable reference. #MachineLearning #DataScience #LLM #LargeLanguageModel #AI #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gj6aPBBY    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gvaUuF4G