Enhancing Product Recommendations

Product Discovery is the most critical area for a PM. But, it is largely misunderstood. Teams waste time and energy delivering ideas that do not work and do not drive the expected outcomes. Product Discovery 101: 1. Why do we need Product Discovery? „The first truth is that at least half of your ideas are just not going to work” - Marty Cagan, Inspired I’d argue that Product Management is, at its heart, about managing risk. And for every product, there are 5 risks that can materialize: - Value. Will it create value for the customers? - Usability. Will users figure out how to use it? - Viability. Can our business support it? - Feasibility. Can it be done (technology)? - Ethics. Should we do it? Are there any ethical considerations? What will happen if we throw random ideas into the Product Backlog? This is agile "learning by delivering."This approach results in a waste and rework. So, we would like to understand: - How can we come up with better ideas? - How can we validate those ideas before the implementation? And the answer is Product Discovery. --- 2. When Does it Happen? Continuous Discovery and Continuous Delivery. Those two streams run in parallel: - The goal of Product Discovery is to discover the product to build. - Product Delivery aims to deliver that product to the market. Product Discovery results in a validated Product Backlog. In particular, high-risk assumptions are tested before the implementation. --- 3. Who's Responsible? Some say that the Product Manager decides what to build, and Engineers and designers should focus on how to build it. Have you heard that before? It hurts my ears because Product Discovery is not a task for a single person. Make sure that a Product Designer and at least one Engineer are included. This will help you build a shared understanding and stay open to different perspectives. And if we believe that customers don't know how to solve their problems, why should a Product Manager know it? Product Managers may be tech-literate, but they are not tech experts. --- 4. What’s inside Continuous Product Discovery? There are two groups of activities: - Exploring the Problem Space to understand and define opportunities (problems, needs, desires). My favorite default approach to mapping opportunities is using the Opportunity Solution Tree, as defined by Teresa Torres. - Exploring the Solution Space to explore possible solutions, formulate testable assumptions, and run experiments to prove or disprove those assumptions. --- What's the state of Product Discovery in your company? What is one improvement you can implement starting tomorrow? Hope that helps! --- 🎁 P.S. In my free post, I described Product Discovery in depth. No email, no paywall: https://lnkd.in/dNUB__n3 And here you can download all my infographics: https://lnkd.in/d5bHGj5j

I've reviewed 50+ #RecSys architectures. The most expensive mistake I keep seeing: Using a 70B LLM to do real-time ranking. It fails in production. Every time. Here's why and what actually works instead 👇 At Amazon and Twitter, we operated under strict latency budgets. You cannot run autoregressive inference over a massive catalog in ~50ms. That's not a modeling problem. That's physics. The real issue isn't that LLMs are weak. It's that they're being used in the wrong place. The pattern is called LLM-ERS (LLM-Enhanced Recommender Systems): Layer 1 - Retrieval + Ranking (keep it traditional) Fast. Scalable. No LLM needed here. → Two-Tower embeddings → Vector DB retrieval → Top ~1K candidates in sub-10ms Layer 2 - Offline Data Augmentation (highest ROI) Move the LLM off the critical path entirely. Use it to improve your training data instead: → Generate hard negatives for triplet loss → Plausible-but-wrong items that force tighter decision boundaries Zero latency cost. Massive modeling gain. Layer 3 - Post-Ranking Personalization (controlled LLM use) If you need real-time LLMs, keep the blast radius small: → Take only your top 3–5 ranked items → Generate "Why you might like this" explanations You get personalization + better UX - without breaking P99. The rule that separates robust systems from expensive ones: → Traditional ML decides what to show → LLMs enhance how it's understood Boring architecture for scale. LLMs for the final mile. This is Part 3 of my Generative RecSys series. Next up: I'll break down the Generative Retrieval Paradigm. I'm genuinely curious: where in your stack have LLMs actually moved the needle vs. just adding cost and latency? Drop your answer + one sentence on why. A) Retrieval B) Ranking C) Explanation / UX layer D) Offline data augmentation E) Nowhere yet - still evaluating

Zomato faced a big problem: How can we turn app browsers into loyal customers? The goal was clear, improve the user experience with personalized restaurant suggestions. But there were a few challenges too: 🔴 Understanding user preferences from massive data. 🔴 Combining multiple data sources for meaningful insights. 🔴 Developing accurate recommendation algorithms. 🔴 Processing data in real time to keep users engaged. 🔴 Building trust in the recommendations to ensure they felt helpful, not intrusive. To tackle this, Zomato used a structured approach: 🟢 Data Collection and Cleaning - They collected user behavior data (searches, clicks, abandoned carts). - They analyzed restaurant details (cuisine types, delivery times, ratings). - Past orders were also analyzed for trends. 🟢 User Segmentation - Users were grouped based on age, location, past orders, and browsing habits. - This helped them identify patterns and preferences. 🟢 Developing the Recommendation System - Combined collaborative filtering (what others like you prefer) and content-based filtering (what matches your past orders). - Fine-tuned algorithms with ongoing testing for better accuracy. 🟢 Implementation and Testing - They rolled out the recommendations and tested them through A/B experiments. - Adjusted based on user feedback and data performance. 🟢 Continuous Improvement - Introduced feedback loops for real-time adjustments. - Regular updates ensured the system stayed relevant to evolving user needs. And, the impact was impressive: ⬆️ 35% more time spent on the app by users receiving personalized suggestions. ⬆️ 28% higher click-through rates, showing better engagement. ⬆️ 22% increase in orders per user per month due to tailored suggestions. ⬆️ 18% boost in retention rates, turning occasional users into loyal customers. ⬆️ 12% higher average order value, leading to revenue growth. ⬆️ 15% jump in monthly revenue, proving personalization works! I see this as the perfect example of using data to deepen customer relationships. It's not just about the tech—it’s about understanding people and making their experience smoother and more personal. 📊 Data is the secret to building trust and loyalty. What do you think? Can other industries learn from Zomato’s success? How can personalization improve your industry? #zomato #deepindergoyal

Can LLMs improve product recommendations with re-ranking? Fascinating new paper from Meta on applying LLMs to recommendation systems. The domain discussed in the paper is content re-ranking, but I don't see why this couldn't be applied to ads. Re-ranking takes a ranked list of candidate items (following retrieval and, sometimes, pre-ranking) and updates the ordering to better optimize some objective function (eg., purchase). The authors describe how they utilize an LLM to re-rank candidates with a number of novel innovations: - Instead of building the LLM vocabulary from item embeddings, which would likely be too large to be useful, they decompose each item embedding into a sequence of "tokens" produced from a K-stage quantization process (with RQ-VAE). This process accepts the item embedding at k=1, calculates the residual vector from the nearest of C learned centroids for that step (called "codebooks"), and passes that output to k=2, and so on to K. This produces a Semantic ID (SID) of length K. - Because re-ranking must be done quickly, it requires a smaller LLM (the paper uses 8B parameters). So in training, they prompt a large model (Qwen-32B) with the user's history, the candidate items produced in ranking, the SIDs, and instructions to reason through its process of re-ranking these items. That model produces a reasoning trace and a re-ranked list. The authors use rejection sampling to retain only the outputs (reasoning traces + rankings) for which the ground-truth item is ranked sufficiently high. The 8B student model is fine-tuned via SFT on that distribution, learning P( reasoning trace + ranking | prompt ). - Finally, the authors fine-tune this model with RL on the outcome, using the ground truth's location in the list as the reward. This aligns the model's policy with the reward, enabling more thorough comparison across candidates (versus reasoning collapse). The authors make the point that LLMs can introduce additional product context, scalability, and "world knowledge" to RecSys, turning ranking into a structured reasoning task rather than a pure scoring task. The paper is quite dense but worth reading in full; link below.

Meta just showed how to finally apply scaling laws to Recommender Systems by killing the monolithic DLRM. For years, RecSys engineers have watched NLP and Vision benefit from simply adding parameters and data, while recommendation models hit a wall. A new paper from Meta changes that trajectory by introducing a Foundation-Expert paradigm. Here are the biggest takeaways from my analysis of their architecture: The Monolith is dead. Instead of building one massive model per surface (Reels, Feed, etc.), they split the stack. A massive Foundation Model learns universal user representations from lifelong history, while lightweight Expert models handle surface-specific adaptation. They solved the distillation bottleneck. Standard Knowledge Distillation usually results in a low transfer ratio where the student model can't capture the teacher's capacity. Meta fixed this by replacing generic user embeddings with "target-aware embeddings." The Foundation Model modifies the user representation based on the specific target candidate before passing it to the Expert. This decoupling allows for trillion-parameter scale capacity. By separating general knowledge from task-specific logic, they can serve tens of billions of requests daily without the inference latency exploding. The full breakdown discusses why standard fine-tuning approaches led to catastrophic forgetting in this streaming environment and exactly how the target-aware mechanism bridges the gap. Read the full deep dive in the newsletter, link below.

RankMixer: Scaling Industrial Recommender Systems to Billion Parameters ByteDance just published breakthrough research on scaling recommendation models that addresses a critical challenge in industrial ML: how to dramatically increase model capacity while maintaining strict latency constraints. The Core Problem: Traditional ranking models were designed for the CPU era, combining heterogeneous handcrafted cross-feature modules that achieve dismally low GPU utilization (often single-digit MFU). Simply scaling these architectures doesn't work - the computational cost grows proportionally with parameters, making aggressive scaling economically unfeasible. The RankMixer Solution: The architecture introduces two key innovations that work in tandem: Multi-Head Token Mixing replaces expensive self-attention mechanisms with parameter-free operations. Unlike NLP tasks where tokens share unified embedding spaces, recommendation features are inherently heterogeneous (user IDs, item features, behavioral signals). Computing attention weights between disparate semantic spaces proves inefficient. Instead, RankMixer splits each token into multiple heads and concatenates corresponding heads across tokens, enabling cross-feature interaction without the quadratic complexity. Per-Token Feed-Forward Networks tackle the feature domination problem where high-frequency fields drown out long-tail signals. Each token gets dedicated transformation parameters, isolating modeling capacity for different feature subspaces. This differs fundamentally from traditional shared FFNs or mixture-of-experts approaches. Scaling Strategy: For extreme scale, they extend to Sparse MoE with novel training strategies. Dense-Training/Sparse-Inference combined with ReLU routing solves expert imbalance - all experts receive sufficient gradients during training while only relevant ones activate during inference. Real-World Impact: Deployed on Douyin's recommendation system, RankMixer achieved: - 70x parameter increase (16M to 1.1B) with stable inference latency - MFU improvement from 4.5% to 45% - 0.3% increase in user active days and 1.08% boost in app duration The key insight: decoupling parameter growth from FLOPs through hardware-aware design enables practical scaling that respects production constraints while delivering measurable business impact. This work demonstrates that scaling laws aren't just for LLMs - they apply to recommendation systems when the architecture is properly aligned with both the data characteristics and hardware capabilities.

Recommendation systems often reach a point where model improvements are no longer limited by modeling itself—they are limited by infrastructure. A more expressive model may promise better predictions, but if it cannot operate within strict latency budgets, those gains never make it into production. In this tech blog, engineers at Pinterest describe how they re-architected their ads serving stack to move beyond the traditional Two-Tower model. While Two-Tower architectures are highly efficient for retrieval and lightweight ranking, they struggle to capture rich interactions between users and items. The team introduced GPU-based inference into one of the most latency-sensitive parts of the serving pipeline, but doing so required much more than simply replacing CPUs with GPUs. To make richer ranking models practical at Pinterest's scale, the team redesigned several parts of the serving architecture. They reduced costly data movement by embedding frequently used features directly into model artifacts, moved business logic such as utility computation and candidate selection onto the GPU, streamlined candidate retrieval to minimize network overhead, and optimized inference through techniques such as kernel fusion and parallel GPU execution. The overall goal was to keep data close to computation and eliminate unnecessary handoffs throughout the ranking pipeline. This is a good reminder that recommendation quality often depends on balanced investments in both modeling and infrastructure. Better models create the potential for better outcomes, but as machine learning systems mature, it is the serving infrastructure that determines how much of that potential actually reaches users. #DataScience #MachineLearning #Ranking #Retrieval #GPU #Inference #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gFYvfB8V    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/e_pqvKku

For years, we've forced knowledge graphs into recommender systems, hoping their structure would magically yield explanations. Usually, it doesn't. We get accuracy gains, but the "why" remains trapped in vector space - a statistical ghost, not a logical chain. A new research work titled "Evolutionary Reinforcement Learning for Explainable Recommendation on Knowledge Graph", aces at it !!! Here’s what I found most compelling: 1) The "Mutation" hack: To navigate huge decision spaces, the AI doesn't just pick the top-ranked options. It intentionally mutates its list - swapping a few obvious choices for high-potential "dark horses." It's a brilliant, biologically-inspired trick to avoid local optima and stay creative. 2) The stunning (and puzzling) result: On most datasets, it beats state-of-the-art models by ~2-3%. But on the sparse, messy Amazon Cell Phones dataset, performance exploded: +51% Precision, +44% Hit Rate. This suggests the model isn't just a lab benchmark winner - it might be a secret weapon for noisy, real-world data where obvious patterns fail. 3) The honest limitation: The entire elegant system depends on a clean, structured Knowledge Graph (the map of connections between users, items, and features). The authors openly admit that building and maintaining this "map" is the hard, expensive, human part. The AI is a brilliant navigator, but it needs a good map. 4) The future vision: They propose teaming this system with Large Language Models. Let the RL agent find the rigorous, causal path. Then let the LLM translate that path into fluent, human-friendly language. This splits the work perfectly: reliability for the machine, articulation for the machine. This isn't just another accuracy bump. It's a philosophical shift - treating the "why" as a first-class output, not a post-hoc justification. The pressing question it leaves us with: If explainability at this level requires pristine knowledge graphs, how do we build and maintain them at scale in our messy, ever-changing digital world? The algorithm is ready. Is our data infrastructure? #ExplainableAI #XAI #ReinforcementLearning #KnowledgeGraph #RecommenderSystems #MachineLearning #AIResearch #DataScience #TechEthics

You click "play" on Netflix. In 200 milliseconds, a recommendation engine just processed millions of videos. Most ML engineers know these systems exist. Few understand what's actually running under the hood. I spent the last 6 months building a complete deep-dive series on production recommendation systems — from first principles to the exact architectures running at YouTube, Spotify, and TikTok. Here's the complete roadmap: 🎯 Foundation Layer 1️⃣ RecSys Fundamentals — Content-based, collaborative filtering, and hybrid approaches that power every modern recommender 2️⃣ How Recommendation Systems Learned to Think — The evolution from matrix factorization to transformer-based generative agents ⚡ Retrieval & Ranking Pipeline 3️⃣ The 3-Stage Funnel — How two-tower models, vector databases, and cross-encoders work together at scale 4️⃣ How YouTube Finds Your Next Video in Milliseconds — Two-tower retrieval, in-batch negatives, and the engineering tricks that make it work 5️⃣ Vector Search at Scale — IVF, PQ compression, and making 100M+ vector search actually possible in production 6️⃣ From Candidates to Clicks — The complete ranking stack: from 1,000 candidates to the one item you actually tap 🔧 Production Reality 7️⃣ Solving the Cold Start Problem — Contextual bandits, meta-learning, and LLMs for new users and items (how Spotify, TikTok, YouTube do it) 8️⃣ Beyond Ranking — How diversity, freshness, and business constraints turn a ranked list into a product-ready feed Every post includes: → Production architecture diagrams → Real code examples (PyTorch, Faiss, ranking models) → Case studies from actual systems → The engineering tradeoffs that matter Full series: https://buff.ly/GKEvulv If you're building RecSys or joining a team that does — this is your blueprint.

Should product discovery be done differently for Platforms vs. Products? Let's Explore! Product discovery is crucial, but it's not one-size-fits-all. While there's no shortage of materials on consumer product discovery, the unique challenges of platform product discovery often go unaddressed. Here's what makes platform product discovery distinct: 👨💻 𝗙𝗼𝗿 𝗠𝗶𝗱𝗱𝗹𝗲 𝗨𝘀𝗲𝗿𝘀: Whether they're internal teams or external developers, these users need tools that enhance their ability to innovate on your platform. Does your platform equip them to efficiently meet end-user needs? 🚀 𝗙𝗼𝗿 𝗘𝗻𝗱 𝗨𝘀𝗲𝗿𝘀: It’s all about the final experience. Does the platform serve the end goals of the ultimate customers effectively? 🤝 Dual-Layer Discovery: Platform products require a two-pronged approach: • 𝗘𝗻𝗱 𝗨𝘀𝗲𝗿 𝗙𝗼𝗰𝘂𝘀: Dive into understanding their specific goals and needs. What fundamental capabilities should your platform have to serve these needs?    • 𝗠𝗶𝗱𝗱𝗹𝗲 𝗨𝘀𝗲𝗿 𝗙𝗼𝗰𝘂𝘀: Assess whether the platform provides the necessary tools for these key developers. Is it easier and more cost-effective for them to use your platform than starting from scratch? Combining insights from both user types ensures your platform is not just functional but preferred choice. How do you approach product discovery for platform products? Do you find it differs significantly from consumer products? Share your thoughts and strategies in the comments!👇 #ProductDiscovery #ProductManagement #PlatformProductStrategy