Multimodal AI Developments

🧐 Multimodal with just fine-tuning? ByteDance makes it happen with VoRA. The new Vision as LoRA (VoRA) paper introduces a bold, streamlined approach to building multimodal LLMs—no vision encoder, no connector, no architecture bloat. Just LoRA. Instead of bolting a vision tower onto an LLM, VoRA injects vision understanding directly into the LLM using Low-Rank Adaptation (LoRA). That means: 🔹 No new inference overhead- LoRA layers are merged into the LLM after training. 🔹 Frozen base LLM- Only LoRA + visual embeddings (~6M params) are trained, preserving language ability and ensuring stability. 🔹 Image inputs at native resolution- No resizing, no tiling hacks—VoRA leverages the LLM’s flexible token handling. 🔹 Bidirectional attention for vision- Instead of using causal masks across all tokens, VoRA allows vision tokens to attend freely—boosting context modeling. To teach the LLM visual features: 🔹 VoRA uses block-wise distillation from a pretrained ViT, aligning intermediate hidden states across layers. This improves visual alignment while keeping the LLM’s core untouched. 🔹 The training objective combines distillation loss (cosine similarity between ViT + LLM visual features) and standard language modeling loss over image-caption pairs. What does this get you? 🔹 A modality-agnostic architecture ready for extension to audio, point clouds, and beyond. This might be one of the most efficient takes yet on vision-language modeling. Excited to see how this evolves. 📄 Paper + Code: https://lnkd.in/gwZr33Vj Follow Aman Chadha and I for more updates!

Enterprises today are drowning in multimodal data - text, images, audio, video, time-series, and more. Large multimodal LLMs promise to make sense of this, but in practice, embeddings alone often collapse nuance and context. You get fluency without grounding, answers without reasoning, “black boxes” where transparency matters most. That’s why the new IEEE paper “Building Multimodal Knowledge Graphs: Automation for Enterprise Integration” by Ritvik G, Joey Yip, Revathy Venkataramanan, and Dr. Amit Sheth really resonates with me. Instead of forcing LLMs to carry the entire cognitive burden, their framework shows how automated Multi Modal Knowledge Graphs (MMKGs) can bring structure, semantics, and provenance into the picture. What excites me most is the way the authors combine two forces that usually live apart. On one side, bottom-up context extraction - pulling meaning directly from raw multimodal data like text, images, and audio. On the other, top-down schema refinement - bringing in structure, rules, and enterprise-specific ontologies. Together, this creates a feedback loop between emergence and design: the graph learns from the data but also stays grounded in organizational needs. And this isn’t just theoretical elegance. In their Nourich case study, the framework shows how a food image, ingredient list, and dietary guidelines can be linked into a multimodal knowledge graph that actually reasons about whether a recipe is suitable for a diabetic vegetarian diet - and then suggests structured modifications. That’s enterprise relevance in action. To me, this signals a bigger shift: LLMs alone won’t carry enterprise AI into the future. The future is neurosymbolic, multimodal, and automated. Enterprises that invest in these hybrid architectures will unlock explainability, scale, and trust in ways current “all-LLM” strategies simply cannot. Link to the paper -> https://lnkd.in/gv93znbQ #KnowledgeGraphs #MultimodalAI #NeurosymbolicAI #EnterpriseAI #KnowledgeGraphLifecycle #MMKG #AIResearch #Automation #EnterpriseIntegration

I consider prompting techniques some of the lowest-hanging fruits one can use to achieve step-change improvement with their model performance. This isn’t to say that “typing better instructions” is that simple. As a matter of fact, it can be quite complex. Prompting has evolved into a full discipline with frameworks, reasoning methods, multimodal techniques, and role-based structures that dramatically change how models think, plan, analyse, and create. This guide that breaks down every major prompting category you need to build powerful, reliable, and structured AI workflows: 1️⃣ Core Prompting Techniques The foundational methods include few-shot, zero-shot, one-shot, style prompts. They teach the model patterns, tone, and structure. 2️⃣ Reasoning-Enhancing Techniques Approaches like Chain-of-Thought, Graph-of-Thought, ReAct, and Deliberate prompting help LLMs reason more clearly, avoid shortcuts, and solve complex tasks step-by-step. 3️⃣ Instruction & Role-Based Prompting Define the task clearly or assign the model a “role” such as planner, analyst, engineer, or teacher to get more predictable, domain-focused outputs. 4️⃣ Prompt Composition Techniques Methods like prompt chaining, meta-prompting, dynamic variables, and templates help you build multi-step, modular workflows used in real agent systems. 5️⃣ Tool-Augmented Prompting Combine prompts with vector search, retrieval (RAG), planners, executors, or agent-style instructions to turn LLMs into decision-making systems rather than passive responders. 6️⃣ Optimization & Safety Techniques Guardrails, verification prompts, bias checks, and error-correction prompts improve reliability, factual accuracy, and trustworthiness. These are essential for production systems. 7️⃣ Creativity-Enhancing Techniques Analogy prompts, divergent prompts, story prompts, and spatial diagrams unlock creative reasoning, exploration, and alternative problem-solving paths. 8️⃣ Multimodal Prompting Use images, audio, video, transcripts, diagrams, code, or mixed-media prompts (text + JSON + tables) to build richer and more intelligent multimodal workflows. Modern prompting has fully evolved to designing thinking systems. When you combine reasoning techniques, structured instructions, memory, tools, and multimodal inputs, you unlock a level of performance that avoids costly fine tuning methods. What best practices have you used when designing prompts for your LLM? #LLM

LLM inference concepts (2/n): Vision language models are different from LLM in that they have an image encoder to capture and represent image/video information as tokens. We can take that encoder and serve it on separate workers for multimodal serving. Multimodal inference has three different phases: 1. Encoder   > This is where images are processed (e.g. ViT). It’s a one-shot, compute-bound stage with high variance. 2. Prefill   > Text + embeddings are loaded into the model. Memory-bandwidth heavy, large matrix multiplications. 3. Decode   > Tokens are generated one at a time. Long-lived, real-time, and memory-bound. Similar to my post yesterday many VLM systems currently run encoder + prefill + decode on the same GPUs. This mean that: > Encoder work blocks text requests → jittery latency > Text-only requests wait behind image jobs > Compute-bound encoder and memory-bound decode fight for the same hardware > You must scale all GPUs for rare multimodal spikes Disaggregating encoder inference from the other parts fixes this. By separating the visual encoder into its own service on separate GPUs: > Encoder runs in parallel with prefill/decode for other requests > Text-only requests completely bypass the encoder > The system becomes pipeline-parallel instead of serial This unlocks: > smoother latency > higher throughput > independent scaling per stage It also enables encoder output caching: > Common images are encoded once and reused > Cache hits skip the encoder entirely > TTFT drops and encoder load shrinks over time This nice post from vLLM dives into the implementation details 👇 - figure is from the LLaVa paper.

Multimodal interfaces open a new stage in human–technology communication, where voice, touch, and gestures converge into a coherent and adaptive experience that makes digital interaction more natural and expressive. When users can speak, touch, or gesture simultaneously, the interface begins to mirror the complexity of real human communication. This shift requires advanced systems capable of processing diverse inputs in real time, but it also brings a level of fluidity that enhances accessibility and inclusion. Recent studies show that multimodal systems can increase user engagement and reduce interaction errors by over 30 percent in some contexts. These results suggest that the effort to design adaptive, context-aware systems has tangible value for both efficiency and user satisfaction. I believe that bridging human and computer languages through multimodal interaction will open new opportunities for creativity and collaboration. Technology should learn to speak our language, not the other way around. #DigitalTransformation #UserExperience #AI #Innovation

Multimodal Machine Learning (MML) is a field of AI that focuses on integrating and understanding information from multiple modalities, such as text, images, audio, video, and sensor data. The concept has been around for decades, however, modern MML began gaining prominence in the early 2000s, driven by advancements in Machine-/Deep-Learning over the last decade or so and the availability of large, multimodal datasets. For readers familiar with Multi-View Learning (MVL), MML is conceptually similar. Both fields (MML and MVL) involve: • working with multiple types of inputs to improve learning. In MVL, these inputs are called "views," while in MML, they are "modalities". • focus on integrating or aligning information from different data sources to enhance performance on tasks like classification, regression, or clustering. • leverage complementary information from different data sources to improve predictive accuracy, robustness, and generalization. The MML domain has significantly enhances medical informatics by integrating diverse data types (e.g., clinical notes, imaging, genomic data, and sensor readings), offering benefits over traditional unimodal machine learning, which typically analyzes a single data type. MML combines diverse data (e.g., MRI scans with patient history), providing a comprehensive view of patient health that aids in better diagnosis and treatment planning. By leveraging complementary information from multiple modalities, MML also improves predictive accuracy for tasks like disease detection and prognosis. MML-powered tools streamline clinical workflows, such as radiology reporting or real-time decision-making during surgeries. Combining electronic health records (EHR) with medical images enhances diagnosis and prognosis by improving the accuracy of machine learning models in clinical prediction. The asynchronous and complementary nature of EHR and medical images presents unique challenges. Missing modalities due to clinical and administrative factors are inevitable in practice, and the importance of each data modality varies depending on the patient and the prediction target, leading to inconsistent predictions and suboptimal model performance. To address these challenges, the authors of [1] propose an MML workflow, DrFuse, to enable effective clinical multimodal fusion. It addresses the issue of missing modalities by disentangling features shared across modalities from those unique to each modality. Additionally, DrFuse tackles modal inconsistency through a disease-wise attention layer that generates patient- and disease-specific weightings for each modality to make the final prediction. #MedicalInformatics They validate the proposed DrFuse method using real-world large-scale datasets, MIMIC-IV and MIMIC-CXR. Experimental results show that the proposed method significantly outperforms the state-of-the-art models. The links to the preprint [1] and #Python GitHub repo [2] are shared in the first comment.

Exciting New Research on Multimodal Retrieval-Augmented Generation (MRAG)! I just finished reading a fascinating survey paper on Multimodal Retrieval-Augmented Generation (MRAG) from researchers at Huawei Cloud. This cutting-edge technology represents a significant advancement in enhancing large language models by integrating multimodal data like text, images, and videos into both retrieval and generation processes. Traditional Retrieval-Augmented Generation (RAG) systems primarily rely on textual data, which limits their ability to leverage rich contextual information available in multimodal sources. MRAG addresses this limitation by extending the RAG framework to include multimodal retrieval and generation, enabling more comprehensive and contextually relevant responses. The paper outlines the evolution of MRAG through three distinct stages: >> MRAG1.0 ("Pseudo-MRAG") This initial stage extended RAG by converting multimodal data into textual representations. The architecture consisted of three key components: - Document Parsing and Indexing: Processing multimodal documents using OCR and specialized models to generate captions for images and videos - Retrieval: Using vector embeddings to find relevant information - Generation: Synthesizing responses using LLMs While effective, this approach suffered from information loss during modality conversion and retrieval bottlenecks. >> MRAG2.0 ("True Multimodal") This stage preserved original multimodal data within the knowledge base and leveraged Multimodal Large Language Models (MLLMs) for direct processing. Key improvements included: - Using unified MLLMs for captioning instead of separate models - Supporting cross-modal retrieval to minimize data loss - Employing MLLMs for generation to directly process multimodal inputs >> MRAG3.0 (Advanced Integration) The latest evolution introduces: - Enhanced document parsing that retains document screenshots to minimize information loss - Multimodal Search Planning that optimizes retrieval strategies through retrieval classification and query reformulation - Multimodal output capabilities that combine text with images, videos, or other modalities in responses The technical architecture includes sophisticated components like multimodal retrievers (using single/dual-stream and generative structures), rerankers (fine-tuning or prompting-based), and refiners (hard or soft prompt methods) to optimize the information flow. What's particularly impressive is how MRAG outperforms traditional text-modal RAG in scenarios where both visual and textual information are critical for understanding and responding to queries. The researchers have systematically analyzed essential components, datasets, evaluation methods, and current limitations to provide a comprehensive understanding of this promising paradigm.

Wisdom & Workflow Wednesday: The Future Is Multimodal Radiology has always been about images. But the future of precision medicine will not come from images alone. The next frontier is multimodal integration - where imaging is combined with pathology, genomics, clinical data, and even digital health streams to create a richer, more complete picture of the patient. Why does this matter? A CT scan can show a tumor, but genomics can reveal its mutations. An MRI can map anatomy, but clinical data tells us how symptoms evolve. Pathology defines the diagnosis, while imaging tracks the response. #AI has the potential to bring these worlds together: Integrating multiple data types into one coherent view Supporting clinicians in making earlier, more personalized decisions Driving a true “digital twin” of the patient, where therapy can be predicted and optimized I see this as both an opportunity and a leadership challenge. Building multimodal systems is not about technology alone, it’s about breaking silos, aligning stakeholders, and ensuring data serves patients safely and effectively. The future of imaging will be less about single modalities and more about symphonies of data. And radiology, as the discipline that already interprets the invisible, is perfectly positioned to lead. What’s your view: will multimodal AI arrive first in oncology, cardiology, or another field? #WorkflowWednesday #Radiology #MultimodalAI #Leadership #GEHealthcare

New research in JACC: Advances shows that the eye may offer a powerful, noninvasive window into coronary artery disease detection. In a multicenter study of 383 patients, deep learning models trained on retinal images were able to identify CAD with strong performance, outperforming traditional clinical risk scores, particularly in intermediate risk patients where clinical uncertainty is highest. When retinal imaging was combined with clinical indicators using a multimodal AI approach, diagnostic accuracy improved further, achieving an AUC of 0.91 with over 92 percent sensitivity. Because retinal and coronary vessels share similar vascular origins, microvascular changes captured by OCT and OCTA appear to reflect underlying coronary disease. AI enables these subtle patterns to be translated into scalable, radiation free screening and risk stratification tools. This work points toward a future where cardiovascular risk can be assessed earlier, more safely, and more equitably, especially in settings where invasive testing is limited. Multimodal AI may be key to shifting CAD detection upstream and personalizing prevention before clinical events occur. 🔗 https://lnkd.in/gWJUU447 Follow Zain Khalpey, MD, PhD, FACS for more on Ai & Healthcare. #AIinHealthcare #Cardiology #CoronaryArteryDisease #PreventiveCardiology #DigitalHealth #MedicalAI #MultimodalAI #DeepLearning #NonInvasiveDiagnostics #RetinalImaging #OCTA #OCT #CardiovascularHealth #RiskStratification #PrecisionMedicine #ClinicalInnovation #HealthEquity #CVImaging

Multimodal AI is shaping a shift in healthcare by combining different kinds of patient data to improve care across diagnostics, treatment, and monitoring. 1️⃣ It links data from imaging, wearables, clinical notes, genomics, and more to create a fuller picture of patient health. 2️⃣ Imaging, physiological signals, and clinical notes are the most commonly used data types, especially in oncology, cardiovascular, and neurological disorders. 3️⃣ Intermediate fusion is the most used integration method, combining data at the feature level for better balance between complexity and interpretability. 4️⃣ These systems enable early diagnosis, prognosis, treatment planning, and real-time monitoring, with growing applications in areas like digital twins and automated reporting. 5️⃣ Personalized medicine is a major driver, with multimodal models supporting tailored treatment decisions by analyzing combined molecular, physiological, and behavioral data. 6️⃣ Despite progress, challenges remain: data heterogeneity, privacy concerns, lack of benchmarks, and regulatory constraints slow adoption. 7️⃣ Explainability is key for clinical trust. Emerging models include attention maps, concept attribution, and human-in-the-loop feedback for better transparency. 8️⃣ Energy demands of training large models have sparked interest in "green AI", focusing on efficiency and scalability in clinical settings. 9️⃣ Future systems may rely more on self-supervised and federated learning to handle data gaps and maintain privacy across institutions. 🔟 Clinical validation and regulatory reform are needed for multimodal systems to move from labs into widespread practice. ✍🏻 Florenc Demrozi, Mina Farmanbar, Kjersti Engan. Multimodal AI for Next-Generation Healthcare: Data Domains, Algorithms, Challenges, and Future Perspectives. Current Opinion in Biomedical Engineering. 2025. DOI: 10.1016/j.cobme.2025.100632 (pre-proof)