Engineering Workflow Management Systems

When working with multiple LLM providers, managing prompts, and handling complex data flows — structure isn't a luxury, it's a necessity. A well-organized architecture enables: → Collaboration between ML engineers and developers → Rapid experimentation with reproducibility → Consistent error handling, rate limiting, and logging → Clear separation of configuration (YAML) and logic (code) 𝗞𝗲𝘆 𝗖𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀 𝗧𝗵𝗮𝘁 𝗗𝗿𝗶𝘃𝗲 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 It’s not just about folder layout — it’s how components interact and scale together: → Centralized configuration using YAML files → A dedicated prompt engineering module with templates and few-shot examples → Properly sandboxed model clients with standardized interfaces → Utilities for caching, observability, and structured logging → Modular handlers for managing API calls and workflows This setup can save teams countless hours in debugging, onboarding, and scaling real-world GenAI systems — whether you're building RAG pipelines, fine-tuning models, or developing agent-based architectures. → What’s your go-to project structure when working with LLMs or Generative AI systems? Let’s share ideas and learn from each other.

You learn a lot about an organisation from the way it monitors its estate. When monitoring platforms have been patched together over the years, they eventually stop giving teams confidence in what they're seeing. - Duplicate alerts become normal. - Blind spots are accepted. - Engineers spend more time validating incidents than resolving them. That's an operational problem, not a monitoring one. On one programme, we inherited an environment where the monitoring estate had grown faster than the operating model around it. Rebuilding the platform wasn't the first job. Understanding how teams actually worked was. 1. Which alerts mattered? 2. Which devices genuinely needed visibility? 3. Where were engineers losing time every day? Only then did we start modernising the platform. Automation replaced manual onboarding. Policies became consistent. Runbooks were written before the cutover, not afterwards. Every change was introduced in phases because continuity mattered more than speed. The result wasn't simply a newer monitoring platform. It was an operational model that gave teams confidence in the data they were working from. More than 12,000 devices were brought into a modernised monitoring estate. Around 400 devices could be onboarded each day through automation. The transition completed without downtime, while the in-house team took ownership of the service. Technology programmes often focus on the platform that's being delivered. The bigger challenge is designing the operational discipline that allows that platform to keep delivering long after the project has finished. Where have you seen operational processes make a bigger difference than the technology itself? #GovTech #DigitalTransformation #ITOperations

𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 𝗔𝗜 𝗜𝘀 𝗮 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 Much of today’s conversation around AI agents focuses on #graphs, #models, #prompts, #context, or orchestration #frameworks. These topics matter, but they rarely determine whether an AI system succeeds once it moves from prototype to enterprise production. The real challenges appear when AI systems operate inside long-running business workflows. Consider a workflow that analyzes documents, retrieves data from multiple systems, calls APIs, and produces a structured decision. Such processes may run for twenty or thirty minutes and involve dozens of steps. Now imagine something routine happens: a network call fails, an API times out, or a container restarts. No problem, the agent says. It starts the workflow again. That may be acceptable for chatbots. It quickly becomes impractical for enterprise processes such as financial analysis, document processing, underwriting, or claims review. These workflows are long-running, resource-intensive, and deeply connected to operational systems. In these situations, the limitation is rarely the model’s intelligence. More often, the challenge lies in the #engineering #discipline around the system. At Cognida.ai, our focus is on building practical enterprise AI systems rather than demos or PoCs. We consistently find that several principles from #distributedsystems engineering become essential once AI moves into production. Here are three such constructs: 𝗗𝘂𝗿𝗮𝗯𝗹𝗲 𝗘𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 Agent workflows should not be treated as temporary requests. Each step should persist its state so that if a failure occurs, the system can resume from the last successful step rather than restarting the entire process. In practice, this means workflow orchestration with checkpointed state, deterministic execution, and event-driven recovery. For long-running processes, this is often the difference between a prototype and a production system. 𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝘁 𝗔𝗰𝘁𝗶𝗼𝗻𝘀 AI agents increasingly trigger real-world actions: sending emails, calling APIs, updating records, moving files, or initiating financial transactions. Retries are inevitable in distributed systems. If actions are not idempotent, retries can create duplicate or inconsistent results. Reliable AI systems must ensure the same action cannot run twice unintentionally. 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗲𝗻𝘁 𝗦𝘁𝗮𝘁𝗲 𝗕𝗲𝘆𝗼𝗻𝗱 𝘁𝗵𝗲 𝗠𝗼𝗱𝗲𝗹 Large language models operate within limited context windows rather than durable memory. Enterprise workflows often run longer and across many stages. The system managing the workflow must maintain its own persistent state instead of relying on the model’s temporary context. It means treating AI workflows as structured state machines, not simple prompt-response interactions. Are you treating AI workflows more like state machines, event-driven systems, or traditional #microservices? #PracticalAI #EnterpriseAI

Small variations in prompts can lead to very different LLM responses. Research that measures LLM prompt sensitivity uncovers what matters, and the strategies to get the best outcomes. A new framework for prompt sensitivity, ProSA, shows that response robustness increases with factors including higher model confidence, few-shot examples, and larger model size. Some strategies you should consider given these findings: 💡 Understand Prompt Sensitivity and Test Variability: LLMs can produce different responses with minor rephrasings of the same prompt. Testing multiple prompt versions is essential, as even small wording adjustments can significantly impact the outcome. Organizations may benefit from creating a library of proven prompts, noting which styles perform best for different types of queries. 🧩 Integrate Few-Shot Examples for Consistency: Including few-shot examples (demonstrative samples within prompts) enhances the stability of responses, especially in larger models. For complex or high-priority tasks, adding a few-shot structure can reduce prompt sensitivity. Standardizing few-shot examples in key prompts across the organization helps ensure consistent output. 🧠 Match Prompt Style to Task Complexity: Different tasks benefit from different prompt strategies. Knowledge-based tasks like basic Q&A are generally less sensitive to prompt variations than complex, reasoning-heavy tasks, such as coding or creative requests. For these complex tasks, using structured, example-rich prompts can improve response reliability. 📈 Use Decoding Confidence as a Quality Check: High decoding confidence—the model’s level of certainty in its responses—indicates robustness against prompt variations. Organizations can track confidence scores to flag low-confidence responses and identify prompts that might need adjustment, enhancing the overall quality of outputs. 📜 Standardize Prompt Templates for Reliability: Simple, standardized templates reduce prompt sensitivity across users and tasks. For frequent or critical applications, well-designed, straightforward prompt templates minimize variability in responses. Organizations should consider a “best-practices” prompt set that can be shared across teams to ensure reliable outcomes. 🔄 Regularly Review and Optimize Prompts: As LLMs evolve, so may prompt performance. Routine prompt evaluations help organizations adapt to model changes and maintain high-quality, reliable responses over time. Regularly revisiting and refining key prompts ensures they stay aligned with the latest LLM behavior. Link to paper in comments.

"𝐖𝐡𝐲 𝐢𝐬 𝐦𝐲 𝐋𝐋𝐌 𝐠𝐢𝐯𝐢𝐧𝐠 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐚𝐧𝐬𝐰𝐞𝐫𝐬 𝐭𝐨 𝐭𝐡𝐞 𝐬𝐚𝐦𝐞 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧?"  If you have asked this in the last month, here is your Debugging Playbook. Most teams treat inconsistent LLM outputs as a Model Problem.  It is almost never the Model.  It is your System Architecture exposing variability you did not know existed. After debugging 40+ production AI systems, I have developed a 6-Step Framework that isolates the real culprit: Step 1: Confirm the Inconsistency Is Real • Compare responses across identical prompts • Control temperature, top-p, and randomness • Check prompt versions and hidden changes • Goal: Rule out noise before debugging the system Step 2: Break the Output into System Drivers • Decompose your response pipeline into components • Prompt structure, retrieved context (RAG), tool calls, model version, system instructions • Use a "dropped metric" approach to test each driver independently • Goal: Identify where variability can be introduced Step 3: Analyze Variability per Driver • Inspect each driver independently for instability • Does retrieval return different chunks? Are tool outputs non-deterministic? Are prompts dynamically constructed? • Test drivers across same period vs previous period • Goal: Isolate the component causing divergence Step 4: Segment by Execution Conditions • Slice outputs by environment or context • User input variants, model updates/routing, time-based data changes, token limits or truncation • Look for patterns in when inconsistency spikes • Goal: Find conditions where inconsistency spikes Step 5: Compare Stable vs Unstable Runs • Contrast successful outputs with failing ones • Same prompt/different output, same context/different reasoning, same goal/different execution • Surface the exact difference that matters • Goal: Surface the exact difference that matters Step 6: Form and Test Hypotheses • Turn findings into testable explanations • Hypothesis: retrieval drift, prompt ambiguity, tool response variance • Move from suspicion to proof • Goal: Move from suspicion to proof The pattern I see repeatedly: Teams jump straight to "let's try a different model" or "let's add more examples." But inconsistent outputs are rarely a model issue-they are usually a system issue. • Your retrieval is pulling different documents.  • Your tool is returning non-deterministic results.  • Your prompt is being constructed differently based on context length. The 6-step framework forces you to treat LLM systems like the distributed systems they actually are. Which step do most teams skip? Step 1. They assume inconsistency without proving it. Control your variables first. ♻️ Repost this to help your network get started ➕ Follow Anurag(Anu) Karuparti for more PS: If you found this valuable, join my weekly newsletter where I document the real-world journey of AI transformation. ✉️ Free subscription: https://lnkd.in/exc4upeq #GenAI #AIAgents

It shouldn’t surprise people that LLMs are not fully deterministic, they can’t be. Even when you set temperature to zero, fix the seed, and send the exact same prompt, you can still get different outputs in production. There’s a common misconception that nondeterminism in LLMs comes only from sampling strategies. In reality, part of the variability comes from how inference is engineered at scale. In production systems, requests are often batched together to optimize throughput and cost. Depending on traffic patterns, your prompt may be grouped differently at different times. That changes how certain low-level numerical operations are executed on hardware. And because floating-point arithmetic is not perfectly associative, tiny numerical differences can accumulate and lead to different token choices. The model weights haven’t changed, neither has the prompt. But the serving context has. Enterprise teams often evaluate models assuming reproducibility is guaranteed if parameters are fixed. But reliability in LLM systems is not only a modeling problem. It is a systems engineering problem. You can push toward stricter determinism. But doing so may require architectural trade-offs in latency, cost, or scaling flexibility. The point is not that LLMs are unreliable, but that nondeterminism is part of the stack. If you are deploying AI in production, you need to understand where it enters, and design your evaluation, monitoring, and governance around it.

I just wrapped a deep-dive conversation with Dave Marcus evaluating Replit (and adjacent “idea-to-prototype” platforms) through a very practical test: he was building a mini-CRM with human workflow approvals and a Microsoft SharePoint integration. Here are the key takeaways - especially for anyone considering these tools for enterprise use cases: 1) These platforms are excellent at “idea → prototype.” Replit/Lovable/Base44 are winning on speed, UX, and approachability particularly for product, ops, and automation teams who want to stand something up quickly. 2) Prompting discipline matters more than people admit. A big portion of successful builds seem to come from structured “LLM-optimized” prompting (vs. ad hoc prompting). Plan/structured modes early can also help prevent context loss as you bounce between artifacts/tools. 3) Integration UX is still fragile. Even when integrations exist (e.g., SharePoint), the platform may not reliably “pick up” preconfigured connections creating placeholders that require manual correction. That’s a big adoption barrier for less technical builders. 4) Data modeling & dependency management are the biggest gaps. Compared to traditional low-code, there’s a noticeable absence of: true entity relationship modeling / ERDs business-friendly workflow diagrams (vs. verbose, technical outputs) guardrails when changing/deleting fields that are referenced across forms/workflows 5) Cross-app reuse of data is limited. You can’t easily share a canonical set of entities across multiple apps. The workaround looks like “build a database + expose APIs,” which is workable—but not the seamless “virtual entity layer” many enterprise teams expect. 6) “Production readiness” usually means leaving the built-in database behind The built-in DB experience can feel closer to lightweight tooling (think Access-level). For real production requirements; security, scale, rollback/transactions, protections against accidental deletion the pattern becomes: move data to AWS/Azure and treat the platform as the UX layer. 7) Env/secrets management exists, but enterprise configuration patterns are immature. It’s not yet straightforward to define dev/test/prod configurations cleanly across services (email, auth providers, etc.) in a way that enterprise teams are used to. To summarize: If your goal is rapid UX prototyping and getting to a working demomvp fast, these tools are strong. If your goal is a durable enterprise application crossing multiple systems with strong governance, you’ll need to create architecture patterns (and guardrails) that many of these platforms don’t fully provide yet. Curious how others are drawing the line between “prototype platforms” and “production platforms” right now especially where governance, data modeling, and multi-app reuse are requirements. #vibecoding #ai #production #lovable #base44 #claudecode #codex #cursor

If your engineers are spending more time managing paperwork than managing processes, that is not a productivity problem. That is a workflow problem. And workflow problems have solutions. I recently met an engineer, his name is Faizal. A Senior process engineer. Twelve years with the company. The kind of guy who knew every machine on the floor by sound alone. Every time something changed on the production line, Faizal was the one who caught it. A valve adjustment. A pressure tweak. A sequence that needed updating because a vendor changed their specs last minute. He would scribble it down in his notebook first. Then type it up. Then send it to his supervisor for sign-off. Then the supervisor would bring it to the weekly ops meeting. The meeting would run long because someone always had questions. Someone else was not in the room. They would schedule a follow-up. Meanwhile, Faizal’s change note sat in an email thread, buried under thirty replies. By the time the paper form made its way through department heads, got printed, signed, scanned, and filed into a cabinet on level three, two weeks had passed. Sometimes more. And sometimes, the form never came back at all. He once spent forty minutes searching for an approved change note from eight months ago. A compliance audit was coming. The document existed. He had seen it. He just could not find it. On day, he told his manager: “I spend more time chasing paper than I do solving actual problems.” His manager nodded. He had heard this before. From other engineers. In other departments. Across different sites. The problem was not Faizal. The problem was a process built for a time when paper was the only option. When Faizal’s company moved their change management workflow into DocuWare, a few things happened almost immediately. Change notes were submitted digitally, with version control. Approvals happened through automated routing, no more chasing signatures across floors. Every document was timestamped, traceable, and retrievable in seconds. And the audit that used to take days of frantic searching, took under an hour. Faizal still carries his notebook. Old habits. But now it is just for his own thinking. Everything else has a proper home. What does your change management process look like today?

The challenge of integrating multiple large language models (LLMs) in enterprise AI isn’t just about picking the best model, it’s about choosing the right mix for each specific scenario. When I was tasked with leveraging Azure AI Foundry alongside Microsoft 365 Copilot, Copilot Studio, Claude Sonnet 4, and Opus 4.1 to enhance workflows, the advice I heard was to double down on a single, well‑tuned model for simplicity. In our environment, that approach started to break down at scale. Model pluralism turned out to be the unexpected solution, using multiple LLMs in parallel, each optimised for different tasks. The complexity was daunting at first, from integration overhead to security and governance concerns. But this approach let us tighten data grounding and security in ways a single model couldn’t. For example, routing the most sensitive tasks to Opus 4.1 helped us measurably reduce security exposure in our internal monitoring, while Claude Sonnet 4 noticeably improved the speed and quality of customer‑facing interactions. In practice, the chain looked like this: we integrated multiple LLMs, mapped each one to the tasks it handled best, and saw faster execution on specialised workloads, fewer security and compliance issues, and a clear uplift in overall workflow effectiveness. Just as importantly, the architecture became more robust, if one model degraded or failed, the others could pick up the slack, which matters in a high‑stakes enterprise environment. The lesson? The “obvious” choice, standardising on a single model for simplicity, can overlook critical realities like security, governance, and scalability. Model pluralism gave us the flexibility and resilience we needed once we moved beyond small pilots into real enterprise scale. For those leading enterprise AI initiatives, how are you balancing the trade‑off between operational simplicity and a pluralistic, multi‑model architecture? What does your current model mix look like?

𝗜𝘀 𝘆𝗼𝘂𝗿 𝗟𝗟𝗠 𝘁𝗿𝘆𝗶𝗻𝗴 𝗵𝗮𝗿𝗱 𝘁𝗼 𝗮𝗴𝗿𝗲𝗲 𝘄𝗶𝘁𝗵 𝘆𝗼𝘂? 𝗜𝗳 𝘆𝗼𝘂𝗿 𝗺𝗼𝗱𝗲𝗹 𝗻𝗲𝘃𝗲𝗿 𝗱𝗶𝘀𝗮𝗴𝗿𝗲𝗲𝘀 𝘄𝗶𝘁𝗵 𝘆𝗼𝘂, 𝗶𝘁’𝘀 𝗻𝗼𝘁 𝗮 𝗰𝗼𝗽𝗶𝗹𝗼𝘁. 𝗜𝘁’𝘀 𝗮 𝗺𝗶𝗿𝗿𝗼𝗿. If you’ve ever thought, “This model really gets me,” there’s a good chance it’s actually just trying not to upset you. Modern LLMs are heavily trained on being helpful, harmless and be aligned to user intent. But in practice, that can morph into something dangerous in enterprise settings: 𝘀𝘆𝗻𝘁𝗵𝗲𝘁𝗶𝗰 𝘆𝗲𝘀-𝗺𝗲𝗻. Have you experienced this?  • You suggest a solution, and the model “refines” it instead of challenging it.  • You embed a wrong assumption in your prompt, and it builds a beautiful, confident answer on top of it.  • You ask for a comparison, and it over-weights your preferred option because your prompt sounded more enthusiastic about it. In other words, Your LLM isn’t always optimizing for truth. It’s optimizing for agreement + politeness. If you’re building serious AI products, especially agents, you need 𝗱𝗲𝘀𝗶𝗴𝗻𝗲𝗱 𝗱𝗶𝘀𝗮𝗴𝗿𝗲𝗲𝗺𝗲𝗻𝘁. 1. 𝗖𝗼𝘂𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝘁𝘂𝗮𝗹 𝗽𝗿𝗼𝗺𝗽𝘁𝘀: Force the model to argue the opposite. E.g.: List 3 reasons why this approach might fail. 2. 𝗠𝘂𝗹𝘁𝗶-𝗽𝗮𝘀𝘀 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴: Ask the model critique its own answer and highlight weak assumptions 3. 𝗥𝗼𝗹𝗲 𝘀𝗲𝗽𝗮𝗿𝗮𝘁𝗶𝗼𝗻: Use different “personas” in your system. Use an explorer to generate options and a devil's advocate to question assumptions, check data and test edge cases 4. 𝗛𝗲𝗹𝗽𝗳𝘂𝗹 != 𝗔𝗴𝗿𝗲𝗲𝗮𝗯𝗹𝗲 : Make it explicit in your system messages: “Your job is not to agree with the user. Your job is to help them be right.”