Engineering Quality Assurance Methods

Last week, I described four design patterns for AI agentic workflows that I believe will drive significant progress: Reflection, Tool use, Planning and Multi-agent collaboration. Instead of having an LLM generate its final output directly, an agentic workflow prompts the LLM multiple times, giving it opportunities to build step by step to higher-quality output. Here, I'd like to discuss Reflection. It's relatively quick to implement, and I've seen it lead to surprising performance gains. You may have had the experience of prompting ChatGPT/Claude/Gemini, receiving unsatisfactory output, delivering critical feedback to help the LLM improve its response, and then getting a better response. What if you automate the step of delivering critical feedback, so the model automatically criticizes its own output and improves its response? This is the crux of Reflection. Take the task of asking an LLM to write code. We can prompt it to generate the desired code directly to carry out some task X. Then, we can prompt it to reflect on its own output, perhaps as follows: Here’s code intended for task X: [previously generated code] Check the code carefully for correctness, style, and efficiency, and give constructive criticism for how to improve it. Sometimes this causes the LLM to spot problems and come up with constructive suggestions. Next, we can prompt the LLM with context including (i) the previously generated code and (ii) the constructive feedback, and ask it to use the feedback to rewrite the code. This can lead to a better response. Repeating the criticism/rewrite process might yield further improvements. This self-reflection process allows the LLM to spot gaps and improve its output on a variety of tasks including producing code, writing text, and answering questions. And we can go beyond self-reflection by giving the LLM tools that help evaluate its output; for example, running its code through a few unit tests to check whether it generates correct results on test cases or searching the web to double-check text output. Then it can reflect on any errors it found and come up with ideas for improvement. Further, we can implement Reflection using a multi-agent framework. I've found it convenient to create two agents, one prompted to generate good outputs and the other prompted to give constructive criticism of the first agent's output. The resulting discussion between the two agents leads to improved responses. Reflection is a relatively basic type of agentic workflow, but I've been delighted by how much it improved my applications’ results. If you’re interested in learning more about reflection, I recommend: - Self-Refine: Iterative Refinement with Self-Feedback, by Madaan et al. (2023) - Reflexion: Language Agents with Verbal Reinforcement Learning, by Shinn et al. (2023) - CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, by Gou et al. (2024) [Original text: https://lnkd.in/g4bTuWtU ]

How to Write UAT Test Cases (for Business Analysts) I remember when I was a junior BA - the idea of UAT scared me. I didn’t want to admit I didn’t know where to start... So I stayed quiet and tried to figure it out on my own. Turns out, I’d built it up to be more complicated than it really is. Here’s what I learned: As a BA, your role in User Acceptance Testing (UAT) is to ensure the solution actually meets the business need… not just that it functions. To do that effectively, you need a structured approach to writing UAT test cases. Here's how I do it: 1️⃣ Start with the Requirement → Begin with a single requirement or user story. → Each requirement must be tested, and depending on how many acceptance criteria it has, you may need multiple test cases. (Think: What is the business expecting from this requirement?) 2️⃣ Review the Acceptance Criteria → Acceptance criteria define the boundary of success for a requirement. → They help you understand what “good” looks like from the business’s perspective. (Use these criteria as your guideposts for what to test) 3️⃣ Develop Test Cases Based on the Acceptance Criteria → Each acceptance criterion should translate into at least one test case. → Some may need both a positive (happy path) and negative (error or edge case) scenario. (If a criterion says “User must receive a confirmation email,” test both a valid scenario and one where the email fails) 4️⃣ Complete the UAT Template for Each Test Case → For each test case, fill in these fields: ☑ Test Description – A clear statement of what’s being tested e.g. “Test password reset email is triggered for valid email addresses” ☑ Preconditions – Any setup required before testing e.g. “User is logged out and on the login page” ☑ Test Steps – Step-by-step actions for the tester to perform e.g. Click “Forgot Password”, enter email, submit form ☑ Expected Result – What should happen if the system works correctly e.g. “User receives reset email within 2 minutes” (TIP: Keep the language business-friendly so anyone can run the test) 5️⃣ Repeat for Each Requirement → Once you've completed the test cases for one requirement, move to the next and repeat the process. → This ensures full coverage and traceability back to each business objective. 6️⃣ Review with Business Stakeholders → Once your test cases are drafted, share them with your business SMEs or stakeholders. (This step is critical - their feedback confirms that you’re testing what really matters to them) 7️⃣ Prepare for Execution → After validation, the test cases are ready to be run. → Depending on your project, UAT may be carried out by business users, or you may help execute or facilitate it as a BA. 📩 Want a copy of my UAT test case template? → Send me a message and I’ll be happy to share it with you 😊 Found this interesting? Repost to your network, and follow me → Matthew Thomas Holliday #BusinessAnalysis #UAT #BAskills #BAmethods #UATtemplate

I would like to introduce some useful things for solar panel Testing: ⚡ Solar Panel Testing: What We Check Before Procurement & Installation Before any solar panel hits the field, rigorous testing is essential. Here's a detailed breakdown of the key tests and standards we perform to ensure top-tier quality, performance, and long-term reliability. ✅ 1. Flash Test (I-V Curve under STC) 📌 Purpose: Measures actual electrical performance under Standard Test Conditions (STC) 📊 STC Parameters: 1000 W/m² irradiance 25°C cell temperature Air Mass 1.5 🔍 Key Checks: Pmax (Maximum Power): Must be within ±3% of rated capacity Voc (Open Circuit Voltage) & Isc (Short Circuit Current): Should show tight consistency between modules 💡 Why it matters: Verifies that real output matches the manufacturer’s datasheet—no surprises after installation. ✅ 2. NOCT – Nominal Operating Cell Temperature 📌 Purpose: Predicts real-world performance under actual outdoor conditions 📊 Typical Conditions: 800 W/m² irradiance 20°C ambient temp 1 m/s wind speed 🎯 Ideal Range: 42°C – 48°C 💡 Why it matters: Lower NOCT = less heat = better energy yield in the field. ✅ 3. Electroluminescence (EL) Imaging 📌 Purpose: Reveals hidden cell-level defects 🔬 Method: Apply low voltage in darkness to produce infrared emission 🔍 Detects: Microcracks Broken cells Soldering faults 💡 Why it matters: Early detection prevents hotspots, power loss, and premature failure. ✅ 4. Insulation Resistance & High-Voltage Withstand Test 📌 Purpose: Ensures electrical safety and system durability 📊 Test Voltage: 1000–1500V DC, depending on system design 🎯 Minimum Resistance: >40 MΩ at 1000V (per IEC 61730) 💡 Why it matters: Critical for shock prevention, fire safety, and long-term reliability. ✅ 5. PID (Potential Induced Degradation) Test 📌 Purpose: Assesses vulnerability to voltage-induced performance loss 📊 Test Conditions: ~85°C 85% RH -1000V applied for 96–168 hours 🎯 Degradation Threshold: <5% power loss 💡 Why it matters: Vital for high-voltage and humid-climate installations. ✅ 6. QAP (Quality Assurance Plan) Review 📌 Purpose: Evaluates the manufacturer’s internal QA processes 📝 What We Verify: ISO Certifications (e.g., ISO 9001) Recent factory audits Random sampling results (IEC 61215 / 61730) Raw material traceability 💡 Why it matters: Adds confidence beyond lab tests—ensures production consistency and traceability. ✅ 7. Thermal Cycling & Damp Heat Test 📌 Standard: IEC 61215 📊 Test Parameters: Thermal Cycling: 200 cycles from -40°C to +85°C Damp Heat: 1000 hours at 85°C / 85% RH 🎯 Acceptable Loss: <5% degradation 💡 Why it matters: Demonstrates durability in extreme environments (deserts, tropics, snow zones). ✅ 8. Visual Inspection 📌 What We Check: Glass cracks Delamination Frame warping Junction box damage Edge sealing & backsheet integrity 💡 Why it matters: Catching cosmetic or structural issues early prevents installation delays and long-term performance risks.

Here is how you can test your applications using an LLM: We call this "LLM as a Judge", and it's much easier to implement than most people think. Here is how to do it: (LLM-as-a-judge is one of the topics I teach in my cohort. The next iteration starts in August. You can join at ml.school.) We want to use an LLM to test the quality of responses from an application. There are 3 scenarios in one of the attached pictures: 1. Choose the best of two responses 2. Assess specific qualities of a response 3. Evaluate the response based on additional context I'm also attaching three example prompts to test each of the scenarios. These prompts are a big part of a successful judge, and you'll spend most of your time iterating on these prompts. Here is the process to create a judge: 1. Start with a labeled dataset 2. Design your evaluation prompt 3. Test it on the dataset 4. Iteratively refine it until you are happy with it Evaluating an answer is usually easier than producing that answer in the first place, so you can use a smaller/cheaper model to build the judge than the one you are evaluating. But you can also use the same model, or even a stronger model than the one you are evaluating. My recommendation: Build the judge using the same model your application uses. When you have the judge working as intended, replace it with a smaller or cheaper model and see if you can achieve the same performance. Repeat until satisfied. When your judge is ready, use it to evaluate a percentage of outputs to detect drift and track any trends over time. Advantages: • Produces high-quality evaluations closely matching human judgment • Simple to set up. Don’t need reference answers • Flexible. You can evaluate anything • Scalable. Can handle multiple evaluations very fast • Easy to adjust as criteria change Disadvantages: • Probabilistic - different prompts can lead to different outputs • May suffer from self-bias, first-position, or verbosity bias • May introduce privacy risks • Slower/more expensive than rule-based evaluations • Requires effort to prepare and run Final tip: Do not use opaque judges (pre-built judges that you can't see how they work). Any changes in the judge’s model or prompt will change its results. If you can’t see how the judge works, you can’t interpret its results.

"Quality starts before code exists", This is how AI can be used to reimagine the Testing workflow Most teams start testing after the build. But using AI, we can start it in design phase Stage - 1: WHAT: Interactions, font-size, contrast, accessibility checks etc. can be validated using GPT-4o / Claude / Gemini (LLM design review prompts) - WAVE (accessibility validation) How we use them: Design files → exported automatically → checked by accessibility scanners → run through LLM agents to evaluate interaction states, spacing, labels, copy clarity, and UX risks. Stage - 2: Tools: • LLMs (GPT-4o / Claude 3.5 Sonnet) for requirement parsing • Figma API + OCR/vision models for flow extraction • GitHub Copilot for converting scenarios to code skeletons • TestRail / Zephyr for structured test storage How we use them: PRDs + user stories + Figma flows → AI generates: ✔ functional tests ✔ negative tests ✔ boundary cases ✔ data permutations SDETs then refine domain logic instead of writing from scratch. Stage - 3: Tools: • SonarQube + Semgrep (static checks) • LLM test reviewers (custom prompt agents) • GitHub PR integration How we use them: Every test case or automation file passes through: SonarQube: static rule checks LLM quality gate that flags: - missing assertions - incomplete edge coverage - ambiguous expected outcomes - inconsistent naming or structure We focus on strategy -> AI handles structural review. Stage - 4: Tools: • Playwright, WebDriver + REST Assured • GitHub Copilot for scaffold generation • OpenAPI/Swagger + AI for API test generation How we use them: Engineers describe intent → Copilot generates: ✔ Page objects / fixtures ✔ API client definitions ✔ Custom commands ✔ Assertion scaffolding SDETs optimise logic instead of writing boilerplate. THE RESULT - Test design time reduced 60% - Visual regressions detected with near-pixel accuracy - Review overhead for SDETs significantly reduced - AI hasn’t replaced SDETs. It removed mechanical work so humans can focus on: • investigation • creativity • user empathy • product risk understanding -x-x- Learn & Implement the fundamentals required to become a Full Stack SDET in 2026: https://lnkd.in/gcFkyxaK #japneetsachdeva

Hipot (High Potential) testing in the solar industry is a critical safety and quality assurance step used to verify the dielectric strength of insulation in PV modules, inverters, and cabling. If you are preparing for an interview, these "handwritten-style" notes cover the technical, safety, and procedural aspects likely to be discussed. 1. Fundamental Concepts * What is a Hipot Test? It is a stress test of the insulation. A high voltage (greater than operating voltage) is applied between the current-carrying conductors and the grounding (frame/earth) to ensure no leakage current flows through the insulation. * Purpose: To detect crushed insulation, stray wire strands, conductive contaminants, or spacing issues that could cause a short circuit or fire. * Hipot vs. Megger (Insulation Resistance): * Megger: Measures the resistance value (typically in M\Omega or G\Omega). * Hipot: Tests the integrity by applying high voltage and monitoring for a "breakdown" or excessive leakage current. 2. Test Procedures & Parameters * Standard Voltage Calculation: A common industry rule of thumb is 2 \times \text{Operating Voltage} + 1000V. For a 1500V solar string, the test might reach 4000V DC. * AC vs. DC Hipot: * DC Hipot: Preferred for solar because PV modules have high capacitance. DC avoids the "reactive current" that AC would trigger, allowing for a smaller, more portable tester. * AC Hipot: More aggressive; tests both polarities of the insulation. * Step-by-Step Execution: * Isolate: Disconnect the string from the inverter and combiner boxes. * Short: Short the Positive (+) and Negative (-) leads of the string together. * Apply: Connect the High Voltage lead to the shorted PV strings and the Return/Ground lead to the module frames/racking. * Ramp: Slowly increase voltage to the target level. * Dwell: Hold at target voltage (usually for 60 seconds). * Discharge: Safely bleed off the residual capacitive charge. 3. Interview "Red Flags" & Troubleshooting * What causes a "Fail"? * Nicked cable jackets during installation. * Moisture inside a junction box. * Improperly crimped MC4 connectors. * Arc Detection: If the tester shuts down immediately with an "Arc" error, it usually indicates a physical gap where electricity is jumping (e.g., a loose wire near a grounded rail). * Leakage Current: If the current slowly creeps up, it often indicates moisture or "dirty" insulation. 4. Safety & PPE (The most important part) * The "One Hand" Rule: When possible, keep one hand in your pocket to prevent a current path through the heart if an accidental shock occurs. * PPE: Class 0 or 00 Electrical Gloves (tested/certified), safety glasses, and arc-rated clothing. * Discharge: Always verify the "Zero Voltage" state with a voltmeter after the test, as PV strings act like giant capacitors and can hold a lethal charge. * Barriers: Use cones or "Danger: High Voltage" tape

One of the most misunderstood responsibilities of a Business Analyst is writing User Acceptance Testing (UAT) test cases. Many BAs either avoid it or get too technical. But remember—UAT is about validating that the solution meets the business needs, not system-level defects. 𝐇𝐞𝐫𝐞’𝐬 𝐚 𝐩𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐚𝐩𝐩𝐫𝐨𝐚𝐜𝐡 𝐲𝐨𝐮 𝐜𝐚𝐧 𝐟𝐨𝐥𝐥𝐨𝐰: 💡 𝐒𝐭𝐞𝐩-𝐛𝐲-𝐒𝐭𝐞𝐩 𝐭𝐨 𝐖𝐫𝐢𝐭𝐞 𝐄𝐟𝐟𝐞𝐜𝐭𝐢𝐯𝐞 𝐔𝐀𝐓 𝐓𝐞𝐬𝐭 𝐂𝐚𝐬𝐞𝐬: 1️⃣ Start with a Business Requirement ➤ Pick one requirement or user story. e.g., “As a user, I want to view my transaction history for the past 6 months.” 2️⃣ Understand the Acceptance Criteria ➤ Break it down into what “success” looks like. e.g., Transactions are listed, date range filter works, amounts are accurate. 3️⃣ Define Test Case Components ➤ Test Case ID: UAT-001 ➤ Test Scenario: View transaction history ➤ Test Steps: Login Navigate to 'Transactions' Select 6-month range Click “View” ➤ Expected Result: List of all transactions within date range appears accurately ➤ Actual Result: [To be filled during testing] ➤ Status: Pass/Fail ➤ Comments: [For notes or issues] 4️⃣ Use Business Language ➤ Keep it simple and clear so business users can easily execute it. 5️⃣ Organize in a UAT Template ➤ Use Excel, Google Sheets, or tools like Azure DevOps, Jira with Xray, or TestRail. 🚫 𝐂𝐨𝐦𝐦𝐨𝐧 𝐌𝐢𝐬𝐭𝐚𝐤𝐞𝐬 𝐁𝐀𝐬 𝐌𝐚𝐤𝐞 𝐃𝐮𝐫𝐢𝐧𝐠 𝐔𝐀𝐓: ❌ Writing technical or backend-level test cases (that’s QA’s job) ❌ Not mapping test cases to specific requirements (loses traceability) ❌ Not involving business users early in reviewing test scenarios ❌ Overcomplicating test steps with jargon ❌ Skipping edge cases and negative flows ❌ Assuming UAT is just “happy path” testing ❌ Not validating the test data ahead of time ❌ Forgetting to include Post Conditions or what happens next ✅ 𝐏𝐫𝐨 𝐓𝐢𝐩: As a BA, your goal is not just to check if the system works, but to ensure it solves the business problem it was designed for. BA Helpline

Solar Module Reliability Tests These are a critical part of ensuring photovoltaic (PV) modules perform safely and efficiently throughout their expected lifespan (typically 25–30 years). These tests are defined by international standards such as IEC 61215, IEC 61730, and UL 1703, and are typically conducted in certified laboratories. 🔧 1. Thermal Cycling Test (IEC 61215) Purpose: Simulates stress from daily temperature changes. Conditions: -40°C to +85°C for 200–600 cycles. Failure Criteria: Cracked cells, delamination, or power degradation beyond specified limit. 💧 2. Damp Heat Test (IEC 61215) Purpose: Simulates long-term exposure to high humidity and heat. Conditions: 85°C, 85% RH (Relative Humidity) for 1000 hours. Failure Criteria: Moisture ingress, delamination 🧊 3. Humidity-Freeze Test Purpose: Simulates moisture ingress. Conditions: Cycles of 85°C/85% RH to -40°C. Used to detect: Encapsulant failures ☀️ 4. UV Preconditioning Test Purpose: Exposes modules to UV radiation equivalent to sunlight exposure over time. Conditions: 15 kWh/m² at 60°C. Checks for: Discoloration, encapsulant degradation ⚡ 5. Insulation Resistance & Dielectric Voltage Withstand Test (IEC 61730) Purpose: Ensures electrical safety under wet or humid conditions. Conditions: High-voltage testing of insulation layers. 🌧️ 6. Hot Spot Endurance Test Purpose: Simulates shading or cell mismatch causing local heating (hot spots). Outcome: Identifies risk of fire or localized damage. 🧪 7. Potential Induced Degradation (PID) Test Purpose: Tests susceptibility to voltage-induced degradation. Conditions: High system voltage 1Kv Important for: Utility-scale PV plants. 🌪️ 8. Mechanical Load Test Purpose: Simulates wind and snow loading. Conditions: Typically 5400 Pa (snow) and 2400 Pa (wind). Assesses: Frame integrity, glass cracking, and mounting strength. 🔍 9. Electroluminescence (EL) Imaging Not a standard test, but widely used. Purpose: Detects microcracks, broken cells, or interconnect issues. Used: Before and after mechanical/thermal tests for failure analysis. 🔄 10. Light-Induced Degradation (LID) Test Purpose: Evaluates performance drop after initial sunlight exposure. Mainly affects: Mono PERC and other high-efficiency Si modules. 📉 11. Power Output (Flash Test) Purpose: Measures module output under Standard Test Conditions (STC). Criteria: Power degradation should not exceed 5% (usually tighter in warranties. Test Purpose Standard Thermal Cycling Temperature fluctuation resistance IEC 61215 Damp Heat Humidity and heat endurance IEC 61215 Humidity-Freeze Cold and moisture stress IEC 61215 UV Exposure UV resistance IEC 61215 Insulation Resistance Electrical safety IEC 61730 Hot Spot Test Local heating from shading IEC 61215 PID Test Voltage stress tolerance IEC 62804 Mechanical Load Wind/snow impact IEC 61215 Electroluminescence Imaging Microcrack detection Non-standard tool Flash Test Output performance IEC 61215

What is RTM and How is Used by Business Analysts? RTM (Requirements Traceability Matrix) is a document that tracks and ensures that all project requirements are properly addressed throughout the project lifecycle. It links requirements to their corresponding test cases, design documents, and deliverables, ensuring complete coverage and reducing the risk of missing critical functionalities. How Business Analysts Use RTM: 1. Tracking Requirements – Ensures all business, functional, and technical requirements are addressed. 2. Validation & Verification – Helps confirm that each requirement is implemented and tested. 3. Change Management – Assists in assessing the impact of changes on existing requirements. 4. Project Transparency – Provides clear visibility to stakeholders on requirement progress and gaps. RTM is crucial in bridging the gap between business needs and project execution, making it a key tool for Business Analysts. #rtm #requirementtraceabilitymatrix #businessanalyst #Bas

✅ Quality Management System (QMS) Components & Key Industry Concepts A QMS is a structured framework used by organizations to ensure that their products or services consistently meet customer and regulatory requirements. A well-implemented QMS fosters continuous improvement, operational efficiency, and enhanced customer satisfaction. 🔹 QMS Core Components 1. Risk Management Identify, assess, and mitigate risks that could impact product quality or safety. Tools: Risk Assessments, FMEA, SWOT Analysis 2. Deviation Management Detect and handle any deviations from standard operating procedures or quality expectations. Tools: Deviation Reports, Root Cause Analysis, Corrective Action Plans 3. Equipment Management Maintain, calibrate, and qualify equipment to ensure reliable and accurate performance. Tools: Maintenance Logs, Calibration Records, Qualification Protocols 4. Document Management Control creation, revision, distribution, and archiving of critical quality documents (SOPs, policies, etc.). Tools: Document Control Systems, SOP Templates, Electronic Record Systems 5. Audits & Inspections Conduct internal and external audits to ensure compliance with quality standards and regulatory requirements. Tools: Audit Checklists, Inspection Reports, Compliance Dashboards 6. CAPA Management Address root causes of nonconformities and implement preventive measures to avoid recurrence. Tools: CAPA Forms, 5 Whys, Fishbone Diagrams 7. Supplier Management Qualify, monitor, and evaluate suppliers to ensure they meet quality expectations. Tools: Supplier Audits, Qualification Protocols, Performance Scorecards 8. Training Management Ensure employees are trained, competent, and aware of QMS responsibilities. Tools: Training Curricula, LMS, Competency Evaluations 📘 Keywords & Industry Concepts 1. Quality Assurance (QA) A process-oriented approach focused on preventing defects by ensuring quality is embedded in every step. 2. Quality Control (QC) A product-focused method involving testing and inspections to detect defects. 3. Lean Manufacturing A production philosophy aimed at reducing waste and optimizing processes without compromising value. 4. Six Sigma (DMAIC) A methodology for process improvement through a structured five-step approach: • Define, Measure, Analyze, Improve, Control 5. 5S Methodology A workplace organization system: • Sort, Set in Order, Shine, Standardize, Sustain 6. ISO 9001 An international standard specifying QMS requirements to ensure consistent product/service quality and continual improvement. 7. FMEA A risk analysis technique used to identify and prioritize potential failure modes and their effects. 8. PDCA (Plan-Do-Check-Act) A cycle for continuous improvement and iterative process enhancement. 9. Total Quality Management (TQM) An organization-wide philosophy where all employees participate in improving processes, products, and services.