Did you know you can save a lot of AWS costs with just one simple trick?
👉 Batching!
I run a Shopify app called FraudFalcon that analyzes every order for fraud patterns.
That means:
📦 Lots of incoming orders
⚡️ Near real-time processing
💸 Costs were going through the roof
Before:
Each order triggered one Lambda invocation → super expensive at scale.
So 100 orders = 100 invocations 😬
After:
We changed the setup:
➡️ Order goes into SQS
➡️ Lambda picks up batches of 40 orders from the queue
That one change brought costs way down 💰 (check the chart 👇)
Sometimes it’s not about changing the whole architecture — just making one part smarter.
If you’re building on AWS and seeing high Lambda costs, check if you can batch. It might save you more than you think.
Order fulfillment can make or break your business.
If it's slow, customers notice.
If it's chaotic, your team burns out.
Here’s how to fix it:
→ Centralize your inventory.
Use one system to track everything.
No spreadsheets.
No guesswork.
→ Automate where you can.
Automate order tracking, customer notifications, and stock updates.
→ Standardize your packaging process.
Every product gets the same treatment.
This speeds things up and ensures consistency.
→ Batch tasks.
Pick, pack, and ship in batches instead of one at a time.
This saves hours.
Your goal isn’t just faster fulfillment—it’s happier customers and a more efficient team.
Don’t let a messy backend ruin your front-end reputation.
Small changes lead to big results.
🚀 Big Performance Win in Our Batch Ingestion Pipeline
Recently, I ran into a familiar challenge in one of our bronze ingestion jobs:
500+ tiny files being processed in each batch → slow execution + overhead everywhere.
Classic small-file bottleneck. 🐌
🔍 After reviewing the Spark UI, it was clear that the job spent more time scheduling tasks and listing files than actually processing data.
Here’s what I implemented to fix it:
✔️ Added a pre-bronze compaction step – merging hundreds of small files into a few optimally sized ones
✔️ Used repartition to control output file size (targeting ~200 MB per file)
✔️ Enabled optimizeWrite + autoCompact to prevent small files inside Delta
✔️ Scheduled periodic OPTIMIZE on the bronze Delta tables for long-term health
✔️ Tuned batch sizing to read more files per micro-batch where applicable
📈 Result:
Faster batch throughput, cleaner metadata, and more efficient downstream reads.
Small files may look harmless — but they can quietly drain performance in distributed systems. Optimizing file strategy = major productivity gains.
#DataEngineering#ApacheSpark#Databricks#DeltaLake#BigData
29 orders. 2.5 hours saved. That's one batch.
A customer sent us that feedback this week after running their daily POs through Docuumai instead of keying them into NetSuite by hand.
Before: open the PDF, find the customer, look up item mappings, type every line item, verify pricing. 5-10 minutes per order.
After: customer emails the PO, Docuumai reads it, sales order appears in NetSuite with everything mapped. Review and approve. About 30 seconds.
The math: as a daily batch, they're saving 55+ hours a month.
And it's not just sales orders. We're seeing the same results with remittance processing; complex payment advices covering 100+ invoices going from 20-minute manual tasks to quick reviews.
Docuumai learns from every correction your team makes. Fix a mapping once, it remembers forever. No templates. No per-customer configuration. It just gets smarter over time.
Currently live on:
→ Customer POs → Sales Orders
→ Remittances → Customer Payments
→ Vendor Invoices → Bills
→ Packing Slips → Item Receipts
Vendor quotes → POs coming soon.
If your team is drowning in NetSuite data entry, let's talk.
How I’ve Seen Digital MES Deliver Real Business Value – A Case Study
A real-world example of how a fully integrated MES with review-by-exception delivers business results, enforcing compliance, efficiency, and cost reduction—before jumping into AI and Pharma 4.0.
Imagine reducing batch release times by 80% while improving quality. Sounds too good to be true? It’s not. My customers have succeeded on the very first project many times.
🔬 Case Study: The Power of MES & Review-by-Exception
A biopharma manufacturer I worked with faced a common challenge:
❌ Paper batch printing and review delays were adding days, sometimes weeks, to product release timelines. The creation of the process orders printing paper, stapling, and organizing was the full time job of 3 people.
❌ Quality teams (QC, QP) were drowning in unnecessary checks instead of focusing on real deviations. Sometimes batch records are literally hundreds of pages and each page, label, attachment has to be reviewed by human eyes for compliance.
❌ Compliance audits were becoming a nightmare due to scattered data across paper records and disconnected systems.
✅ The Solution: Implementing a Fully Integrated MES with Review-by-Exception
The company moved away from digital paper-on-glass solutions (which only digitize manual processes) and instead fully integrated MES across manufacturing and quality operations.
Tip: know the difference between an EBR software system and a MES Software system. They’re not the same, but they look very similar!
The results?
✔ Batch release times cut by 80%, eliminating unnecessary manual reviews.
✔ Batch order creation cut by 100%, automatic by the MES
✔ 40% reduction in compliance deviations, thanks to real-time tracking and automated alerts. Just moving to a system eliminates the human error and ensures ALCOA and GMP
✔ Increased operational efficiency across 5 global sites, reducing costs and improving agility.
The Key Takeaway
💡 Before jumping into AI and Pharma 4.0, companies need to first build a strong foundation.
A fully integrated MES enables:
✔ Real-time process control & data visibility
✔ Automation of compliance workflows
✔ Scalability for future AI & analytics adoption
🔎 Strategy before technology—not the other way around.
Have you seen a successful digital transformation case in biopharma? What made it work? Let’s discuss this in the comments!