Reason org.springframework.dao.CannotAcquireLockException: PreparedStatementCallback; SQL [INSERT INTO BATCH_JOB_EXECUTION_CONTEXT (SHORT_CONTEXT, SERIALIZED_CONTEXT, JOB_EXECUTION_ID) VALUES(?, ?, ?) #5474

we recently upgrade spring batch from 5.2 to 6.0.4( latest spring batch available) as part of our spring boot 3.x to 4.x upgrade.
The below issue started to appear in our performance environment and we could see approximately 5 to 10 occurrences of this error when we try to process around 500 files in a span of 4 hours.
The same performance run on a spring batch 5.2 does not has this issue and we have repeadtly tested and conculded that this is only present in spring batch 6.x upgrade.

reason org.springframework.dao.CannotAcquireLockException: PreparedStatementCallback; SQL [INSERT INTO BATCH_JOB_EXECUTION_CONTEXT (SHORT_CONTEXT, SERIALIZED_CONTEXT, JOB_EXECUTION_ID)
VALUES(?, ?, ?)
]; ERROR: could not serialize access due to read/write dependencies among transactions
Detail: Reason code: Canceled on identification as a pivot, during conflict in checking.
Hint: The transaction might succeed if retried.", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at com.xxx.xxx.xxx.job.EPPParserJob.run(EPPParserJob.java:120)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at com.xxx.pymtsrv.settl.sdi.scheduled.ParserFileWatcher.listFiles(XXXXXFileWatcher.java:125)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at java.base/java.lang.reflect.Method.invoke(Method.java:580)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:128)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$1(ScheduledMethodRunnable.java:122)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at io.micrometer.observation.Observation.observe(Observation.java:569)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:122)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at datadog.trace.instrumentation.springscheduling.SpringSchedulingRunnableWrapper.run(SpringSchedulingRunnableWrapper.java:65)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at org.springframework.scheduling.config.Task$OutcomeTrackingRunnable.run(Task.java:88)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
", "class ch.qos.logback.classic.spi.StackTraceElementProxy: at java.base/java.lang.Thread.run(Thread.java:1583)
"

We are running our application on Java 21.

we are using the default isolation level of SERIALIZABLE and even tried changing it to READ_COMMITTED but that didn't help.

We also tried below creating indexes
CREATE INDEX idx_batch_job_exec_params_fk ON BATCH_JOB_EXECUTION_PARAMS (JOB_EXECUTION_ID);
CREATE INDEX idx_batch_step_exec_job_fk ON BATCH_STEP_EXECUTION (JOB_EXECUTION_ID);
CREATE INDEX idx_batch_job_exec_job_ins_fk ON BATCH_JOB_EXECUTION (JOB_INSTANCE_ID);
but that didn't help.

We do not have a very big database with the BATCH_STEP_EXECUTION table having around 500K rows only

Also we are using AWS Aurora DB, PostgresSqlDB version 16.11 and even tried incresing ACU from 10 to 20 but still no luck.

All other runtime environment parameters remain exactly the same from spring batch 5.2 to 6.x

Can you please let us know why this error has started coming from spring batch 6.x onwards and what can be done to resolve.