Why your job queue gets overloaded when you launch too many individual background jobs

Published on June 2, 20263 min read#background jobs #automation

This is one of those problems that shows up “out of nowhere” after months of working fine: one day, SM37 fills up with jobs stuck in Released status that won’t start, or that start hours late compared to their scheduled time. The typical reaction is to ask for more background (BTC) work processes in RZ04, and it holds for a few more weeks before happening again.

The symptom

In SM37 you see dozens (or hundreds) of jobs with nearly identical names — ZINTERFACE_ORDER_000123, ZINTERFACE_ORDER_000124… — all scheduled almost at the same minute. In SM66 all available BTC processes are busy, and SM50 shows that most of them spend seconds of actual execution inside minutes of queue time.

The real cause

The problem is almost never “we’re short on background processes.” The problem is the design: someone decided that each document (an order, an invoice, an IDoc) should launch its own individual job instead of processing several documents per execution. It’s a decision that seems reasonable at first — isolating each document simplifies error handling, each job has its own log — but it doesn’t scale:

Each job holds an entire BTC process from release to completion, even if the actual work takes two seconds.
The job scheduler has to evaluate the queue on every cycle, and with hundreds of entries that has a cost too.
If document volume grows 30%, the queue doesn’t grow 30%: it overloads, because BTC processes are a fixed resource shared with other processes on the instance (spool, update, etc. if they aren’t properly isolated).

How to confirm it

Before touching anything, I measure. A couple of quick checks:

" How many jobs of the same family are active/scheduled right now
SELECT jobname, status, sdlstrtdt, sdlstrttm
  FROM tbtco
  INTO TABLE @DATA(lt_jobs)
  WHERE jobname LIKE 'ZINTERFACE_ORDER%'
    AND status IN ( 'R', 'Y' ). " Running, Released/Scheduled

If lt_jobs has hundreds of rows for a window of a few minutes, that’s your problem. It’s also worth checking RZ04/SM61 to see how many BTC processes are actually available on the instance versus how many you’d need if every document consumes one.

The fix that actually works

It’s not adding more BTC processes (that’s patching without fixing). The two solutions that genuinely scale:

Batch processing inside a single job. Instead of one job per document, a job that processes N documents in an internal loop, with a COMMIT WORK per document (so a single failure doesn’t lose the whole batch) and its own error log in a Z table, not in the job log.
Controlled parallelization with CALL FUNCTION ... STARTING NEW TASK when volume is high and per-document processing is heavy. Here you do want several processes in parallel, but a bounded, known number (say, 4 or 6 parallel tasks), not one per document.

With either approach, you go from “hundreds of two-second jobs fighting over the queue” to “a handful of jobs doing real, predictable work.” Per-document error handling stays in place with your own log table, not the job’s standard log — which is exactly what made the original approach attractive.

The lesson

One job per document is easy to code and hard to operate. When volume grows, the bottleneck is never CPU: it’s the BTC process queue, a shared and finite resource. Design for batches from the start, even if the initial volume looks small.