How Parallel LLM Task-Matching Cuts Latency 4x Without Breaking Logic

A developer was asked to build an AI agent that could generate tasks from meeting notes or messy brainstorming sessions and synchronize them with the team's project management platform. The first version came together quickly. It generated structured tasks, created issues, and worked smoothly in testing.

Then it reached production.

A real project contained hundreds of existing tasks. To reconcile every new task, the system sent large amounts of project context back to the LLM and waited for each matching request to finish. As the workload grew, latency increased; requests began hitting rate limits, retries added even more pressure, and new tasks spent longer waiting in the queue. What had looked like a simple task-generation feature had quietly turned into a scalability problem.

Why Task Synchronization Is Harder Than Task Generation

Generating structured tasks is relatively straightforward. Most LLMs can identify action items from meeting transcripts, notes, or brainstorming sessions with reasonable accuracy.

The real challenge begins when those tasks must be synchronized with a project management system such as Redmine or Jira. For each new task, the agent must decide whether it should remain standalone, match an existing issue, become a subtask, or act as the parent of another task.

A naive solution is to send all newly extracted tasks, together with hundreds of existing project issues, back into an LLM and ask it to reconcile everything in one pass.

This approach does not scale well. As the context grows, inference becomes slower and more expensive. The model must also track too many tasks and possible relationships at once, which increases the risk of missed matches, incorrect hierarchies, and inconsistent decisions.

The approach explored in this article combines semantic matching with parallel processing. Semantic matching reduces the candidate set for each task, while parallelization allows independent matching requests to run concurrently. The resulting proposals are then reconciled in a separate post-processing step to preserve valid task relationships.

Core Concepts

Before looking at the implementation, it helps to understand the main concepts used in the pipeline.

Task Hierarchy

Project management platforms such as Redmine and Jira usually organize work using hierarchical relationships.

A task may be:

• A standalone issue
• A parent task that represents a larger scope of work
• A subtask that contributes to a broader task

For example, a task hierarchy structure can be:

When an agent generates a new task, it must decide where that task belongs in the existing hierarchy. A wrong decision may create duplicates, attach a task to an unrelated parent, or restructure an existing issue incorrectly.

Semantic Matching

Semantic matching compares two sentences to determine whether they express the same or a similar meaning.

The idea is to use a bit of mathematical magic. Each sentence is mapped into a latent vector space, where it is represented as a numerical vector. If two vectors are close to each other, their sentences are likely to have similar meanings. If they are far apart, the sentences are probably less related.

For example, these two tasks use different wording but describe closely related work:

• Implement permission checks for admin APIs
• Add role-based authorization to backend endpoints

Semantic matching converts them into a vector, which is a numerical representation of its meaning. Sentences with similar meanings should have vectors that are closer together.

The system can, then, look for existing tasks that are closest in meaning and select a small group of relevant candidates from hundreds or thousands of issues.

Parallel Processing

Parallel processing is an execution model in which multiple tasks are worked during the same period of time.

In a sequential process, one task must finish before the next task begins:

The total execution time is therefore close to the sum of the processing time of all tasks.

In a parallel process, multiple tasks can make progress concurrently:

Therefore, the total execution time is equal to the longest-running of the three tasks. This is especially useful for I/O-bound workloads, where a program spends much of its time waiting for external services, network responses, or database operations.

That best case assumes every task gets its own worker. In practice, worker pools are bounded — a fixed number of workers pull from a queue of tasks.

When N tasks share W workers, they queue up in batches instead of all running at once:

T_parallel ≈ ⌈N / W⌉ × average task time

With 20 tasks and 4 workers, for example, that's 5 batches — not one. Adding more workers shrinks the number of batches, not the time any single task takes.

Reconciliation

Reconciliation is the process of combining multiple independent results into one consistent final state. It is commonly used when several workers, services, or models produce outputs without having full visibility into each other's decisions.

For example, two workers may both select the same issue for different relationships. Each result may appear reasonable on its own, but both cannot necessarily be applied at the same time.

Reconciliation typically involves three steps:

1. Collect results
2. Detect conflicts
3. Apply resolution rules

The Proposed Architecture

Instead of asking a single LLM to reconcile every new task with the entire project at once, the proposed architecture breaks the problem into smaller and more manageable stages.

For each new task, the system first retrieves a small set of existing tasks that are closest in meaning. A reranker then refines this candidate list so that the LLM only needs to evaluate the most relevant relationships.

The remaining matching requests are processed concurrently. Each worker evaluates one new task and returns a relationship proposal tag.

Once all proposals have been collected, a reconciliation step checks them as a complete batch. It removes conflicting relationships, applies deterministic resolution rules, and produces a final set of valid updates.

Only after this validation step are the tasks and their relationships synchronized to Redmine or Jira.

The architecture, therefore, separates the workflow into three main responsibilities:

• Semantic retrieval reduces the search space
• Parallel matching reduces processing time
• Reconciliation preserves global consistency

The Method Under the Hood

The architecture looks simple from the outside, but each stage solves a different problem. In a nutshell, semantic retrieval keeps the search space manageable, parallel processing reduces waiting time, and reconciliation keeps the final relationships consistent.

Semantic Candidate Retrieval

A real project may contain hundreds or even thousands of existing issues. Sending all of them to an LLM every time a new task is created would make the context unnecessarily large. Instead of that, each new task is first compared with the existing tasks using semantic similarity.

As explained earlier, semantic matching represents tasks as vectors in a latent space. Tasks with similar meanings tend to stay closer together. This allows us to quickly narrow down a large project to a much smaller candidate set.

For example:

• 100+ existing issues → Semantic retrieval → Top 10 candidates

At this stage, the goal is not to decide the final relationship. Consequently, retrieval only answers a simpler question:

• Which existing tasks are worth looking at?

The candidate set is then passed through a reranker, which evaluates the pairs more carefully and pushes the strongest candidates toward the top.

• Top 50 candidates → Reranking → Most relevant candidates → LLM relationship evaluation

This gives the LLM a smaller and cleaner context instead of asking it to reason over the entire project.

For each candidate, the LLM can then propose a relationship such as:

• PARENT
• CHILD
• UNRELATED

So far, we have reduced the amount of work required for a single task. The next problem is what happens when an agent needs to synchronize many new tasks at once.

Parallel Relationship Matching

Let's suppose a long meeting transcript is distilled into 20 new tasks.

If we process them sequentially, each task must wait until the previous task finishes its LLM calls.

Most of the time spent on an LLM request is actually waiting for the external endpoint to return a response. If tasks are processed sequentially, each new task has to wait for the previous request to finish. By running independent matching operations concurrently, these waiting periods can overlap, significantly reducing the total processing time:

In parallel implementation, the system creates a fixed number of workers, and each worker runs the same semantic matching pipeline independently for a different task. A worker retrieves relevant candidates, asks the LLM to determine the relationship, and produces a relationship proposal. As workers finish, their proposals are collected into a shared proposal set. Once all tasks have been processed, the complete set of proposals is passed to the reconciliation stage, where conflicts are detected and resolved before any changes are committed to the project management system.

The Hidden Business Trap

The sequential implementation implicitly relied on execution order. Once Task A selected Issue X for a relationship, the candidate state was updated before Task B started.

Parallel execution removes this guarantee. Multiple workers can evaluate the same candidate before any result is applied.

This can produce conflicts such as:

• Task A → Issue X = PARENT
• Task B → Issue X = CHILD

or

• Task A → Issue X = CHILD
• Task B → Issue X = CHILD

Post-processing Reconciliation

Instead of coordinating shared state between workers, each worker generates an independent relationship proposal.

• Task A → Proposal A
• Task B → Proposal B
• Task C → Proposal C

After all workers finish, the proposals are reconciled as a batch.

1. Parallel proposals
2. Detect conflicts
3. Apply resolution rules
4. Valid relationships

The current implementation uses two rules:

• If the same issue is selected as both parent and child, the parent relationship wins.
• If multiple new tasks select the same existing issue as their child, the earliest task in the original input order keeps it.

Benchmark

Purpose

This benchmark checks two separate claims:

• Sequential and bounded-parallel task matching preserve the same business result.
• Parallelism reduces latency when independent task work includes I/O wait.

The second claim is measured with a deterministic I/O simulation. It is not presented as a live embedding or LLM measurement.

Fixture and common configuration

The frozen, sanitized fixture contains:

• 65 selected Redmine issues
• 179 reviewed task cases
• Exact matches, parent/subtask relationships, ancestor relationships, no-match cases, and a deliberately ambiguous equal-score case with multiple valid issue IDs
• Fixture SHA-256: cba9aba878f35a91342400fd2cf9386ed2131684421bb2579a0f211a738a32b1

Both execution modes use the same deterministic evaluator and reducer:

1. Rank candidate issues by cosine similarity.
2. Break score ties by ascending issue ID.
3. Apply a similarity threshold of 0.84 and an ambiguity margin of 0.03.
4. Produce per-case proposals.
5. Reduce proposals into the final decision and relationship state.

The parallel run uses a bounded ThreadPoolExecutor with four workers. Workers generate independent read-only proposals; final state reduction remains deterministic.

Measured I/O-simulated result

This is the run that supports the approximately 4× latency claim. It measures a complete batch of 179 cases with a deterministic 8 ms I/O delay per case. The delay represents waiting on independent remote work such as embedding, retrieval, reranking, LLM matching, or task-management API calls. It does not call those production services.

The measured p50 speedup is:

1,473.764 ms / 371.900 ms = 3.966× ≈ 4.0×

The measured throughput increase is approximately 3.98× as well:

480.91 cases/s / 121.40 cases/s = 3.96×

The result comes from overlapping independent I/O waits. It is not caused by faster local cosine scoring.

Cached CPU-only baseline

The cached baseline uses fixture vectors and local cosine scoring only. It verifies deterministic behavior and scheduler overhead, but contains almost no waiting for workers to overlap.

For the cached CPU-only workload, the p50 ratio is 6.83 / 10.35 = 0.66×; sequential is faster because thread scheduling costs more than the local computation. This does not contradict the I/O-simulated result: the two tests measure different workload shapes.

Business-logic preservation

The I/O-simulated sequential and parallel runs produced identical deterministic business results.

These results show that parallel scheduling preserved the reviewed deterministic decision and relationship state. They do not establish live model quality.

Retry and retrieval-ablation status

The harness now accepts deterministic controls for:

• simulated I/O delay and bounded jitter;
• rate-limit probability;
• retry count and exponential backoff base delay;
• retrieval-only versus retrieval-plus-reranking mode;
• optional reranking delay.

The reported 4× run used no injected 429s and a zero reranking delay, so it is specifically an I/O-overlap measurement. A separate retry experiment should report rate-limit count, retry count, exhausted requests, and latency impact. A separate ablation should compare quality and latency for retrieval-only versus retrieval-plus-reranking.

Reconciliation flow

Parallel proposals → Detect conflicts → Apply resolution rules → Valid relationships

Workers generate read-only proposals. Conflict detection, resolution, and final relationship application remain centralized and deterministic.

Conclusion and limitations

The benchmark supports both of these bounded claims:

1. Parallel scheduling preserved the deterministic business result exactly on the reviewed fixture.
2. Under the measured 8 ms-per-case I/O simulation, four workers reduced p50 complete-batch latency from 1,473.76 ms to 370.46 ms, a 3.978× (approximately 4×) speedup.

The 4× result is a measured simulation, not a live-service result. Before using it as a production SLA, repeat the experiment with real embedding, retrieval, reranking/LLM, Redmine, and Jira calls, including service quotas, jitter, HTTP 429 responses, backoff, and 10–20 repetitions under production-like load.

Realistic Application

The production pipeline has a different workload. Matching a generated task to Redmine issues may require:

1. fetching possible candidates from Redmine;
2. generating or retrieving embeddings;
3. calculating similarity and reranking candidates;
4. asking an LLM to resolve uncertain matches;
5. creating issues and task relations through the Redmine API.

Embedding, LLM, and Redmine calls are mostly blocking I/O. During these waits, Python threads can overlap other independent requests, making bounded concurrency more useful than it was in the CPU-only benchmark.

A production-oriented flow can therefore use two phases:

# Phase 1: parallel, read-only evaluation
with ThreadPoolExecutor(max_workers=4) as executor:
    evaluated = list(executor.map(evaluate_task, tasks))

# Restore deterministic input order
evaluated.sort(key=lambda row: row.input_index)

# Phase 2: centralized state mutation
final_state = reduce_and_apply(evaluated)

The worker phase may fetch candidates, call embedding services, score matches, and prepare proposed actions. However, it should not immediately create issues or parent/subtask relations.

The reducer remains responsible for:

• resolving conflicts between proposals;
• preventing duplicate issue creation;
• preserving parent-before-child dependencies;
• determining relation direction;
• applying mutations in a stable order;
• retrying failed writes safely.

This is particularly important when two generated tasks independently select the same Redmine issue, or when one task must be created before another can reference it as a parent.

Conclusion

Splitting task-matching into semantic retrieval, parallel LLM evaluation, and centralized reconciliation turns an unbounded reconciliation problem into three smaller, independently testable stages. The benchmark above confirms the property that matters most before any performance claim: bounding concurrency and routing every write through a single centralized reducer keeps candidate selection, relation direction, and duplicate handling identical to the sequential baseline, even when worker threads finish in a different order. That correctness guarantee is what makes it safe to introduce parallelism into a workflow that mutates a shared project hierarchy.

A deterministic I/O simulation now answers part of the remaining question. With an 8 ms per-case wait standing in for embedding, retrieval, reranking, and LLM calls, four workers cut p50 batch latency by roughly 4×, while the cached CPU-only run showed the opposite: thread-dispatch overhead outweighing the tiny compute cost per case. Both results are real and both are correct; they simply measure different workload shapes, which is why the performance claim needed splitting into two in the first place. What is still missing is the live version of that same experiment (real embedding, retrieval, reranking, and Redmine/Jira calls under production quotas), plus the retry/backoff and retrieval-vs-reranking ablations the harness now supports but has not yet run.