ASP.NET Core Background Jobs with RabbitMQ: Reliable Patterns for HTTP Offloading

Ryan

3 months ago

When an HTTP endpoint needs to trigger slow, expensive, or fragile operations, keeping that work inside the request is rarely the best choice: perceived latency goes up, failures become much harder to manage, and, above all, job completion becomes tightly coupled to the lifetime of the client connection, which is anything but guaranteed, especially on mobile devices and wireless networks. In ASP.NET Core, the real step forward comes when you separate request intake from actual job execution, delegating the latter to a dedicated worker process.

One of the most established, and still most reliable, approaches looks like this: the web application validates the request, persists the minimum state it needs, and publishes a message to a message broker such as RabbitMQ; a background .NET worker reads from the queue, performs the work, updates the database, and handles retries, failures, and telemetry. The model is easy enough to understand, but it still needs some discipline to work well in production: idempotency, flow control, graceful shutdown, a clear distinction between transient and permanent errors, and, above all, a credible strategy to preserve consistency between database and broker.

Why moving work outside the request pays off

The use cases are familiar: document processing, email delivery, report generation, synchronization with external systems, indexing, bulk imports, calls to slow or unstable providers. In all these scenarios, the HTTP request should do as little as possible: validate input, persist the initial data, and enqueue the job.

The benefit is not limited to lower user-facing latency. Decoupling producer and consumer lets you retry jobs independently from the original HTTP session, scale workers horizontally without touching the Web API, and contain partial failures much more effectively. This is the core pattern behind a large number of systems that need to stay responsive even when the real work is slow, intermittent, or failure-prone. The material you shared points exactly in that direction, even though it was tied to a specific project; here I am deliberately generalizing it.

Baseline architecture

A sensible baseline, without overengineering it, usually includes the following elements:

an ASP.NET Core Web API that receives the request and publishes a job;
an internal abstraction, such as IJobQueue, so RabbitMQ client dependencies do not spread throughout the codebase;
a concrete producer implementation that serializes the payload and sends it to the appropriate exchange and queue;
a separate worker, based on the Generic Host, that consumes messages;
an application table used to track job state, errors, and retry count;
telemetry for outcomes such as completed, failed, discarded, and retried.

The typical lifecycle is fairly linear: the producer writes metadata to the database, publishes a message containing the job identifier, the consumer receives it, moves the job into Processing, performs the operation, and then marks it as Completed or Failed. If the message is redelivered, idempotency and retry policies come into play. The attached material follows exactly this approach, with explicit state persistence to avoid treating the broker as the only source of truth.

HTTP producer: when, and how, to publish

There is a common misunderstanding here: “responding immediately” does not mean publishing blindly and moving on. The request should only return after the application has saved everything required to reconstruct the job, validated its consistency, and, ideally, ensured that the message will actually be published. That last part is what separates demos from systems that hold up in production.

A simplified example might look like this:

app.MapPost("/documents", async (
    UploadDocumentRequest request,
    AppDbContext db,
    IJobQueue jobQueue,
    CancellationToken ct) =&gt;
{
    var document = new Document
    {
        Id = Guid.NewGuid(),
        FileName = request.FileName,
        State = DocumentState.Pending
    };

    db.Documents.Add(document);
    await db.SaveChangesAsync(ct);

    await jobQueue.EnqueueAsync(new ParseDocumentJobDto
    {
        DocumentId = document.Id
    }, ct);

    return Results.Accepted($"/documents/{document.Id}", new
    {
        document.Id,
        document.State
    });
});

app.MapPost("/documents", async (

UploadDocumentRequest request,

AppDbContext db,

IJobQueue jobQueue,

CancellationToken ct) =>

{

var document = new Document

{

Id = Guid.NewGuid(),

FileName = request.FileName,

State = DocumentState.Pending

};

db.Documents.Add(document);

await db.SaveChangesAsync(ct);

await jobQueue.EnqueueAsync(new ParseDocumentJobDto

{

DocumentId = document.Id

}, ct);

return Results.Accepted($"/documents/{document.Id}", new

{

document.Id,

document.State

});

It works, but it has an obvious flaw: between SaveChangesAsync and EnqueueAsync there is a window where the database may be updated correctly while the message never reaches the broker. At that point, the outbox pattern stops being an architectural nicety and becomes a real requirement.

Outbox pattern: keeping database and broker consistent

The problem is straightforward: in most modern systems, there is no distributed transaction between your application database and RabbitMQ that is both reliable and practical. If you update the database and then publish, you may lose the message; if you publish first and the commit fails afterward, you may emit an event describing something that was never actually saved.

The transactional outbox solves this by storing the message in an outbox table within the same transaction used to save the business data; a separate process, or a dedicated publisher, then reads the outbox and forwards those messages to the broker. In this model, the database update and event registration are atomically consistent at the local transaction level. That is exactly the problem the pattern is meant to solve.

In practice, the flow becomes:

save business entities and an outbox record in the same transaction;
commit once;
a background publisher reads the unsent outbox records;
publish them to RabbitMQ;
mark the outbox record as sent.

This does not remove the need for consumer-side idempotency, because the publisher can still retry or duplicate delivery in edge cases, but it removes the most dangerous failure mode: silent misalignment between database state and broker state.

.NET consumer: IHostedService or BackgroundService?

In ASP.NET Core, and more broadly in the Generic Host, a background service implements IHostedService. BackgroundService is not a conceptual alternative: it is a framework-provided base class that already implements IHostedService and lets you focus your long-running logic inside ExecuteAsync. Microsoft documents it explicitly that way: BackgroundService is a base class for long-running services, while IHostedService remains the fundamental contract.

In practice, the rule is simple:

use BackgroundService when you have a continuous consumption loop, a poller, or a worker that should live for the whole lifetime of the process;
implement IHostedService directly when you need finer control over start, stop, timers, unmanaged resources, or multiple initialization and teardown phases. Microsoft explicitly points to this path when BackgroundService is not enough.

For a typical RabbitMQ consumer, BackgroundService is almost always the most natural fit:

public sealed class JobConsumerService : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger&lt;JobConsumerService&gt; _logger;
    private readonly IRabbitMqConnectionFactory _connectionFactory;

    public JobConsumerService(
        IServiceScopeFactory scopeFactory,
        ILogger&lt;JobConsumerService&gt; logger,
        IRabbitMqConnectionFactory connectionFactory)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
        _connectionFactory = connectionFactory;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await using var connection = await _connectionFactory.CreateAsync(stoppingToken);
        await using var channel = await connection.CreateChannelAsync(cancellationToken: stoppingToken);

        await channel.BasicQosAsync(prefetchSize: 0, prefetchCount: 16, global: false, cancellationToken: stoppingToken);

        // queue / exchange / binding declaration

        var consumer = new AsyncEventingBasicConsumer(channel);

        consumer.ReceivedAsync += async (_, ea) =&gt;
        {
            using var scope = _scopeFactory.CreateScope();
            var handler = scope.ServiceProvider.GetRequiredService&lt;IJobMessageHandler&gt;();

            try
            {
                await handler.HandleAsync(ea.Body.ToArray(), stoppingToken);
                await channel.BasicAckAsync(ea.DeliveryTag, multiple: false, cancellationToken: stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error while processing message {DeliveryTag}", ea.DeliveryTag);
                await channel.BasicNackAsync(ea.DeliveryTag, multiple: false, requeue: false, cancellationToken: stoppingToken);
            }
        };

        await channel.BasicConsumeAsync(
            queue: "jobs",
            autoAck: false,
            consumer: consumer,
            cancellationToken: stoppingToken);

        await Task.Delay(Timeout.Infinite, stoppingToken);
    }
}

public sealed class JobConsumerService : BackgroundService

{

private readonly IServiceScopeFactory _scopeFactory;

private readonly ILogger<JobConsumerService> _logger;

private readonly IRabbitMqConnectionFactory _connectionFactory;

public JobConsumerService(

IServiceScopeFactory scopeFactory,

ILogger<JobConsumerService> logger,

IRabbitMqConnectionFactory connectionFactory)

{

_scopeFactory = scopeFactory;

_logger = logger;

_connectionFactory = connectionFactory;

}

protected override async Task ExecuteAsync(CancellationToken stoppingToken)

{

await using var connection = await _connectionFactory.CreateAsync(stoppingToken);

await using var channel = await connection.CreateChannelAsync(cancellationToken: stoppingToken);

await channel.BasicQosAsync(prefetchSize: 0, prefetchCount: 16, global: false, cancellationToken: stoppingToken);

// queue / exchange / binding declaration

var consumer = new AsyncEventingBasicConsumer(channel);

consumer.ReceivedAsync += async (_, ea) =>

{

using var scope = _scopeFactory.CreateScope();

var handler = scope.ServiceProvider.GetRequiredService<IJobMessageHandler>();

try

{

await handler.HandleAsync(ea.Body.ToArray(), stoppingToken);

await channel.BasicAckAsync(ea.DeliveryTag, multiple: false, cancellationToken: stoppingToken);

}

catch (Exception ex)

{

_logger.LogError(ex, "Error while processing message {DeliveryTag}", ea.DeliveryTag);

await channel.BasicNackAsync(ea.DeliveryTag, multiple: false, requeue: false, cancellationToken: stoppingToken);

}

};

await channel.BasicConsumeAsync(

queue: "jobs",

autoAck: false,

consumer: consumer,

cancellationToken: stoppingToken);

await Task.Delay(Timeout.Infinite, stoppingToken);

}

Real back-pressure: why prefetchCount matters

Anyone who has worked with RabbitMQ long enough runs into this sooner or later: if you give the broker too much freedom, the consumer may end up flooded with unacked messages, memory usage grows, latency gets worse, and behavior becomes harder to predict. The key setting here is prefetchCount, meaning the maximum number of unacknowledged messages a consumer is allowed to hold at the same time.

RabbitMQ documents prefetch precisely as a mechanism to limit unconfirmed deliveries; it also points out that high prefetch values lead to more unacked messages and higher broker memory usage. With quorum queues, the topic becomes even more relevant, and the documentation notes that consumers benefit from appropriate prefetch values so they do not remain starved, which is not the same thing as setting the number arbitrarily high.

A practical starting point looks like this:

if the job is heavy, start with a low prefetchCount, such as 4, 8, or 16;
if the job is fast and mostly I/O-bound, you can increase it, but only after measuring throughput, RAM usage, and acknowledgment timing;
avoid autoAck in consumers that do real work: manual acknowledgments, combined with back-pressure, are one of the most useful protections RabbitMQ gives you.

prefetchCount is not some final tuning detail to adjust at the end; it is part of the operational contract between broker and worker.

Idempotency and duplicate handling

With RabbitMQ, as with most traditional brokers, you need to reason in terms of at-least-once delivery. If the consumer crashes after starting the work but before sending the acknowledgment, the message can be redelivered. If the publisher retries, you may see the same payload twice. If a dead-letter loop is misconfigured, you may even multiply the issue.

That is why idempotency should never be treated as a nice extra. It has to be part of the design. The file you shared makes this point clearly: the key should not be the broker delivery tag, but the logical job payload, so the protection also covers publisher-side retries.

There are at least three pragmatic strategies:

a unique database constraint on a logical job key, for example DocumentId + JobType;
a table or distributed cache that records jobs already in progress or already completed, with a reasonable TTL;
handlers designed so they can run more than once without producing duplicate business side effects.

A minimal, simplified example could look like this:

public async Task HandleAsync(ParseDocumentJobDto message, CancellationToken ct)
{
    var key = $"parse-document:{message.DocumentId}";

    if (!await _idempotency.TryAcquireAsync(key, ttl: TimeSpan.FromMinutes(30), ct))
    {
        _logger.LogInformation("Skipping duplicate job for document {DocumentId}", message.DocumentId);
        return;
    }

    try
    {
        var existing = await _db.Jobs
            .SingleOrDefaultAsync(x =&gt; x.ExternalKey == key, ct);

        if (existing is not null &amp;&amp; existing.State == JobState.Completed)
            return;

        // actual job execution
    }
    finally
    {
        await _idempotency.ReleaseOrKeepAsync(key, ct);
    }
}

public async Task HandleAsync(ParseDocumentJobDto message, CancellationToken ct)

{

var key = $"parse-document:{message.DocumentId}";

if (!await _idempotency.TryAcquireAsync(key, ttl: TimeSpan.FromMinutes(30), ct))

{

_logger.LogInformation("Skipping duplicate job for document {DocumentId}", message.DocumentId);

return;

}

try

{

var existing = await _db.Jobs

.SingleOrDefaultAsync(x => x.ExternalKey == key, ct);

if (existing is not null && existing.State == JobState.Completed)

return;

// actual job execution

}

finally

{

await _idempotency.ReleaseOrKeepAsync(key, ct);

}

The concrete implementation may vary, but the principle stays the same: a redelivered message must not duplicate business effects.

Retry, dead-lettering, and application-level job state

Another strong point in the source material is the decision to persist job state inside the application domain instead of relying on the queue alone. I agree with that choice. The broker knows how to deliver messages; it does not do a good job of representing functional process state. If you want to show in the UI that a job is Processing, Completed, Failed, or Skipped, you almost always need a dedicated application table.

Separating those outcomes also helps operationally:

Completed: the work finished successfully;
Failed: the attempt failed and requires retry or analysis;
Skipped: the message arrived, but the domain decided that execution was not needed;
Dead-lettered: the broker stopped retrying and isolated the message for manual inspection.

That distinction avoids one of the most common mistakes: treating the dead-letter exchange as the only audit mechanism.

RabbitMQ classic vs quorum queues

For years, classic queues were the default choice in a huge number of projects. Today, when the requirement is reliability rather than raw throughput in simple scenarios, quorum queues deserve serious attention. RabbitMQ documents them as replicated, fault-tolerant queues based on a Raft-like algorithm, designed for environments where availability truly matters; at the same time, the documentation makes it clear that they are heavier on disk I/O and that throughput drops with larger messages.

Translated into practical terms:

use classic queues when your requirements are modest, the topology is simple, or you are in a development or test setup where you want the lightest possible configuration;
prefer quorum queues when queue loss is unacceptable, the cluster is real, and you need more robust replication semantics;
always measure against your own workload: quorum does not mean “better in every case”, it means “better suited” to a specific set of operational trade-offs.

For critical background jobs, I now tend to see quorum queues as a very sensible default, provided the team understands that they come with extra cost in resources and tuning effort.

Comparing RabbitMQ with in-process Channel<T>

This point is worth stating clearly, because confusion is common: Channel<T> and RabbitMQ do not solve exactly the same problem. .NET channels are in-memory synchronization structures for asynchronous producers and consumers running inside the same process; they work extremely well for local queues, internal pipelines, and bounded sequential work inside a single application instance. Microsoft’s documentation presents them in exactly those terms.

That leads to a straightforward distinction:

Channel<T> is excellent when the work can stay inside the same process, you do not need persistence across restarts, and you want to avoid the complexity of an external broker;
RabbitMQ becomes relevant when you need durability, decoupling across processes, independent scaling, broker-level retry, and clear workload isolation between WebApp and worker;
Channel<T> is simpler to maintain and faster in-process, but when the process dies, the in-memory queue is gone unless you build extra mechanisms around it.

In many real systems, both approaches coexist quite naturally: Channel<T> for lightweight local work, RabbitMQ for work that must survive process restarts and scale outside the Web API boundary.

Graceful shutdown and cancellation tokens

Graceful shutdown in a worker is often underestimated; then, in production, teams end up with half-finished jobs, chaotic redeliveries, and unpredictable behavior during deployment or scale-down. In the Generic Host, the application lifecycle is managed by the runtime, and you can hook into stop events through IHostApplicationLifetime; Microsoft also shows this approach in the documentation for queued background services.

In practical terms, it is worth doing the following:

stop accepting new messages as soon as shutdown begins;
let in-flight work complete, within a reasonable timeout;
ack only after the work has genuinely finished;
close connection and channel after draining, not before.

A simplified lifetime hook may look like this:

public sealed class WorkerShutdownCoordinator
{
    public WorkerShutdownCoordinator(
        IHostApplicationLifetime lifetime,
        ILogger&lt;WorkerShutdownCoordinator&gt; logger)
    {
        lifetime.ApplicationStopping.Register(() =&gt;
        {
            logger.LogInformation("Application stopping: stop consuming new messages and drain in-flight jobs.");
        });
    }
}

public sealed class WorkerShutdownCoordinator

{

public WorkerShutdownCoordinator(

IHostApplicationLifetime lifetime,

ILogger<WorkerShutdownCoordinator> logger)

{

lifetime.ApplicationStopping.Register(() =>

{

logger.LogInformation("Application stopping: stop consuming new messages and drain in-flight jobs.");

});

}

The code itself does not perform the magic, of course; what matters is the operational discipline behind it, namely avoiding process shutdown while the consumer still has in-flight work and no controlled drain strategy.

Reference architecture example

If I had to outline a reliable baseline for background work in ASP.NET Core with RabbitMQ, I would go in this direction:

a Web API that persists the initial state and records the event in an outbox table;
an outbox publisher that forwards messages to the broker with appropriate confirmation handling;
a separate worker based on BackgroundService;
manual acknowledgments and a prefetchCount tuned against the real workload;
consumer-side idempotency as a non-negotiable requirement;
job state persisted in the application database;
a dead-letter queue for permanent failures;
clean shutdown integrated with the host lifetime;
telemetry covering duration, outcomes, retries, and consumer saturation.

It is not the only possible architecture, but it is one of the few that usually survives the transition from demo to real system without too many surprises.

Conclusions

Background work in ASP.NET Core is not just about moving things out of the request path. It requires you to decide where the boundary between synchronous and asynchronous execution belongs, how much responsibility should be delegated to an external broker, how duplicates and retries should be handled, and how to build a consumer that shuts down as cleanly as it starts up.

In this context, RabbitMQ remains a very solid option: it gives you real decoupling between producer and worker, lets you scale consumers independently, and, when combined with manual acknowledgments, sensible prefetch values, dead-letter queues, idempotency, and an outbox strategy, it puts you on much firmer ground than the classic fire-and-forget code hanging off an HTTP request. Channel<T> remains extremely useful, but for a different class of problems; BackgroundService is almost always the right choice for a long-running consumer loop, while IHostedService is there when you need finer lifecycle control; quorum queues and graceful shutdown are not advanced details to postpone, but part of the original design.

When these pieces are assembled well, the system becomes faster from the user’s point of view, more predictable operationally, and much less fragile when real failures start to show up, which is usually the moment when architecture stops being a diagram and starts proving its value.

References

Background tasks with hosted services in ASP.NET Core - Microsoft documentation on hosted background services in ASP.NET Core.
System.Threading.Channels in .NET - Official guide to in-memory producer/consumer channels in .NET.
.NET Generic Host - Overview of the hosting model behind workers and background services.
Implement the IHostedService interface - Microsoft guidance on implementing IHostedService directly.
RabbitMQ Consumer Prefetch - Official RabbitMQ documentation on consumer-side flow control.
RabbitMQ Quorum Queues - Official documentation on replicated and fault-tolerant quorum queues.
Quorum Queues and Flow Control - RabbitMQ blog post explaining quorum queue behavior and flow control.
Transactional Outbox Pattern - Reference explanation of the outbox pattern and its consistency guarantees.

Print Friendly & PDF Download