📬 Part 3 — Dead Letter Queue, Retries, and Monitoring in Azure Service Bus with .NET

👉 Now in Part 3, we’ll focus on:

• 🔎 Dead Letter Queue (DLQ) — what it is and how to handle it
• 🔄 Retry strategies — built-in + custom
• 📊 Monitoring with Application Insights — so you always know what’s happening in production

When building microservices, things will fail — bad data, network hiccups, downstream services offline. Instead of crashing, Service Bus gives you:

• Retries → automatic redelivery of failed messages
• DLQ (Dead Letter Queue) → final resting place for poisoned messages
• Monitoring → visibility into failures, retries, and latency

Let’s break each one down with .NET examples.

---

🪦 Dead Letter Queue (DLQ)

🔹 What is DLQ?

• Each queue and subscription in Service Bus has a Dead Letter Queue.
• Messages go to DLQ when:
  • Max delivery attempts exceeded
  • Message expired (TTL)
  • Explicitly dead-lettered by the app

📍 DLQ path:

<queue-name>/$DeadLetterQueue
<topic-name>/Subscriptions/<sub-name>/$DeadLetterQueue

---

🔹 Example: Reading DLQ in .NET

using Azure.Messaging.ServiceBus;

var client = new ServiceBusClient("<CONNECTION_STRING>");
var receiver = client.CreateReceiver("orders-queue", new ServiceBusReceiverOptions
{
    SubQueue = SubQueue.DeadLetter
});

var messages = await receiver.ReceiveMessagesAsync(maxMessages: 10);

foreach (var msg in messages)
{
    Console.WriteLine($"DLQ: Id={msg.MessageId}, Reason={msg.DeadLetterReason}, Desc={msg.DeadLetterErrorDescription}");
    await receiver.CompleteMessageAsync(msg);
}

💡 Best practice: build a DLQ Processor microservice that runs periodically to inspect and decide:

• Retry → re-queue message
• Discard → log for audit
• Manual intervention → alert support

---

🔄 Retry Strategies

🔹 1. Built-in Retries

• By default, Service Bus will redeliver messages until MaxDeliveryCount is reached (default = 10).
• After that → message goes to DLQ.
• Example: set at queue level

az servicebus queue update -g myRg --namespace-name mysbnamespace -n orders-queue --max-delivery-count 5

---

🔹 2. Application-Level Retries

Sometimes you want fine-grained retry logic inside your worker.

Example: Exponential backoff with Polly:

using Polly;
using Polly.Retry;

var retryPolicy = Policy
    .Handle<Exception>()
    .WaitAndRetryAsync(3, attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)));

processor.ProcessMessageAsync += async args =>
{
    await retryPolicy.ExecuteAsync(async () =>
    {
        var body = args.Message.Body.ToString();
        Console.WriteLine($"Processing: {body}");
        
        // Simulate transient error
        if (body.Contains("FAIL")) throw new Exception("Simulated failure");

        await args.CompleteMessageAsync(args.Message);
    });
};

This way, transient failures (e.g., DB locked, API timeout) don’t unnecessarily move messages to DLQ.

---

🔹 3. Poison Message Handling

• If a message always fails (bad schema, invalid data), retries won’t help.
• Strategy:
  • Fail fast (move to DLQ after N retries).
  • Notify support via Azure Monitor alert.
  • Manually fix or requeue after patching bug.

---

📊 Monitoring with Application Insights

To run production systems, you need visibility. Luckily, Service Bus integrates easily with Azure Monitor + App Insights.

---

🔹 1. Enable Diagnostic Logs

In Azure Portal → Service Bus Namespace → Monitoring → Diagnostic settings

• Send logs to Application Insights (or Log Analytics).
• Categories:
  • OperationalLogs (management events)
  • AuditLogs (access)
  • RuntimeLogs (message metrics)

---

🔹 2. Add App Insights to .NET Services

In API / Worker projects:

dotnet add package Microsoft.ApplicationInsights.AspNetCore
dotnet add package Microsoft.ApplicationInsights.WorkerService

Program.cs

builder.Services.AddApplicationInsightsTelemetry();

Now, all logs and dependencies flow into App Insights.

---

🔹 3. Track Custom Telemetry

In OrderApi (producer):

var telemetry = new TelemetryClient();
telemetry.TrackEvent("OrderQueued", new Dictionary<string, string>
{
    { "OrderId", order.Id.ToString() },
    { "Product", order.Product }
});

In OrderWorker (consumer):

telemetry.TrackDependency("ServiceBus", "ProcessOrder", order.Id.ToString(),
    DateTimeOffset.Now, TimeSpan.FromMilliseconds(200), success: true);

---

🔹 4. Query Failures with KQL

In App Insights Logs:

traces
| where message contains "FAILED"
| project timestamp, message, customDimensions

To see DLQ counts:

customEvents
| where name == "OrderDeadLettered"
| summarize Count = count() by customDimensions.Reason

---

✅ Best Practices Recap

• Always monitor DLQs — they are not trash bins, they’re insights.
• Configure MaxDeliveryCount wisely (don’t retry forever).
• Use Polly or Retry policies for transient failures.
• Use custom telemetry to correlate messages across microservices.
• Set up alerts in Azure Monitor (e.g., DLQ > 0 messages = send email/Teams alert).
• Keep consumers idempotent — safe against duplicate deliveries.

---

🎯 Conclusion

In this part, we learned how to:

• Handle Dead Letter Queues (DLQ)
• Implement retry strategies (built-in + custom)
• Add monitoring with Application Insights

Together, these make your system resilient, observable, and production-ready.