Chapter 3: Interprocess Communication in a Microservice Architecture

In a monolith, modules call each other through language-level method calls. In a microservice architecture, services run in separate processes or on separate machines. They must use an interprocess communication (IPC) mechanism to work together. Choosing the right IPC mechanism is one of the most important architectural decisions you will make.

Interaction styles are classified along two dimensions. The first dimension is how many services receive each request: one-to-one or one-to-many. The second dimension is timing: synchronous (the client waits for a response) or asynchronous (the client does not block).

This taxonomy guides the IPC choice. The two main pattern categories are the Remote Procedure Invocation (RPI) pattern for synchronous communication and the Messaging pattern for asynchronous communication.

An API is a contract between a service and its clients. Getting this contract right is important because changes later can break other services.

Use an API-first design approach: define the API before you write any implementation code. Share the draft with client teams, get their feedback, and iterate. This avoids costly rework after coding starts.

The interface definition language (IDL) you use depends on the IPC mechanism:

• REST — Open API Specification (Swagger)
• gRPC — Protocol Buffers definition files
• Messaging — informal documentation of message channels and message types

APIs change over time. In a microservice architecture, you cannot force all clients to update at once because each service is deployed independently. You need a strategy to evolve APIs without breaking existing clients.

Semantic Versioning

Use Semantic Versioning to communicate the nature of changes:

• MAJOR — breaking, incompatible change
• MINOR — new backward-compatible feature
• PATCH — backward-compatible bugfix

The Robustness Principle

The Robustness Principle says: "Be conservative in what you send, be liberal in what you accept." Services should ignore unknown fields in requests and responses. This makes backward-compatible evolution much easier.

Handling Breaking Changes

When a breaking change is unavoidable, you must support multiple API versions simultaneously for a transition period. Two common approaches:

• URL versioning — embed the version in the URL (e.g., /v1/orders, /v2/orders)
• Content negotiation — use HTTP headers to specify the version

The format you choose for encoding messages affects efficiency, readability, and compatibility. There are two broad categories.

Protocol Buffers uses tagged fields, which makes API evolution easier because the reader can skip unknown tags. Avro requires the writer's schema to read data, making evolution slightly harder.

In the Remote Procedure Invocation (RPI) pattern, a client sends a request to a service and waits for a response. This is the most familiar IPC style. Two popular implementations are REST and gRPC.

REST is an IPC mechanism that uses HTTP. It models the domain as resources (e.g., /orders, /customers/123) and uses standard HTTP verbs (GET, POST, PUT, DELETE) to manipulate them.

REST Maturity Model

Leonard Richardson defined four levels of REST maturity:

• Level 0 — clients invoke a single endpoint using POST (just HTTP as transport)
• Level 1 — introduces resources with distinct URLs
• Level 2 — uses HTTP verbs correctly (GET, POST, PUT, DELETE)
• Level 3 — HATEOAS: responses include links that tell the client what it can do next

Benefits of REST

• Simple and familiar to most developers
• Easy to test with tools like curl or Postman
• Firewall-friendly (uses standard HTTP/HTTPS)
• No intermediate broker needed

Drawbacks of REST

• Only supports request/response interaction style
• Fetching data from multiple resources often requires multiple round trips
• Mapping complex operations to HTTP verbs can be awkward

gRPC is a framework for making synchronous and streaming remote calls. It uses Protocol Buffers as its IDL and binary message format, and runs over HTTP/2.

Benefits of gRPC

• Supports rich operation definitions (not limited to HTTP verbs)
• Compact and efficient binary Protocol Buffers format
• Supports bidirectional streaming
• Works across many programming languages

Drawbacks of gRPC

• JavaScript clients in browsers have limited support
• HTTP/2 can cause problems with older firewalls and load balancers

In a distributed system, a service you depend on may be slow or unavailable. This is called a partial failure. If you do nothing, the problem cascades: your service blocks waiting for the broken service, your threads get exhausted, and you become unavailable too.

Three-Mechanism Defense

Use all three mechanisms together to handle partial failure:

1. Network timeouts — never block indefinitely; always set a timeout on remote calls
2. Limit outstanding requests — cap the number of concurrent requests to a service; reject immediately if the limit is reached
3. Circuit Breaker — track the error rate and stop sending requests when failures exceed a threshold

Recovery Strategies

When a remote call fails or the circuit breaker is open, the service can:

• Return an error to the client
• Return cached data from a previous successful call
• Return a default or fallback value
• Omit the non-essential data from the response

In a cloud environment, service instances are created and destroyed dynamically. Their network addresses change constantly. Service discovery is the mechanism that lets a client find the current network location of a service instance.

At the center is a Service Registry: a database of the network locations of all service instances.

Application-Level Discovery

The services themselves handle registration and discovery:

• Self Registration — each service instance registers itself with the registry on startup and deregisters on shutdown
• Client-side Discovery — the client queries the registry and load-balances across available instances

Technologies: Netflix Eureka (registry) + Ribbon (client-side load balancing) + Spring Cloud.

Works across platforms, but requires a discovery library for each programming language you use.

Platform-Provided Discovery

The deployment platform handles registration and discovery:

• 3rd-party Registration — the platform automatically registers instances
• Server-side Discovery — the platform provides a built-in load balancer (e.g., Kubernetes Service, Docker DNS)

No discovery code needed in your services, but it ties you to a specific platform.

In the Messaging pattern, services communicate by exchanging messages asynchronously. The sender does not wait for a response. Messages flow through channels, which may be managed by a message broker or handled directly between services (brokerless).

Message Structure

A message has two parts:

• Header — metadata: message ID, return address, shard key (for ordering)
• Body — the actual data

Message Types

There are three types of messages:

• Document — contains data only, the receiver decides what to do with it
• Command — tells the receiver to perform an operation (like a remote method call)
• Event — notifies that something happened in the sender's domain

Channel Types

• Point-to-point — delivers each message to exactly one consumer (used for commands)
• Publish-subscribe — delivers each message to all subscribed consumers (used for events)

Messaging is flexible enough to implement all the interaction styles from the taxonomy:

• Request / Response — the client sends a command message and includes a return address; the service sends a reply message to that address
• One-way Notification — the client sends a command or document message with no reply expected
• Publish / Subscribe — the service publishes event messages to a publish-subscribe channel; all interested services receive them
• Publish / Async Responses — the client publishes a message with a reply channel header; interested services process it and send responses back on that channel

Unlike REST (Open API) or gRPC (Protocol Buffers), there is no widely adopted standard for documenting messaging-based APIs. You typically create informal documentation that describes:

• The message channels the service uses (both incoming and outgoing)
• The message types and their structure
• The message format (JSON, Protocol Buffers, etc.)

A message broker is an intermediary through which messages flow. Services send messages to the broker, and the broker delivers them to the intended recipients.

Broker-Based Messaging

Most production systems use a broker. Popular brokers include ActiveMQ, RabbitMQ, Apache Kafka, Amazon Kinesis, and Amazon SQS.

• Benefits: buffering (the sender does not need the receiver to be available), loose coupling between services
• Drawbacks: the broker can become a performance bottleneck and a single point of failure; it adds operational complexity

Brokerless Messaging

In brokerless messaging (e.g., ZeroMQ), services communicate directly without an intermediary.

• Benefits: lower latency, no broker single point of failure
• Drawbacks: services need to discover each other, availability is reduced (both sides must be running), guaranteed delivery is harder to implement

To scale message processing, you run multiple instances of a service that all consume from the same channel. These are called competing receivers. The problem is that concurrent consumers may process messages out of order.

Sharded (Partitioned) Channels

The solution is sharded channels (also called partitioned channels). Each message has a shard key (e.g., the order ID). The broker hashes the key to assign the message to a specific shard. Each shard is consumed by exactly one instance.

In Kafka terminology, this is called Consumer Groups: the broker assigns each partition to exactly one consumer in the group.

This guarantees per-shard-key ordering while still enabling horizontal scaling.

Message brokers typically guarantee at-least-once delivery: a message will be delivered, but it might be delivered more than once if a failure occurs. Your service must handle duplicates.

Strategy 1: Idempotent Handlers

If your message handler is naturally idempotent (processing the same message twice has the same effect as processing it once), duplicates are harmless. This only works if the broker also preserves ordering on redelivery.

Strategy 2: Message Tracking

Record each processed message ID in a PROCESSED_MESSAGES table as part of the same database transaction that performs the business logic. If a duplicate arrives, the INSERT into the tracking table fails (duplicate key), and the handler skips it.

A common need is to atomically update a database and publish a message. For example, when creating an order, you want to insert the order into the database and publish an "OrderCreated" event. If either operation fails alone, the system becomes inconsistent.

Transactional Outbox Pattern

MessageRelay Implementations

The MessageRelay component reads messages from the OUTBOX table and publishes them to the broker. Two approaches:

Polling Publisher: The relay periodically queries the OUTBOX table for unpublished messages. Simple to implement, but the polling adds load to the database. It may not work well with NoSQL databases that lack efficient querying.

Transaction Log Tailing: The relay reads the database's transaction log (also called commit log or WAL). When an INSERT into the OUTBOX is found in the log, the relay publishes the corresponding message. This is more sophisticated but very performant because it uses the database's built-in change capture.

Technologies for log tailing: Debezium, Eventuate Tram, DynamoDB Streams.

Rather than using broker client libraries directly, use higher-level frameworks that abstract away the details of messaging infrastructure. These frameworks handle message serialization, channel management, and transactional outbox concerns so you can focus on business logic.

The IPC mechanism you choose has a direct effect on your system's availability. This section explains why synchronous communication hurts availability and how to fix it.

When service A calls service B synchronously, A is available only if B is also available. If B also calls service C, then A is available only if both B and C are available. The overall availability is the product of all dependent service availabilities.

There are three strategies to eliminate synchronous dependencies and improve availability:

Strategy 1: Use Asynchronous Interaction Styles End-to-End

Replace synchronous request/response with asynchronous messaging throughout the system. The service that receives the initial request sends it as a message and responds immediately. Downstream processing happens asynchronously.

This is not always possible, especially for external REST APIs that clients expect to return a result immediately.

Strategy 2: Replicate Data Locally

Each service maintains a local copy of the data it needs from other services. It subscribes to events from those services and keeps its copy up to date. When it needs the data, it reads from its local copy instead of making a synchronous call.

• Benefit: No synchronous dependency at query time
• Drawback: Can require storing large amounts of data; does not solve the problem for write operations

Strategy 3: Return Response Immediately, Finish Processing Later

The service creates the entity in a PENDING state and returns a response immediately to the client. It then validates and completes the operation asynchronously using a saga. When the saga finishes, the entity moves to a final state (e.g., APPROVED or REJECTED).

• Benefit: High availability because the initial call has no synchronous dependencies
• Tradeoff: The client must handle the PENDING state. It needs to either poll for the final result or receive a notification when processing completes.