Back to Glossary

Pub/Sub Defined

Pub/Sub (short for publish/subscribe) is a messaging technology that facilitates communication between different components in a distributed system. This communication model differs from traditional point-to-point messaging, in which one application sends a message directly to another. Instead, it is an asynchronous and scalable messaging service that separates the services responsible for producing messages from those responsible for processing them.

Is Redis Streams or Redis Pub/Sub is the best choice for your next app? Intro to Redis Streams and Pub/Sub

Understanding Pub/Sub

Pub/Sub is a messaging model that allows different components in a distributed system to communicate with one another. Publishers send messages to a topic, and subscribers receive messages from that topic, allowing publishers to send messages to subscribers while remaining anonymous, though they can be identified by subscribers if they include identifying information in the message payload. The Pub/Sub system ensures that the message reaches all subscribers who are interested in the topic. If configured appropriately, it is a highly scalable and dependable messaging system that can handle large amounts of data. In addition, Pub/Sub allows services to communicate asynchronously with latencies of 1 millisecond with appropriate message size, network conditions, and subscriber processing time, making it highly desirable for fast and modern distributed applications.

How Pub/Sub works

Pub/Sub is fundamentally a simple communication model where a broker receives messages from a publisher and distributes them to one or more subscribers. The messages are then delivered to the subscribers, who interpret them according to the needs of their particular use cases.

They are usually classified under four models based on the number of publishers and subscribers involved in the communication, which include one-to-one, one-to-many, many-to-one, and many-to-many.

Types of Pub/Sub ModelsDescription 
One-to-oneThis model comprises one publisher and one subscriber. Direct messages are sent from the publisher to the subscriber.
One-to-manyThis model comprises one publisher and multiple subscribers. The publisher sends messages to the topic; all interested subscribers receive that message.
Many-to-oneThis model comprises multiple publishers and one subscriber. The subscriber receives messages on a specific topic from multiple publishers.
Many-to-manyThis model comprises multiple publishers and subscribers. Messages are sent to a topic by the publishers, and all subscribers receive the message.

Pub/Sub core concepts

Pub/Sub system consists of several components; some of the main components are described in the table below:

Components Description
1. PublisherPublisher is an application or service that sends messages.
2. SubscriberSubscriber is an application or service that receives messages.
3. TopicTopic is the subject or the information feed. The publisher can push messages to the topic, which will broadcast messages to the subscribers.
4. MessageMessage holds the received or transmitted data throughout the system.
5. BrokerBroker is responsible for guiding the messages throughout the system. It acts as a middleman to establish and exchange communication between the publisher and subscriber. It can hold and a list of topics and their respective subscribers, which helps it route the messages received from the publishers to be sent to appropriate subscribers.
6. RoutingRouting is the process of messages flowing within the system from publishers to subscribers and ensuring message delivery to correct subscribers based on specific subscriptions.

Pub/Sub use cases

The asynchronous integration offered by Pub/Sub increases the system’s overall flexibility and robustness, which enables having various use cases, including:

  1. Real-time messaging and chat: Pub/Sub can create real-time messaging and chat applications, such as in social media platforms, instant messaging apps, and collaborative work environments.
  2. IoT devices: Pub/Sub can be used to link IoT devices to the cloud, where they can communicate with a centralized broker and send and receive data. With this method, massive amounts of data they produce can be gathered and processed, which can later be used for data analysis..
  3. News updates and alerts: Subscribers can receive real-time news updates and alerts. This use case is typical in stock trading platforms, news applications, and emergency response systems.
  4. Distributed computing and microservices: Pub/Sub can be used to build distributed systems and microservices architectures in which different components of an application communicate in a decoupled manner allowing for greater scalability and flexibility.
  5. Event-driven architectures: Pub/Sub supports event-driven architectures, in which various components of an application react to actions taken by other components. It allows flexibility in the application design and simplifies complicated workflows.
  6. Decoupling components and reducing dependencies: Pub/Sub can decouple and reduce dependencies between application components, allowing easier application maintenance over time.
  7. Fan-out processing: The process of sending a single message simultaneously to numerous subscribers is known as fan-out processing. It is used in distributing data or events to numerous consumers. For instance, pub/sub can be used to fan out data to multiple subscribers, each of which can independently process the data in parallel and feed it to multiple downstream systems.
  8. Fan-in processing: The process of combining multiple messages into a single message is known as fan-in processing. It is useful in combining the processing of data from various sources. For instance, pub/sub can gather the data from each component and fan it into a single stream for subsequent processing where multiple components generated data could be aggregated and analyzed.
  9. Refreshing distributed caches: Maintaining consistency across multiple instances of a distributed cache can be difficult. This issue can be resolved using pub/sub, which offers a cache invalidation and refreshing mechanism. A message is published to a pub/sub topic when data in the backend data source is updated, and this causes all instances of the cache to be refreshed. As a result, it lowers the possibility of serving users with out-of-date data and ensures that all cache instances are kept in sync.
  10. Load balancing for reliability: Pub/Sub can distribute workloads across multiple instances by publishing messages to a topic and having multiple subscribers process those messages in parallel. As a result, you can adjust the number of instances according to workload demands, increasing or decreasing the system’s capacity while maintaining high availability.

Types of Pub/Sub services

Pub/Sub services can be broadly categorized into cloud-based, self-hosted, and real-time.

Cloud-based Pub/Sub services

Cloud-based Pub/Sub services offer a messaging infrastructure managed completely by the cloud providers, such as Google Cloud, AWS, and Microsoft Azure. The cloud-based pub/sub system offers high message durability and availability, which makes it suitable for building cloud-native applications that require constant updates in real-time, such as airline applications for checking available tickets or banking applications for checking foreign currency rates. Additionally, with the help of these major cloud providers, you can easily build highly scalable decoupled messaging systems.

Self-hosted Pub/Sub services

Self-hosted Pub/Sub services offer messaging systems that can be customized to meet the specific needs of organizations’ internal applications, such as streamlining data transmission or implementing asynchronous workflows. This provides businesses with greater flexibility and control over their systems, which can be deployed in a localized or cloud environment. Popular examples of self-hosted pub/sub services include RabbitMQ, Apache Kafka, and Apache Pulsar.

Real-time Pub/Sub services

Real-time Pub/Sub services facilitate the sending and receiving of messages in real-time without the need for frequent polling of message queues. This reduces delivery latency and improves the overall user experience. The model also includes security features such as channel encryption, presence detection for system responsiveness, message history, and backup. Real-time pub/sub service systems are best suited for use cases like social media chat applications, online gaming, and live streaming. Examples of real-time pub/sub service systems include Pusher and PubNub.

Pub/Sub compared to other messaging technologies

There are various messaging technologies available, including point-to-point messaging and message queuing, in addition to pub/sub. The choice of messaging technology depends on an individual or organization’s expectations from the application. Typically, the advantages and disadvantages of the system are assessed before deciding on which technology to use based on its intended use case.

Point-to-point messaging

Point-to-point messaging is a basic messaging model in which a sender sends a message to a specific recipient. The recipient can then read the message and respond. This technology is best suited for situations in which the sender and receiver have a direct connection, and the receiver can handle incoming messages in real-time. Some examples of point-to-point messaging are HTTP and TCP/IP.

Message queuing 

Message queuing is a messaging model where messages are queued and processed by consumers. This technology is ideal for scenarios with multiple producers and consumers, where messages need to be delivered in sequence. Examples of message queuing technologies are RabbitMQ and Apache ActiveMQ.

Compared to point-to-point messaging and message queuing, pub/sub messaging offers several benefits. Firstly, pub/sub messaging enables the decoupling of the publisher and subscribers, eliminating the need for publishers to know their subscribers. This allows the system to be scaled more efficiently, as new subscribers can be added without affecting the publisher. Secondly, subscribers can read messages at their own pace thanks to pub/sub messaging’s support for asynchronous processing. This helps to enhance performance by reducing the load on the system. Furthermore, pub/sub messaging enables the use of event-driven architectures, where events trigger actions in the system.

Tutorial: Implement Event-Based Architecture with Redis Enterprise

Pub/Sub integrations

It is evident now that pub/sub messaging is a powerful tool for building distributed systems and implementing event-driven architectures. It is because it offers several integrations with other technologies and services. Some of the most used types of integrations are mentioned below:

Cloud platform integrations: Pub/sub messaging is a service provided by most cloud computing platforms, including Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure, providing features such as message filtering and backup.

Programming languages integrations: Pub/sub messaging supports several programming languages and their libraries, such as Java, Python, and Node.js, enabling easier communication between the application and pub/sub messaging service.

API integrations: Pub/sub messaging allows sending and receiving messages using standard API calls using client libraries for several languages and standard gRPC and REST service API technologies.

Database integrations: Pub/sub messaging can trigger database updates and data processing through event-driven architectures. For example, when a new message is received, pub/sub messaging could cause a database update to be performed.

Third-party service integrations: Pub/sub messaging system allows for several third-party service integrations such as Apache Kafka and Amazon SNS to make the application feature-rich and enable seamless data exchange.

Custom integrations: Pub/sub also allows custom integrations based on the organization’s use cases and requirements to support their in-house applications.

Security and authentication integrations: The pub/sub system integrates with identity and access management systems such as LDAP, OAuth, and SAML to ensure that only authorized users can access sensitive data and help prevent security breaches by alerting the system through a notification so that appropriate actions can be taken.

Monitoring and logging integrations: The pub/sub system is supported by monitoring, alerting, and logging products such as Prometheus and Grafana. It enables businesses to track and monitor their system’s performance and analyze data for insights and optimization.

CI/CD integrations: The integration of CI/CD tools, such as Jenkins and GitLab, allows for building, testing, and application deployment automation. In addition, it also makes it easier to keep track of the progress of your application.

Python Redis Pub/Sub

Python is a programming language that provides a Redis client library, which allows Python developers to interact with Redis, an in-memory data structure store. Redis provides a publish/subscribe (pub/sub) messaging system that allows clients to subscribe to channels and receive messages when messages are published to those channels.

Python can be used to implement pub/sub functionality using Redis by sending PUBLISH commands to publish messages and SUBSCRIBE commands to subscribe to channels. The Redis client library for Python provides a convenient way to handle pub/sub messaging in Python applications.

Commands for handling pub/sub in Redis

CommandExample Use and Description
SUBSCRIBESUBSCRIBE channel [channel …] – Subscribes to the given channels
UNSUBSCRIBEUNSUBSCRIBE [channel [channel …]] – Unsubscribes from the provided channels, or unsubscribes all channels if no channel is given
PUBLISHPUBLISH channel message – Publishes a message to the given channel
PSUBSCRIBEPSUBSCRIBE pattern [pattern …] – Subscribes to messages broadcast to channels that match the given pattern
PUNSUBSCRIBEPUNSUBSCRIBE [pattern [pattern …]] – Unsubscribes from the provided patterns, or unsubscribes from all subscribed patterns if none are given
Credit: Redis in Action

To effectively showcase the feature, it’s more convenient to utilize a helper thread to handle PUBLISHing due to the implementation of PUBLISH and SUBSCRIBE commands on the Python end.

Pub/Sub best Practices in Redis

In the pub/sub pattern, publishers can issue messages to any number of subscribers on a channel. These messages are fire-and-forget, in that if a message is published and no subscribers exists, the message evaporates and cannot be recovered.

Once subscribed to a channel, the client is put into subscriber mode and no commands can be issued by the client. In this way, the client has become read-only. Publishing has no such limitation.

More than one channel can be subscribed to at a time. Let’s start by subscribing to two channels weather and sports using the SUBSCRIBE command.

> SUBSCRIBE weather sports
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "weather"
3) (integer) 1
1) "subscribe"
2) "sports"
3) (integer) 2

In a separate client (a separate terminal for our example) we can publish items to either of these channels. We do this by running the PUBLISH command:

> PUBLISH sports oilers/7:leafs/1
(integer) 1

The first argument is the channel and the second argument is the message. The message can be anything, in this case, it’s an encoded sports score. It returns the number of clients that it will be delivered to. Back in the subscriber mode client, we’ll instantly see the message:

1) "message"
2) "sports"
3) "oilers/7:leafs/1"

The response has three elements: the notice that’s a message, followed by the channel and finally the message itself. The client returns to listening immediately after the message is received. Flipping back to the other we can publish another message:

> PUBLISH weather snow/-4c
(integer) 1

In the other client we’ll see the same format but with another channel and event:

1) "message"
2) "weather"
3) "snow/-4c"

Let’s issue publish to a channel that no one is subscribing to:

> PUBLISH currency CADUSD/0.787
(integer) 0

Since we don’t have any clients listening to the channel currency the return is 0. This message is now gone and clients that subsequently subscribe to the currency channel will not be notified of this message – the message was fired and how it has been forgotten.

Aside from subscribing to individual channels, Redis allows for pattern-based subscriptions. The glob-style patterns and are enabled by the PSUBSCRIBE command:

> PSUBSCRIBE sports:*

This will get messages for any channel that starts with “sports:*”. In the other client, issue the following commands:

> PUBLISH sports:hockey oilers/7:leafs/1
(integer) 1
> PUBLISH sports:basketball raptors/33:pacers/7
(integer) 1
> PUBLISH weather:edmonton snow/-4c
(integer) 0

Note how the first two commands returned 1 versus the last command which has a 0. Even though we have no direct subscribers of sports:hockey or sports:basketball, it is still picked up as being received by the pattern subscription. Back on our subscribed client, we can see the results are returned only for the channels that match the pattern.

1) "pmessage"
2) "sports:*"
3) "sports:hockey"
4) "oilers/7:leafs/1"
1) "pmessage"
2) "sports:*"
3) "sports:basketball"
4) "raptors/33:pacers/7"

This response is a little different from the straight SUBSCRIBE response as it has both the matched pattern (2) and the actual channel named (3).