Stop testing, start deploying your AI apps. See how with MIT Technology Review’s latest research.
Download now
Pub/Sub (short for publish/subscribe) is a messaging technology that facilitates communication between different components in a distributed system. This communication model differs from traditional point-to-point messaging, in which one application sends a message directly to another. Instead, it is an asynchronous and scalable messaging service that separates the services responsible for producing messages from those responsible for processing them.
Is Redis Streams or Redis Pub/Sub is the best choice for your next app? Intro to Redis Streams and Pub/Sub
Pub/Sub is a messaging model that allows different components in a distributed system to communicate with one another. Publishers send messages to a topic, and subscribers receive messages from that topic, allowing publishers to send messages to subscribers while remaining anonymous, though they can be identified by subscribers if they include identifying information in the message payload. The Pub/Sub system ensures that the message reaches all subscribers who are interested in the topic. If configured appropriately, it is a highly scalable and dependable messaging system that can handle large amounts of data. In addition, Pub/Sub allows services to communicate asynchronously with latencies of 1 millisecond with appropriate message size, network conditions, and subscriber processing time, making it highly desirable for fast and modern distributed applications.
Pub/Sub is fundamentally a simple communication model where a broker receives messages from a publisher and distributes them to one or more subscribers. The messages are then delivered to the subscribers, who interpret them according to the needs of their particular use cases.
They are usually classified under four models based on the number of publishers and subscribers involved in the communication, which include one-to-one, one-to-many, many-to-one, and many-to-many.
| Types of Pub/Sub Models | Description |
| One-to-one | This model comprises one publisher and one subscriber. Direct messages are sent from the publisher to the subscriber. |
| One-to-many | This model comprises one publisher and multiple subscribers. The publisher sends messages to the topic; all interested subscribers receive that message. |
| Many-to-one | This model comprises multiple publishers and one subscriber. The subscriber receives messages on a specific topic from multiple publishers. |
| Many-to-many | This model comprises multiple publishers and subscribers. Messages are sent to a topic by the publishers, and all subscribers receive the message. |
Pub/Sub system consists of several components; some of the main components are described in the table below:
| Components | Description |
| 1. Publisher | Publisher is an application or service that sends messages. |
| 2. Subscriber | Subscriber is an application or service that receives messages. |
| 3. Topic | Topic is the subject or the information feed. The publisher can push messages to the topic, which will broadcast messages to the subscribers. |
| 4. Message | Message holds the received or transmitted data throughout the system. |
| 5. Broker | Broker is responsible for guiding the messages throughout the system. It acts as a middleman to establish and exchange communication between the publisher and subscriber. It can hold and a list of topics and their respective subscribers, which helps it route the messages received from the publishers to be sent to appropriate subscribers. |
| 6. Routing | Routing is the process of messages flowing within the system from publishers to subscribers and ensuring message delivery to correct subscribers based on specific subscriptions. |
The asynchronous integration offered by Pub/Sub increases the system’s overall flexibility and robustness, which enables having various use cases, including:
Pub/Sub services can be broadly categorized into cloud-based, self-hosted, and real-time.
Cloud-based Pub/Sub services offer a messaging infrastructure managed completely by the cloud providers, such as Google Cloud, AWS, and Microsoft Azure. The cloud-based pub/sub system offers high message durability and availability, which makes it suitable for building cloud-native applications that require constant updates in real-time, such as airline applications for checking available tickets or banking applications for checking foreign currency rates. Additionally, with the help of these major cloud providers, you can easily build highly scalable decoupled messaging systems.
Self-hosted Pub/Sub services offer messaging systems that can be customized to meet the specific needs of organizations’ internal applications, such as streamlining data transmission or implementing asynchronous workflows. This provides businesses with greater flexibility and control over their systems, which can be deployed in a localized or cloud environment. Popular examples of self-hosted pub/sub services include RabbitMQ, Apache Kafka, and Apache Pulsar.
Real-time Pub/Sub services facilitate the sending and receiving of messages in real-time without the need for frequent polling of message queues. This reduces delivery latency and improves the overall user experience. The model also includes security features such as channel encryption, presence detection for system responsiveness, message history, and backup. Real-time pub/sub service systems are best suited for use cases like social media chat applications, online gaming, and live streaming. Examples of real-time pub/sub service systems include Pusher and PubNub.
There are various messaging technologies available, including point-to-point messaging and message queuing, in addition to pub/sub. The choice of messaging technology depends on an individual or organization’s expectations from the application. Typically, the advantages and disadvantages of the system are assessed before deciding on which technology to use based on its intended use case.
Point-to-point messaging is a basic messaging model in which a sender sends a message to a specific recipient. The recipient can then read the message and respond. This technology is best suited for situations in which the sender and receiver have a direct connection, and the receiver can handle incoming messages in real-time. Some examples of point-to-point messaging are HTTP and TCP/IP.
Message queuing is a messaging model where messages are queued and processed by consumers. This technology is ideal for scenarios with multiple producers and consumers, where messages need to be delivered in sequence. Examples of message queuing technologies are RabbitMQ and Apache ActiveMQ.
Compared to point-to-point messaging and message queuing, pub/sub messaging offers several benefits. Firstly, pub/sub messaging enables the decoupling of the publisher and subscribers, eliminating the need for publishers to know their subscribers. This allows the system to be scaled more efficiently, as new subscribers can be added without affecting the publisher. Secondly, subscribers can read messages at their own pace thanks to pub/sub messaging’s support for asynchronous processing. This helps to enhance performance by reducing the load on the system. Furthermore, pub/sub messaging enables the use of event-driven architectures, where events trigger actions in the system.
Tutorial: Implement Event-Based Architecture with Redis Enterprise
It is evident now that pub/sub messaging is a powerful tool for building distributed systems and implementing event-driven architectures. It is because it offers several integrations with other technologies and services. Some of the most used types of integrations are mentioned below:
Cloud platform integrations: Pub/sub messaging is a service provided by most cloud computing platforms, including Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure, providing features such as message filtering and backup.
Programming languages integrations: Pub/sub messaging supports several programming languages and their libraries, such as Java, Python, and Node.js, enabling easier communication between the application and pub/sub messaging service.
API integrations: Pub/sub messaging allows sending and receiving messages using standard API calls using client libraries for several languages and standard gRPC and REST service API technologies.
Database integrations: Pub/sub messaging can trigger database updates and data processing through event-driven architectures. For example, when a new message is received, pub/sub messaging could cause a database update to be performed.
Third-party service integrations: Pub/sub messaging system allows for several third-party service integrations such as Apache Kafka and Amazon SNS to make the application feature-rich and enable seamless data exchange.
Custom integrations: Pub/sub also allows custom integrations based on the organization’s use cases and requirements to support their in-house applications.
Security and authentication integrations: The pub/sub system integrates with identity and access management systems such as LDAP, OAuth, and SAML to ensure that only authorized users can access sensitive data and help prevent security breaches by alerting the system through a notification so that appropriate actions can be taken.
Monitoring and logging integrations: The pub/sub system is supported by monitoring, alerting, and logging products such as Prometheus and Grafana. It enables businesses to track and monitor their system’s performance and analyze data for insights and optimization.
CI/CD integrations: The integration of CI/CD tools, such as Jenkins and GitLab, allows for building, testing, and application deployment automation. In addition, it also makes it easier to keep track of the progress of your application.
Python is a programming language that provides a Redis client library, which allows Python developers to interact with Redis, an in-memory data structure store. Redis provides a publish/subscribe (pub/sub) messaging system that allows clients to subscribe to channels and receive messages when messages are published to those channels.
Python can be used to implement pub/sub functionality using Redis by sending PUBLISH commands to publish messages and SUBSCRIBE commands to subscribe to channels. The Redis client library for Python provides a convenient way to handle pub/sub messaging in Python applications.
| Command | Example Use and Description |
|---|---|
| SUBSCRIBE | SUBSCRIBE channel [channel …] – Subscribes to the given channels |
| UNSUBSCRIBE | UNSUBSCRIBE [channel [channel …]] – Unsubscribes from the provided channels, or unsubscribes all channels if no channel is given |
| PUBLISH | PUBLISH channel message – Publishes a message to the given channel |
| PSUBSCRIBE | PSUBSCRIBE pattern [pattern …] – Subscribes to messages broadcast to channels that match the given pattern |
| PUNSUBSCRIBE | PUNSUBSCRIBE [pattern [pattern …]] – Unsubscribes from the provided patterns, or unsubscribes from all subscribed patterns if none are given |
To effectively showcase the feature, it’s more convenient to utilize a helper thread to handle PUBLISHing due to the implementation of PUBLISH and SUBSCRIBE commands on the Python end.
In the pub/sub pattern, publishers can issue messages to any number of subscribers on a channel. These messages are fire-and-forget, in that if a message is published and no subscribers exists, the message evaporates and cannot be recovered.
Once subscribed to a channel, the client is put into subscriber mode and no commands can be issued by the client. In this way, the client has become read-only. Publishing has no such limitation.
More than one channel can be subscribed to at a time. Let’s start by subscribing to two channels weather and sports using the SUBSCRIBE command.
> SUBSCRIBE weather sports Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "weather" 3) (integer) 1 1) "subscribe" 2) "sports" 3) (integer) 2
In a separate client (a separate terminal for our example) we can publish items to either of these channels. We do this by running the PUBLISH command:
> PUBLISH sports oilers/7:leafs/1 (integer) 1
The first argument is the channel and the second argument is the message. The message can be anything, in this case, it’s an encoded sports score. It returns the number of clients that it will be delivered to. Back in the subscriber mode client, we’ll instantly see the message:
1) "message" 2) "sports" 3) "oilers/7:leafs/1"
The response has three elements: the notice that’s a message, followed by the channel and finally the message itself. The client returns to listening immediately after the message is received. Flipping back to the other we can publish another message:
> PUBLISH weather snow/-4c (integer) 1
In the other client we’ll see the same format but with another channel and event:
1) "message" 2) "weather" 3) "snow/-4c"
Let’s issue publish to a channel that no one is subscribing to:
> PUBLISH currency CADUSD/0.787 (integer) 0
Since we don’t have any clients listening to the channel currency the return is 0. This message is now gone and clients that subsequently subscribe to the currency channel will not be notified of this message – the message was fired and how it has been forgotten.
Aside from subscribing to individual channels, Redis allows for pattern-based subscriptions. The glob-style patterns and are enabled by the PSUBSCRIBE command:
> PSUBSCRIBE sports:*
This will get messages for any channel that starts with “sports:*”. In the other client, issue the following commands:
> PUBLISH sports:hockey oilers/7:leafs/1 (integer) 1 > PUBLISH sports:basketball raptors/33:pacers/7 (integer) 1 > PUBLISH weather:edmonton snow/-4c (integer) 0
Note how the first two commands returned 1 versus the last command which has a 0. Even though we have no direct subscribers of sports:hockey or sports:basketball, it is still picked up as being received by the pattern subscription. Back on our subscribed client, we can see the results are returned only for the channels that match the pattern.
1) "pmessage" 2) "sports:*" 3) "sports:hockey" 4) "oilers/7:leafs/1" 1) "pmessage" 2) "sports:*" 3) "sports:basketball" 4) "raptors/33:pacers/7"
This response is a little different from the straight SUBSCRIBE response as it has both the matched pattern (2) and the actual channel named (3).
