1 What Is Pub/Sub
GCP(Google Cloud Platform) Pub/Sub is an asynchronous messaging service that decouples the producers and consumers, just like the Kafka and RabbitMQ. Switching to use Pubsub, do never need to host the MQ by ourselves, this saves a lot of effort. If you want to save more, try to use Pubsub Lite.
2 Concepts
2.1 Key Concepts
Core concepts:
- Topic: A named resource to which messages are sent by publishers.
- Subscription: A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application. For more details about subscriptions and message delivery semantics, see the Subscriber Guide.
- Message: The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.
- Publisher: aka producer, who sends the message to Topic;
- Subscriber: aka consumer, who consumes the message from Subscription.
A publisher application creates and sends messages to a topic. Subscriber applications create a subscription to a topic to receive messages from it. Communication can be one-to-many (fan-out), many-to-one (fan-in), and many-to-many, as the following diagram shows.
2.2 Message Flow
The following diagram is an overview of the components in the Pub/Sub system and how messages flow between them:
(1) A publisher application creates a topic in the Pub/Sub service and sends messages to the topic. A message contains a payload and optional attributes that describe the payload content.
(2) The service ensures that published messages are retained on behalf of subscriptions. A published message is retained for a subscription until it is acknowledged by any subscriber consuming messages from that subscription.
(3) Pub/Sub forwards messages from a topic to all of its subscriptions, individually.
(4) A subscriber receives messages either by Pub/Sub pushing them to the subscriber's chosen endpoint, or by the subscriber pulling them from the service.
(5) The subscriber sends an acknowledgement to the Pub/Sub service for each received message.
(6) The service removes acknowledged messages from the subscription's message queue.
2.3 Integrations with GCP Components
The following diagram shows how Pub/Sub can integrate many components of Google Cloud.
3 Pubsub Quickstarts
3.1 gcloud cmd
Use the gcloud cmd:
# create topic
$ gcloud pubsub topics create pkslow-topic
# create subcription
$ gcloud pubsub subscriptions create pkslow-sub --topic=pkslow-topic
# send message
$ gcloud pubsub topics publish pkslow-topic --message="www.pkslow.com"
# receive message
$ gcloud pubsub subscriptions pull pkslow-sub --auto-ack
3.2 client libaries
Pubsub supports multiple languages such as Python, Java, C++, Go. Please search my website to find more about it.
4 ordering
Ordering is a key feature for some scnario. How to implement it?
(1) The producer need to send the message with the ordering key, the message will keep the order with one key.
(2) You need to enable the ordering feature for the subscription.
Please keep in mind the ordering will lower the performance.
However, the Spring Cloud Stream is not fully support the ordering for Pubsub yet, you need to customize your publisher or just use the Google SDK lib.
5 Other
5.1 Monitoring
GCP provides the Cloud Monitoring for us, you can check the unacked message, delivered message etc.
5.2 Autoscaling for consumers
We have two options for autoscaling: Keda and GCP Cloud Monitoring, They are similar. They will provide the custom metrics server on K8s and keep monitoring the account of unacked message count. If the number of unacked message is too large, the Kubernetes HPA will scale up the Deployment with more Pods to consume the message.
6 Sum-up
GCP Pub/Sub is quite handy, more details will come soon...
References: