pkslow.com 南瓜慢说

  • AllArticles
  • Container
  • Spring
  • Life
  • Cloud
  • Collections
  • About
  • GitHub

  • Search
App Terraform101 English Terraform Middleware config Go Private Kubernetes pkslow Test HTTPS Redis Docker Mac Plan Stream MongoDB Spring DevOps JVM String Map Set List Performance Email Springboot JavaCollections ArrayList Java

GCP Pubsub Introduction - so easy to use

Created on: 2021-06-20 | Category: Cloud | 0 | View: 680

1 What Is Pub/Sub

GCP(Google Cloud Platform) Pub/Sub is an asynchronous messaging service that decouples the producers and consumers, just like the Kafka and RabbitMQ. Switching to use Pubsub, do never need to host the MQ by ourselves, this saves a lot of effort. If you want to save more, try to use Pubsub Lite.

2 Concepts

2.1 Key Concepts

Core concepts:

  • Topic: A named resource to which messages are sent by publishers.
  • Subscription: A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application. For more details about subscriptions and message delivery semantics, see the Subscriber Guide.
  • Message: The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.
  • Publisher: aka producer, who sends the message to Topic;
  • Subscriber: aka consumer, who consumes the message from Subscription.

A publisher application creates and sends messages to a topic. Subscriber applications create a subscription to a topic to receive messages from it. Communication can be one-to-many (fan-out), many-to-one (fan-in), and many-to-many, as the following diagram shows.

2.2 Message Flow

The following diagram is an overview of the components in the Pub/Sub system and how messages flow between them:

(1) A publisher application creates a topic in the Pub/Sub service and sends messages to the topic. A message contains a payload and optional attributes that describe the payload content.

(2) The service ensures that published messages are retained on behalf of subscriptions. A published message is retained for a subscription until it is acknowledged by any subscriber consuming messages from that subscription.

(3) Pub/Sub forwards messages from a topic to all of its subscriptions, individually.

(4) A subscriber receives messages either by Pub/Sub pushing them to the subscriber's chosen endpoint, or by the subscriber pulling them from the service.

(5) The subscriber sends an acknowledgement to the Pub/Sub service for each received message.

(6) The service removes acknowledged messages from the subscription's message queue.

2.3 Integrations with GCP Components

The following diagram shows how Pub/Sub can integrate many components of Google Cloud.

3 Pubsub Quickstarts

3.1 gcloud cmd

Use the gcloud cmd:

# create topic
$ gcloud pubsub topics create pkslow-topic

# create subcription
$ gcloud pubsub subscriptions create pkslow-sub --topic=pkslow-topic

# send message
$ gcloud pubsub topics publish pkslow-topic --message="www.pkslow.com"

# receive message
$ gcloud pubsub subscriptions pull pkslow-sub --auto-ack

3.2 client libaries

Pubsub supports multiple languages such as Python, Java, C++, Go. Please search my website to find more about it.

4 ordering

Ordering is a key feature for some scnario. How to implement it?

(1) The producer need to send the message with the ordering key, the message will keep the order with one key.

(2) You need to enable the ordering feature for the subscription.

Please keep in mind the ordering will lower the performance.

However, the Spring Cloud Stream is not fully support the ordering for Pubsub yet, you need to customize your publisher or just use the Google SDK lib.

5 Other

5.1 Monitoring

GCP provides the Cloud Monitoring for us, you can check the unacked message, delivered message etc.

5.2 Autoscaling for consumers

We have two options for autoscaling: Keda and GCP Cloud Monitoring, They are similar. They will provide the custom metrics server on K8s and keep monitoring the account of unacked message count. If the number of unacked message is too large, the Kubernetes HPA will scale up the Deployment with more Pods to consume the message.

6 Sum-up

GCP Pub/Sub is quite handy, more details will come soon...


References:

Autoscaling with Cloud Monitoring


Code for all: GitHub

欢迎关注微信公众号<南瓜慢说>,将持续为你更新...

file

Recommendations:
Cloud Native
Terraform
Container: Docker/Kubernetes
Spring Boot / Spring Cloud
Https
如何制定切实可行的计划并好好执行

  • Author 作者: LarryDpk 南瓜慢说
  • Link 链接: https://www.pkslow.com/archives/gcp-pubsub-en
  • 版权声明: 本博客所有文章除特别声明外,不可转载!
# App # Terraform101 # English # Terraform # Middleware # config # Go # Private # Kubernetes # pkslow # Test # HTTPS # Redis # Docker # Mac # Plan # Stream # MongoDB # Spring # DevOps # JVM # String # Map # Set # List # Performance # Email # Springboot # JavaCollections # ArrayList # Java
App Terraform101 English Terraform Middleware config Go Private Kubernetes pkslow Test HTTPS Redis Docker Mac Plan Stream MongoDB Spring DevOps JVM String Map Set List Performance Email Springboot JavaCollections ArrayList Java
如何安放你的大文件,MongoDB GridFS可以帮助你
Three ways to create GCE on GCP(Console,gcloud,Terraform)
  • Contents
  • Site Overview
南瓜慢说

南瓜慢说

多年Java开发,主要专注后端技术:Java/Spring/Springboot/微服务/大数据等。

多读书,多分享;多写作,多整理。

243 Posts
9 Categories
31 Tags
RSS
0%
© 2020 — 2022 南瓜慢说 pkslow The WebSite keeping alive:   粤ICP备20036375号