Google Pub/Sub

Google Pub/Sub is the GCP managed service for real-time stream processing

Apica's LogFlow can be used to push data to a PubSub topic with just a few clicks. You can forward data as is, morph the data by stripping down attributes or pick and choose the fields that you want to be sent to the Google PubSub topic

Pre-requisites

You will need the following before you can configure forwarding to Google PubSub

1. Name of the Google PubSub topic

2. Project id ( full name not an integer ) for your GCP account project where you created the topic

3. Service account access JSON

The service account must have pubsub.topics.publish permissions

You can check your service account json permissions with gcloud cli

$ gcloud auth activate-service-account --project=my-project-123 \
--key-file=sevice_account_key_file.json
$ gcloud pubsub topics publish my-gcp-topic-1 --message="hello"
messageIds:
- '6470211179713996'
$ gcloud pubsub subscriptions pull my-gcp-topic-1-sub --auto-ack
┌───────┬──────────────────┬──────────────┬────────────┬──────────────────┐
│  DATA │    MESSAGE_ID    │ ORDERING_KEY │ ATTRIBUTES │ DELIVERY_ATTEMPT │
├───────┼──────────────────┼──────────────┼────────────┼──────────────────┤
│ hello │ 6470211179713996 │              │            │                  │
└───────┴──────────────────┴──────────────┴────────────┴──────────────────┘

Creating the forwarder

We are now ready to create the forwarder

Create Google PubSub forwarder

Enter forwarder configuration

Forwarder configuration

Testing the Google PubSub forwarding

You can now test a data replay with a test stream to see if data went through

Replay data to Google PubSub

Replay status for Google PubSub forwarded data

Visualizing data forwarding to Google PubSub