Invoke On-Prem Logic with AWS AppSync Pub/Sub
Dhaval Nagar / CEO
Cloud adoption and migration are skyrocketing, but that doesn't stop enterprises to continue using or building applications in their private data centers. Hybrid use cases are also growing in popularity to offload some tasks to Cloud services and also support on-prem applications with continuity.
In this post we will look at one of the recent use cases we handled using AppSync Generic Pub/Sub.
Use case
In our use case, we have client applications running on On-Prem instances. Some of the tasks are performed in the AWS Cloud and we need to push the task updates back to the On-Prem applications. We prefer Serverless Infrastructure to basically avoid extra maintenance, pay for the usage and on-demand scaling. We have implemented Serverless-based Infrastructure for use cases like Logistics, Healthcare, IoT, Cloud DevOps, etc.
For our On-Prem Applications to the AWS Cloud data flow, we are already using Serverless APIs with API Gateway, Lambda functions, and DynamoDB. But to send events from Cloud to On-Prem process back we thought of following options:
AWS SQS Queue: SQS is a good option if you want to listen to events in the sequence and maintain the durability of the events. For our use case, we needed more of a "channels" approach, where we are pushing events into separate "channels" and listeners can receive and process events. Usually, it's used the other way around - like sending an event from On-Prem to the AWS Cloud for processing.
Google Cloud Pub/Sub: Google Cloud Pub/Sub is a fully serverless pub/sub service and works as per hour need. We initially had everything working with Cloud Pub/Sub but then thought of the long term management and extra cloud overhead. We needed a service that we could include as part of our IaC and Deployment process easily.
AWS IoT MQTT: AWS IoT MQTT is our current working solution. MQTT is a lightweight messaging protocol and the AWS IoT is fully-serverless. There are two features that is important to our use case: 1) Device ID, on-behalf for a client listener process as connection endpoint 2) Quality of Service (QoS) on whether to send messages with durability or not. QoS 0 means a message is delivered zero or more times and QoS 1 means message is delivered one or more times.
AWS AppSync Generic Pub/Sub: AWS AppSync is something that we are already using in few of our applications. We liked the GraphQL with Real-time events capabilities of AppSync, without doing much of heavy lifting. The Generic Pub/Sub converts the AppSync endpoint as a simple broker. We can easily publish events and can also securely listen to events.
AWS AppSync Generic Pub/Sub
AppSync released the Generic Pub/Sub feature in early 2022 to allow developers to easily create a simple endpoint to leverage the AppSync’s subscriptions feature with no or very little knowledge of GraphQL.
Our use case was very moderate and we are not expected to send large volume of data. Our client application is written in NodeJS and runs as a Docker container. This is also an add-on feature to enable and listen for events from the AWS Cloud application, so we have enough guardrails to ensure that we are not subscribing unexpectedly.
Our listener mechanism is based on the Unique Environment ID and Channel Name, for example, ENV1_process1. This makes the channel name unique for both subscription and publishing.
It was a bit of a challenge to listen to the Pub/Sub events through a NodeJS process, but we figure out the exact message structure and flow. We have a sample application in our GitHub repository for reference.
https://github.com/AppGambitStudio/appsync-websocket-listener
We are using API_KEY authentication for the AppSync Subscriptions. This is workable solution to secure in the private environments where applications are already running behind firewalls and other security mechanism.
API Key expiration is easy to manage through EventBridge CRON rules.
AppSync has a very generous free tier and pricing level once you cross the free tier time duration. Based on calculations, even if our process listeners stay connected throughout the year, we will not be incurring exponential cost.
Let's do a quick calculation for connectivity charges.
50 listener processes, 43800 Minutes Per Month, $0.08 Per Million Minutes
(50*43800*0.08)/1000000 = $0.18 Per Month
2500 Devices, 1500 Minutes Per Month, $0.08 Per Million Minutes
(2500*1500*0.08)/1000000 = $0.30 Per Month
Please visit this AWS Calculator for reference.
In the next post, we will look at how we are using the AWS IoT MQTT for our AWS Cloud to On-Prem implementation.