VMware Tanzu Greenplum

 Howto stream AWS S3 files continuous to Greenplum Streaming Server

Timo Burmeister's profile image
Timo Burmeister posted Apr 04, 2024 03:16 AM

We configured the GPSS Server and created a job to copy AWS S3 files with the gpsscli against the GPDB7 Table. This works fine.

But now we are looking for a solution that streams fresh created S3 files automaticly against GPSS.

 

How can we do this?

Ashwin Agrawal's profile image
Broadcom Employee Ashwin Agrawal

Greenplum Stream Server (GPSS) does not natively poll Amazon S3 for new files continuously; its S3 plugin acts as a batch orchestrator for Greenplum's Foreign tables. To achieve automated ingestion of fresh S3 files, you must wrap GPSS in an event-driven architecture or transition to a true streaming source.

Method 1: Event-Driven Micro-Batching (AWS Native) This approach triggers a GPSS job exactly when a new file lands, keeping your current GPSS S3 architecture intact.

  • S3 Event Notification: Configure your S3 bucket to trigger an event on s3:ObjectCreated:*.

  • SQS Queue (Buffer): Route these events to an Amazon SQS queue to ensure no files are missed if the ingestion server is busy.

  • Worker/Lambda Trigger: Create an AWS Lambda function or a lightweight Python daemon on your GPSS server that listens to the SQS queue.

  • Dynamic GPSS Job: When the worker reads an event, it extracts the new file's specific S3 URI, dynamically generates a gpsscli YAML configuration file for that exact file, and executes gpsscli submit and gpsscli start (or calls the GPSS gRPC API directly).

Method 2: True Continuous Streaming (Kafka Connect) If you want a continuous, always-on stream rather than triggering multiple small gpsscli jobs, replace the GPSS S3 connector with the GPSS Kafka connector.

  • Kafka Connect S3 Source: Deploy Apache Kafka and use the Kafka Connect S3 Source Connector. This connector natively monitors the S3 bucket and streams the contents of newly created files directly into a Kafka topic.

  • GPSS Kafka Job: Reconfigure your GPSS YAML to use Kafka as the source instead of S3.

  • Continuous Run: GPSS runs continuously in the background, listening to the Kafka topic and streaming data into Greenplum the moment Kafka ingests it from S3.

Method 3: Delta Polling (Airflow / Cron) If you prefer a simpler orchestration route without introducing AWS Lambda or Kafka, you can simulate streaming via scheduled micro-batches.

  • Tracking State: Create a script that stores the timestamp of the last successful load in a local file or control table.

  • Scheduled Polling: Run a cron job or Apache Airflow DAG every few minutes (e.g., every 5 minutes).

  • Filter and Load: The script uses the AWS CLI to query S3 for files created after the last timestamp, updates a gpsscli YAML file with the new file paths, and triggers the GPSS load job.