Greenplum Stream Server (GPSS) does not natively poll Amazon S3 for new files continuously; its S3 plugin acts as a batch orchestrator for Greenplum's Foreign tables. To achieve automated ingestion of fresh S3 files, you must wrap GPSS in an event-driven architecture or transition to a true streaming source.
Method 1: Event-Driven Micro-Batching (AWS Native) This approach triggers a GPSS job exactly when a new file lands, keeping your current GPSS S3 architecture intact.
-
S3 Event Notification: Configure your S3 bucket to trigger an event on s3:ObjectCreated:*.
-
SQS Queue (Buffer): Route these events to an Amazon SQS queue to ensure no files are missed if the ingestion server is busy.
-
Worker/Lambda Trigger: Create an AWS Lambda function or a lightweight Python daemon on your GPSS server that listens to the SQS queue.
-
Dynamic GPSS Job: When the worker reads an event, it extracts the new file's specific S3 URI, dynamically generates a gpsscli YAML configuration file for that exact file, and executes gpsscli submit and gpsscli start (or calls the GPSS gRPC API directly).
Method 2: True Continuous Streaming (Kafka Connect) If you want a continuous, always-on stream rather than triggering multiple small gpsscli jobs, replace the GPSS S3 connector with the GPSS Kafka connector.
-
Kafka Connect S3 Source: Deploy Apache Kafka and use the Kafka Connect S3 Source Connector. This connector natively monitors the S3 bucket and streams the contents of newly created files directly into a Kafka topic.
-
GPSS Kafka Job: Reconfigure your GPSS YAML to use Kafka as the source instead of S3.
-
Continuous Run: GPSS runs continuously in the background, listening to the Kafka topic and streaming data into Greenplum the moment Kafka ingests it from S3.
Method 3: Delta Polling (Airflow / Cron) If you prefer a simpler orchestration route without introducing AWS Lambda or Kafka, you can simulate streaming via scheduled micro-batches.
-
Tracking State: Create a script that stores the timestamp of the last successful load in a local file or control table.
-
Scheduled Polling: Run a cron job or Apache Airflow DAG every few minutes (e.g., every 5 minutes).
-
Filter and Load: The script uses the AWS CLI to query S3 for files created after the last timestamp, updates a gpsscli YAML file with the new file paths, and triggers the GPSS load job.