Knowledge Base

Setting Up the AWS S3 Connector App

Prerequisites

• Ensure you have an AWS account. Create one here if needed.

Installing the S3 Connector

1. Access the S3 Connector App via the apps tab in Narrative's platform.
2. Click the install button on the app's tile. Contact support@narrative.io for installation issues.

Configuring Your S3 Connector Bucket Profile

1. Launch the App: Open the S3 Connector App and select "Connector Settings".
2. Create a New Profile: Click "+ New Profile", then enter your profile name, description, and AWS S3 bucket name.
   • Profile Name: Use a descriptive name, such as "North American Location Data".
   • Description: Detail the use case for easier internal recognition.
   • Bucket Name: The exact name of your S3 bucket (e.g., "narrative-testing-bucket").

Granting Narrative Access

1. Bucket Policy: Add the provided policy to your S3 bucket via the AWS console under the Permissions tab.
2. Bucket Tag: Add the provided Tag Key and Value to your bucket for verification.

Finalizing Setup

1. Test Connection: Ensure the setup is correct by testing the connection in the S3 Connector App.
2. File Configuration: Choose your preferred file type (CSV, TSV, JSON, or Parquet) and configure as necessary for CSV and TSV.

Click "Save And Finish" to complete the setup.

Using your S3 Connector Bucket Profile:

1. Select S3 Connector: In Data Studio's metadata section, or in the Connections tab of a dataset in My Data, select the S3 connector and choose your profile.
2. Multiple Profiles: To send data to multiple buckets, select "Add An Additional Profile" and configure each as needed.

Connection Delivery Settings

• Bucket Prefix: This option specifies the prefix that will be prepended to all delivery files when delivering data for this dataset.
• Deliver All Data: When enabled, this option allows the delivery of historical data currently store in the dataset. If disabled, only new data written to the dataset after the connection has been configured will be delivered.

• Remove Metadata from Output Files: When set to true, the remove_metadata option instructs the S3 Connector to remove all fields beginning with _nio from the delivered data such as source and license information. This is particularly useful for users looking to streamline their datasets by excluding metadata that may not be relevant to their analysis or storage needs.
• Use Delivery Date in Output Path: If enabled, new files will be prefixed with the current date and the snapshotId= prefix will be removed .
• Include Success File: If enabled, an empty file with the name '_SUCCESS' will be written into the output folder to indicate that all files have been successfully written and the delivery is complete.* The _SUCCESS file is created once per delivery to indicate successful completion. If all deliveries are directed to the same folder (for example, if you use the date folder), only one _SUCCESS file will exist, and it will be overwritten with each new delivery. This means that if you have multiple deliveries in one day, the interaction with the "use delivery date" option can cause the _SUCCESS file to be overwritten multiple times within that day.

Confirm the settings and click "Connect" to start receiving data in your specified S3 bucket.

API Configuration

Configuration Options

Below is an example of settings for a given Connection available through the API:

curl --location --request POST 'https://app.narrative.io/openapi/connections' \
--header 'Authorization: Bearer {{Bearer-Token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "dataset_id": 12345,
    "profile_id": "XXXXXXX-XXXXX-XXXXXX",
    "quick_settings": {
        "bucket_prefix": "/my-prefix",
        "historical_data_enabled": true,
        "remove_metadata": false,
        "use_delivery_date": true,
        "include_success_file": true
    }
}'

For further assistance or to enable these advanced settings, please contact support@narrative.io.