Select a Workspace

Table Name

Columns

Name Type PK

+ Add Column

Primary-key columns are stored as Iceberg identifier fields and are required (non-null).

Cancel Create Table

Type SFTP AWS S3

Hostname

Username

IAM Role ARN

Public Key — install this in your server's authorized_keys

Copy

Trust Policy — add this to the IAM role above

Copy

Principal (dpfPrincipalArn):
Also attach a permissions policy to the role granting the S3 access DPF needs.

Close Test Connection Create

Spec

Connection

S3 Bucket

S3 Prefix (optional)

⚠ Each run lists at most 5000 objects from this bucket/prefix (oldest first). If it grows past that, newly delivered files can be missed. Add an S3 lifecycle rule to expire or transition older objects out of this prefix to stay under the cap.

Trigger on

Schedule Job complete

Triggering job (spec) This trigger fires when the selected spec's job completes successfully.

Polling frequency Hourly Daily Monthly

Hour of day (UTC, 0–23)

Day of month (1–31)

Don't load the same file twice ⓘ

Pre-processing rules

Post-processing rules

Generated code (read-only — edit the rules above to change it)

selectFiles(files, ctx)

postProcess(sftp, files, ctx)

Close Save

Workspace Name

Description (Optional)

Cancel Create Workspace

Workspace Name

Description

Permission

Created

Cancel Save Changes

Email Address

Access Level Read-only — can view specs, jobs, and query data Full Access — can edit, manage members, and delete

Cancel Add Member

Spec Name *

Description

Source Option * File — upload a sample data file Table(s) — query existing tables

Source Tables *

Search the catalog and select the table(s) the generated query reads from

Target Option * Auto-Infer (AI determines schema) Target Schema File Existing Tables

Target Tables *

Search the catalog and select a table to add it

Sample Data File * (The data file to be processed)

Format Specification File (Optional - describes parsing rules)

Target Schema File * (Defines output column definitions)

Merge Mode ⓘ

Load Sample Data ⓘ

Compute Size Small (1/16 DPU) Large (1 DPU) Controls processing power for the data load job.

Additional AI Instructions Instructions about source parsing, schema/field-mapping guidance, or target loading logic.

Uploading files...

Cancel Create & Start Analysis

Data Files * (Select one or more files to process)

Last Successful Run (Watermark) Override to force this run to reprocess data from an earlier point. Leave as-is to continue from where the last run left off.

Processing...

Cancel Upload & Start Job