Write a custom source spec

Coral ships with popular data sources, but you are not limited to them. Before writing a new spec, check whether a community source already exists. If the data source you need is not available out of the box or in the community catalog, you can write a source spec in YAML and import it. A source spec tells Coral how to connect to an API or read a local dataset, what tables to expose, and what columns each table has. Once added, the source works exactly like a bundled one: same SQL, same Coral metadata tables, same MCP surface.

Install the Coral source spec authoring skill with npx skills add withcoral/skills to help your coding agent write source specs.

Will this work for my source?

How much effort a custom source takes depends on what you are connecting to. File-based sources (Parquet, JSONL, JSON, CSV) are the simplest case — define the file location and columns, and the source is ready to query. API-backed sources vary in difficulty. The source spec model is designed for APIs that have:

HTTP credentials Coral can store locally — an API key, personal access token, service account credential, or OAuth device-code or authorization-code flow that returns an access token for Coral to use on API requests
Structured, accessible documentation — an OpenAPI spec, GraphQL schema, or clearly organized REST docs that describe endpoints, response shapes, authentication, scopes, and OAuth client settings
Real data to validate against — actual records in the account so you can confirm the source is returning the expected results during development

OAuth-backed APIs are good candidates when they support device-code flow or authorization-code flow with a local loopback redirect, such as http://127.0.0.1:<port>/oauth/callback. During coral source add --interactive, Coral can collect an access token and store it as a source secret. If the provider returns a refresh token, Coral can refresh access later; otherwise users reconnect after the access token expires. If you are unsure whether your API is a good candidate, ask us in Discord before getting started.

Agent-driven authoring

Source authoring works well as an agent-driven workflow. Point your coding agent at the Coral CLI and the source spec reference, give it the API docs or describe the endpoints you want, and let it iterate:

The agent writes or updates the source spec YAML
Lints the file with coral source lint ./my-source.yaml to catch errors before installing
Adds it with coral source add --file ./my-source.yaml
Runs strict validation with coral source test my_source
Inspects coral.tables, coral.table_functions, coral.columns, coral.filters and coral.inputs to check the shape
Refines and repeats until the tables look right

How to validate your source spec

Use the validation loop while authoring:

Run coral source lint ./my-source.yaml before installing.
Add or update the source with coral source add --file ./my-source.yaml.
Run strict validation with coral source test <source_name>.
Query the new tables with coral sql.
Inspect coral.tables, coral.table_functions, coral.columns, coral.filters, and coral.inputs.

Add top-level test_queries for cheap, read-only checks that should run during validation. For execution behavior, failure handling, and credentialed-source checks, see Test queries.

Example: local JSONL source

The fastest way to see a custom source working is with a local file.

1. Create sample data

mkdir -p demo-data
cat > demo-data/messages.jsonl <<'EOF'
{"type":"user","session_id":"s1","text":"hello"}
{"type":"assistant","session_id":"s1","text":"world"}
EOF

2. Write the source spec

Create local-messages.yaml:

name: local_messages
version: 0.1.0
dsl_version: 3
backend: file
test_queries:
  - SELECT * FROM local_messages.messages LIMIT 1
tables:
  - name: messages
    description: Demo messages
    format: jsonl
    source:
      location: file:///absolute/path/to/demo-data/
      glob: "**/*.jsonl"
    columns:
      - name: type
        type: Utf8
      - name: session_id
        type: Utf8
      - name: text
        type: Utf8

Replace /absolute/path/to/demo-data/ with the real absolute path on your machine. The optional top-level test_queries field lets you declare read-only SQL checks that Coral runs during post-install validation and strict coral source test. coral source lint validates the field without executing the queries. Prefer cheap SELECT queries that confirm the source connects correctly and the table mapping looks right.

3. Add and validate

coral source add --file ./local-messages.yaml
coral source test local_messages

coral source add --file validates immediately after install. Post-install validation issues, including failing test_queries, are printed as warnings so the source still lands in your workspace. Use coral source test <NAME> when you want strict pass/fail verification.

4. Query it

coral sql "
  SELECT type, session_id, text
  FROM local_messages.messages
  ORDER BY session_id, type
"

Example: HTTP API source

To connect an HTTP API, the source spec declares the base URL, auth, endpoints, and response shape. If the API has search endpoints, table functions, filters, or nested response fields, see HTTP table functions, Table filters, and Nested field naming.

Basic token auth

Authentication has two parts: inputs defines the values Coral collects and stores when you add the source, while auth defines how those stored values are used on outgoing HTTP requests. The example below collects API_TOKEN as a secret, then sends it as a bearer token. For field-level input and auth rules, see Source inputs and HTTP authentication.

name: demo_api
version: 0.1.0
dsl_version: 3
backend: http
inputs:
  API_BASE:
    kind: variable
    default: https://api.example.com
    hint: Base URL for the API
  API_TOKEN:
    kind: secret
    hint: Bearer token for the API
base_url: "{{input.API_BASE}}"
auth:
  type: HeaderAuth
  headers:
    - name: Authorization
      from: bearer
      key: API_TOKEN
tables:
  - name: messages
    description: Messages from the API
    request:
      method: GET
      path: /messages
    response:
      rows_path:
        - data
    pagination:
      mode: link_header
      page_size:
        default: 50
        max: 100
        query_param: per_page
    columns:
      - name: id
        type: Utf8
      - name: text
        type: Utf8
      - name: created_at
        type: Utf8

OAuth credential setup

When a source should offer OAuth setup, add credential.methods to the secret input while keeping runtime auth focused on how the stored secret is sent to the API. Use type: oauth for device-code or authorization-code setup, and add a type: source_config fallback when users can paste an existing token. For field-level details and examples, see Credential methods, OAuth flow and client settings, OAuth redirect URIs, and HeaderAuth for from: bearer and from: one_of. Keep each method’s hint scoped to the fields that method collects. If OAuth endpoint URLs need non-secret install-time configuration, declare those values as source variables; see Source inputs. For the full set of HTTP response, pagination, auth, input, and column fields, see HTTP-backed tables.

Tips

Lint before installing. coral source lint ./my-source.yaml is a fast local precheck because it does not install the source or require credentials.
Start small. Begin with one table and a few columns. Validate with coral source test, check Coral metadata tables, then expand.
Use coral.tables, coral.table_functions, coral.columns, coral.filters, and coral.inputs. They show you exactly what Coral sees after adding a source — including required filters, search functions, and column metadata.
Look at bundled sources for patterns. The bundled source specs in the repo under sources/core/ are working examples of HTTP pagination, response parsing, auth, and inputs.

Get started

Guides

Reference

Project

Write a custom source spec

Will this work for my source?

Agent-driven authoring

How to validate your source spec

Example: local JSONL source

1. Create sample data

2. Write the source spec

3. Add and validate

4. Query it

Example: HTTP API source

Basic token auth

OAuth credential setup

Tips

​Will this work for my source?

​Agent-driven authoring

​How to validate your source spec

​Example: local JSONL source

​1. Create sample data

​2. Write the source spec

​3. Add and validate

​4. Query it

​Example: HTTP API source

​Basic token auth

​OAuth credential setup

​Tips

Will this work for my source?

Agent-driven authoring

How to validate your source spec

Example: local JSONL source

1. Create sample data

2. Write the source spec

3. Add and validate

4. Query it

Example: HTTP API source

Basic token auth

OAuth credential setup

Tips