> ## Documentation Index
> Fetch the complete documentation index at: https://withcoral.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Write a custom source spec

> Extend Coral by adding any API or dataset as a queryable source.

Coral ships with [popular data sources](/docs/reference/bundled-sources), but you are not limited to them. Before writing a new spec, check whether a [community source](/docs/reference/community-sources) already exists. If the data source you need is not available out of the box or in the community catalog, you can write a source spec in YAML and import it.

A source spec tells Coral how to connect to an API or read a local dataset, what tables to expose, and what columns each table has. Once added, the source works exactly like a bundled one: same SQL, same Coral metadata tables, same MCP surface.

<Tip>
  Install the [Coral source spec authoring
  skill](/docs/getting-started/installation#skills) with `npx skills add
      withcoral/skills` to help your coding agent write source specs.
</Tip>

## Will this work for my source?

How much effort a custom source takes depends on what you are connecting to.

File-based sources (Parquet, JSONL, JSON, CSV) are the simplest case — define the file location and columns, and the source is ready to query.

API-backed sources vary in difficulty. The source spec model is designed for APIs that have:

* **HTTP credentials Coral can store locally** — an API key, personal access token, service account credential, or OAuth device-code or authorization-code flow that returns an access token for Coral to use on API requests
* **Structured, accessible documentation** — an OpenAPI spec, GraphQL schema, or clearly organized REST docs that describe endpoints, response shapes, authentication, scopes, and OAuth client settings
* **Real data to validate against** — actual records in the account so you can confirm the source is returning the expected results during development

OAuth-backed APIs are good candidates when they support device-code flow or authorization-code flow with a local loopback redirect, such as `http://127.0.0.1:<port>/oauth/callback`. During `coral source add --interactive`, Coral can collect an access token and store it as a source secret. If the provider returns a refresh token, Coral can refresh access later; otherwise users reconnect after the access token expires.

If you are unsure whether your API is a good candidate, ask us in [Discord](https://withcoral.com/discord) before getting started.

## Agent-driven authoring

Source authoring works well as an agent-driven workflow. Point your coding agent at the Coral CLI and the [source spec reference](/docs/reference/source-spec-reference), give it the API docs or describe the endpoints you want, and let it iterate:

1. The agent writes or updates the source spec YAML
2. Lints the file with `coral source lint ./my-source.yaml` to catch errors before installing
3. Adds it with `coral source add --file ./my-source.yaml`
4. Runs strict validation with `coral source test my_source`
5. Inspects `coral.tables`, `coral.table_functions`, `coral.columns`, `coral.filters` and `coral.inputs` to check the shape
6. Refines and repeats until the tables look right

## How to validate your source spec

Use the validation loop while authoring:

1. Run `coral source lint ./my-source.yaml` before installing.
2. Add or update the source with `coral source add --file ./my-source.yaml`.
3. Run strict validation with `coral source test <source_name>`.
4. Query the new tables with `coral sql`.
5. Inspect `coral.tables`, `coral.table_functions`, `coral.columns`, `coral.filters`, and `coral.inputs`.

Add top-level `test_queries` for cheap, read-only checks that should run during validation. For execution behavior, failure handling, and credentialed-source checks, see [Test queries](/docs/reference/source-spec-reference#test-queries).

## Example: local JSONL source

The fastest way to see a custom source working is with a local file.

### 1. Create sample data

```shellscript theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir -p demo-data
cat > demo-data/messages.jsonl <<'EOF'
{"type":"user","session_id":"s1","text":"hello"}
{"type":"assistant","session_id":"s1","text":"world"}
EOF
```

### 2. Write the source spec

Create `local-messages.yaml`:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
name: local_messages
version: 0.1.0
dsl_version: 3
backend: file
test_queries:
  - SELECT * FROM local_messages.messages LIMIT 1
tables:
  - name: messages
    description: Demo messages
    format: jsonl
    source:
      location: file:///absolute/path/to/demo-data/
      glob: "**/*.jsonl"
    columns:
      - name: type
        type: Utf8
      - name: session_id
        type: Utf8
      - name: text
        type: Utf8
```

Replace `/absolute/path/to/demo-data/` with the real absolute path on your machine.

The optional top-level `test_queries` field lets you declare read-only SQL checks that Coral runs during post-install validation and strict `coral source test`. `coral source lint` validates the field without executing the queries. Prefer cheap `SELECT` queries that confirm the source connects correctly and the table mapping looks right.

### 3. Add and validate

```shellscript theme={"theme":{"light":"github-light","dark":"github-dark"}}
coral source add --file ./local-messages.yaml
coral source test local_messages
```

`coral source add --file` validates immediately after install. Post-install validation issues, including failing `test_queries`, are printed as warnings so the source still lands in your workspace. Use `coral source test <NAME>` when you want strict pass/fail verification.

### 4. Query it

```shellscript theme={"theme":{"light":"github-light","dark":"github-dark"}}
coral sql "
  SELECT type, session_id, text
  FROM local_messages.messages
  ORDER BY session_id, type
"
```

## Example: HTTP API source

To connect an HTTP API, the source spec declares the base URL, auth, endpoints, and response shape. If the API has search endpoints, table functions, filters, or nested response fields, see [HTTP table functions](/docs/reference/source-spec-reference#http-table-functions), [Table filters](/docs/reference/source-spec-reference#table-filters), and [Nested field naming](/docs/reference/source-spec-reference#nested-field-naming).

### Basic token auth

Authentication has two parts: `inputs` defines the values Coral collects and stores when you add the source, while `auth` defines how those stored values are used on outgoing HTTP requests. The example below collects `API_TOKEN` as a secret, then sends it as a bearer token. For field-level input and auth rules, see [Source inputs](/docs/reference/source-spec-reference#source-inputs) and [HTTP authentication](/docs/reference/source-spec-reference#http-authentication).

```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
name: demo_api
version: 0.1.0
dsl_version: 3
backend: http
inputs:
  API_BASE:
    kind: variable
    default: https://api.example.com
    hint: Base URL for the API
  API_TOKEN:
    kind: secret
    hint: Bearer token for the API
base_url: "{{input.API_BASE}}"
auth:
  type: HeaderAuth
  headers:
    - name: Authorization
      from: bearer
      key: API_TOKEN
tables:
  - name: messages
    description: Messages from the API
    request:
      method: GET
      path: /messages
    response:
      rows_path:
        - data
    pagination:
      mode: link_header
      page_size:
        default: 50
        max: 100
        query_param: per_page
    columns:
      - name: id
        type: Utf8
      - name: text
        type: Utf8
      - name: created_at
        type: Utf8
```

### OAuth credential setup

When a source should offer OAuth setup, add `credential.methods` to the secret input while keeping runtime `auth` focused on how the stored secret is sent to the API. Use `type: oauth` for device-code or authorization-code setup, and add a `type: source_config` fallback when users can paste an existing token.

For field-level details and examples, see [Credential methods](/docs/reference/source-spec-reference#credential-methods), [OAuth flow and client settings](/docs/reference/source-spec-reference#oauth-flow-and-client-settings), [OAuth redirect URIs](/docs/reference/source-spec-reference#oauth-redirect-uris), and [`HeaderAuth`](/docs/reference/source-spec-reference#headerauth) for `from: bearer` and `from: one_of`.

Keep each method's `hint` scoped to the fields that method collects. If OAuth endpoint URLs need non-secret install-time configuration, declare those values as source variables; see [Source inputs](/docs/reference/source-spec-reference#source-inputs).

For the full set of HTTP response, pagination, auth, input, and column fields, see [HTTP-backed tables](/docs/reference/source-spec-reference#http-backed-tables).

## Tips

* **Lint before installing.** `coral source lint ./my-source.yaml` is a fast local precheck because it does not install the source or require credentials.
* **Start small.** Begin with one table and a few columns. Validate with `coral source test`, check Coral metadata tables, then expand.
* **Use `coral.tables`, `coral.table_functions`, `coral.columns`, `coral.filters`, and `coral.inputs`.** They show you exactly what Coral sees after adding a source — including required filters, search functions, and column metadata.
* **Look at bundled sources for patterns.** The bundled source specs in the repo under `sources/core/` are working examples of HTTP pagination, response parsing, auth, and inputs.