CLI guide
GitHub

Reading data with AWS tools

Each product's data is at s3://ACCOUNT/PRODUCT/ on the data proxy, https://data.source.coop. Public data needs no sign-in. Access Data on docs.source.coop shows how to read it, anonymously or signed in, with the AWS CLI, DuckDB, GDAL and Python.

This section covers the credentials the CLI gives AWS tools, for restricted data and for writes.

AWS credential_process

  1. Sign in once:

    source-coop auth login

  2. Add a profile to ~/.aws/config:

    [profile source-coop]
credential_process = source-coop creds
endpoint_url = https://data.source.coop

  3. Use AWS tools with that profile:

    aws s3 ls s3://my-account/my-product --profile source-coop

When the credentials expire, source-coop creds gets new ones with the session. Run source-coop auth login again only if this fails, for example because the refresh token expired or was revoked.

aws s3 ls through credential_process, then with creds --format env

Output formats

creds and auth login print credentials in one of three formats. Select one with --format:

credential-process (default): JSON for AWS credential_process.

source-coop creds

env: shell export statements. They set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN in your shell, and point S3 and STS at the data proxy (--proxy-url), so the AWS CLI and SDKs need no --endpoint-url:

eval $(source-coop creds --format env)
aws s3 ls s3://ACCOUNT/PRODUCT/

After the credentials, it prints:

export AWS_REGION=us-west-2
export AWS_ENDPOINT_URL_S3=https://data.source.coop
export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts

Only S3 and STS go to the proxy; other AWS services keep their usual endpoints. The region is there because some SDKs won't run without one: the proxy accepts any. The AWS CLI v2 (2.13 or later), boto3 (1.28 or later), s3fs, obstore and polars read AWS_ENDPOINT_URL_S3. GDAL and DuckDB don't, so give them the endpoint their own way.

aws-credentials: an INI profile for ~/.aws/credentials. --profile sets the section name.

source-coop creds --format aws-credentials >> ~/.aws/credentials

[source-coop]
# expires 2026-07-13T12:00:00Z
aws_access_key_id = ...
aws_secret_access_key = ...
aws_session_token = ...

Tip

These credentials are temporary. When they expire, run the command again. Each run adds a new section, and AWS uses the last one, so remove old sections occasionally. We recommend credential_process in ~/.aws/config instead.

Multiple roles

Warning

The Source Cooperative data proxy does not yet support custom roles.

The CLI keeps the credentials of each role separately:

# credentials for a read-only role
source-coop auth login --role-arn reader-role

# credentials for an admin role, kept separately
source-coop auth login --role-arn admin-role

Use --role-arn with creds to select the role:

[profile source-coop]
credential_process = source-coop creds --role-arn reader-role
endpoint_url = https://data.source.coop

[profile source-coop-admin]
credential_process = source-coop creds --role-arn admin-role
endpoint_url = https://data.source.coop