Reading data with AWS tools

Each product's data is at s3://ACCOUNT/PRODUCT/ on the data proxy, https://data.source.coop. Public data needs no sign-in. Access Data on docs.source.coop shows how to read it, anonymously or signed in, with the AWS CLI, DuckDB, GDAL and Python.

This section covers the credentials the CLI gives AWS tools, for restricted data and for writes.

AWS credential_process

1. Sign in once:

   source-coop auth login
   

2. Add a profile to ~/.aws/config:

   [profile source-coop]
   credential_process = source-coop creds
   endpoint_url = https://data.source.coop
   

3. Use AWS tools with that profile:

   aws s3 ls s3://my-account/my-product --profile source-coop
   

When the credentials expire, source-coop creds gets new ones with the session. Run source-coop auth login again only if this fails, for example because the refresh token expired or was revoked.

Output formats

creds and auth login print credentials in one of three formats. Select one with --format:

credential-process (default): JSON for AWS credential_process.

source-coop creds

env: shell export statements. They set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN in your shell.

eval $(source-coop creds --format env)

aws-credentials: an INI profile for ~/.aws/credentials. --profile sets the section name.

source-coop creds --format aws-credentials >> ~/.aws/credentials

[source-coop]
# expires 2026-07-13T12:00:00Z
aws_access_key_id = ...
aws_secret_access_key = ...
aws_session_token = ...

Tip

These credentials are temporary. When they expire, run the command again. Each run adds a new section, and AWS uses the last one, so remove old sections occasionally. We recommend credential_process in ~/.aws/config instead.

Run a tool with exec

exec runs a command with the data proxy as its S3 endpoint and your credentials in its environment, as aws-vault exec does for AWS. It needs no profile, and works for tools that can't use one: GDAL (QGIS, rasterio, geopandas), Rust's object_store (obstore, Polars) and s5cmd.

source-coop exec -- aws s3 ls s3://my-org/my-data/
source-coop exec -- aws s3 cp s3://my-org/my-data/data.parquet .
source-coop exec -- s5cmd cp 's3://my-org/my-data/*.parquet' ./local/
source-coop exec -- gdalinfo /vsis3/my-org/my-data/image.tif
source-coop exec -- python script.py

Everything after -- is the command, flags included. The command's exit code is passed on.

The command gets:

• Credentials: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN and AWS_CREDENTIAL_EXPIRATION.
• The proxy as its S3 endpoint, under each tool's name for it: AWS_ENDPOINT_URL_S3 (the AWS CLI v2.13+ and SDKs), S3_ENDPOINT_URL (s5cmd), AWS_ENDPOINT (object_store, with AWS_ALLOW_HTTP for a plain-HTTP proxy), and AWS_S3_ENDPOINT, AWS_HTTPS and AWS_VIRTUAL_HOSTING=FALSE (GDAL's /vsis3/). Only S3 goes to the proxy: calls a script makes to other AWS services keep their own endpoints.
• A region, AWS_REGION and AWS_DEFAULT_REGION set to us-west-2, only if neither is set already, as signing needs one.

AWS_PROFILE, AWS_DEFAULT_PROFILE, AWS_IGNORE_CONFIGURED_ENDPOINT_URLS and AWS_ENDPOINT_URL are removed, so a profile can't lend other credentials and no other endpoint can take S3 requests away from the proxy. (object_store reads a general AWS_ENDPOINT_URL as its S3 endpoint.) Signed in, AWS_NO_SIGN_REQUEST and AWS_SKIP_SIGNATURE are removed too, so GDAL and object_store sign even if you exported them for public buckets. --role-arn and --proxy-url (or SOURCE_ROLE_ARN and SOURCE_PROXY_URL) select the role and environment, as for creds.

The credentials are fetched once, when the command starts. Cached ones are used if they have at least 15 minutes left. Otherwise exec gets new ones. A command still running when they expire (an hour after they're fetched, unless you signed in with --duration) must be run again. For long jobs, use the credential_process profile, which renews them.

Not signed in, exec still runs the command, unsigned, so it can read public products. It says so on stderr, and:

• tells GDAL and object_store not to sign (AWS_NO_SIGN_REQUEST=YES, AWS_SKIP_SIGNATURE=true);
• hides every other AWS key from the command, because the proxy refuses keys it doesn't know, even for public data: it removes AWS_ACCESS_KEY_ID and the other credential variables, points AWS_SHARED_CREDENTIALS_FILE and AWS_CONFIG_FILE at an empty file, and sets AWS_EC2_METADATA_DISABLED=true.

The AWS CLI and s5cmd sign every request unless given --no-sign-request, and without credentials they stop with Unable to locate credentials or NoCredentialProviders. exec suggests the flag when it's missing:

source-coop exec -- aws --no-sign-request s3 ls s3://cholmes/admin-boundaries/
source-coop exec -- s5cmd --no-sign-request ls s3://cholmes/admin-boundaries/

A product that isn't public is refused with AccessDenied, as it is to anyone not signed in.

If you are signed in but the session can't be renewed (the refresh token expired or was revoked), exec stops with the error instead of running unsigned. Run source-coop auth login again, or source-coop auth logout to read public data unsigned.

Multiple roles

Warning

The Source Cooperative data proxy does not yet support custom roles.

The CLI keeps the credentials of each role separately:

# credentials for a read-only role
source-coop auth login --role-arn reader-role

# credentials for an admin role, kept separately
source-coop auth login --role-arn admin-role

Use --role-arn with creds to select the role:

[profile source-coop]
credential_process = source-coop creds --role-arn reader-role
endpoint_url = https://data.source.coop

[profile source-coop-admin]
credential_process = source-coop creds --role-arn admin-role
endpoint_url = https://data.source.coop