Authentication and Authorization

To invoke any API's mentioned in this specification a client must first acquire a token (and the url) to authenticate against the server which serves these API's.

The Xet protocol server uses bearer authentication via a token generated by the Hugging Face Hub (https://huggingface.co).

The following section explains how to acquire such a token.

Token Request API Endpoints

URL Pattern:

https://huggingface.co/api/{repo_type}s/{repo_id}/xet-{token_type}-token/{revision}

Parameters:

• repo_type: Type of repository - model, dataset, or space
• repo_id: Repository identifier in format namespace/repo-name
• token_type: Either read or write.
• revision: Git revision (branch, tag, or commit hash; default to using main)

To understand the distinction for between token_type values read onwards in this document to Token Scope.

Example URLs:

https://huggingface.co/api/models/sentence-transformers/all-MiniLM-L6-v2/xet-read-token/main
https://huggingface.co/api/datasets/HuggingFaceM4/the_cauldron/xet-write-token/v1.1
https://huggingface.co/api/spaces/jsulz/ready-xet-go/xet-read-token/main

HTTP Method: GET

Required Headers:

• Authorization: Bearer token for Hugging Face Hub authentication

Response Format

A JSON encoded object with the following format:

{
   "accessToken": string,
   "exp": number,
   "casUrl": string,
}

• accessToken is the token to be used when invoking API's on the Xet CAS service (any Xet API denoted in this specification)
• exp is the unix timestamp of when this token expires
• casUrl is the API service endpoint URL

Users may assume the "accessToken" and "casUrl" fields lengths have an upper limit of 64000 characters.

Example Response Object

{
   "accessToken": "xet_xxxxxxxxxxx",
   "exp": 1848535668,
   "casUrl": "https://cas-server.xethub.hf.co"
}

Error Handling

HTTP Errors

• 401 Unauthorized: Invalid or missing Hub authentication token
• 403 Forbidden: Insufficient permissions for the requested token type
• 404 Not Found: Repository or revision does not exist

Implementation Example

Here's a basic implementation flow:

1. Make the request:

http GET /api/models/black-forest-labs/FLUX.1-dev/xet-read-token/main Host: huggingface.co Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

1. Parse the response

python endpoint = response_json["casUrl"] access_token = response_json["accessToken"] expiration = response_json["exp"]

1. Use the token with Xet service: Use Bearer authentication with the value for the accessToken key to authenticate with the Xet service at endpoint until expiration time.

2. Token refresh (when needed): Use the same API to generate a new token.

In xet-core we use and recommend to add in 30 seconds of buffer time before the provided expiration time to refresh the token.

Token Scope

Xet tokens can have either a read or a write scope. write scope supersedes read scope and all read scope API's can be invoked when using a write scope token. The type of token issued is determined on the token_type URI path component when requesting the token from the Hugging Face Hub (see above).

Revise API specification for what scope level is required to invoke each API (briefly, only POST /shard and POST /xorb/* API's require write scope).

The scope of the Xet tokens is limited to the repository and ref for which they were issued. To upload or download from different repositories or refs (different branches) clients will need to be issued different tokens.

Token Scope Relative to Hugging Face Hub Authentication Token

When requesting a Xet token from the Hugging Face Hub, you will only receive a Xet token matching the requested parameters if you actually have access to them, based on the access afforded to your Hub authentication token.

If you require a write scope Xet token, then you must request it using a Hugging Face Hub token that has write access to the particular repository and ref that you want to access.

If you request a read scope Xet token, then you must request it using a Hugging Face Hub token that has at least read access to the particular repository and ref you want to access.

If you are using Fine-grained Hugging Face Hub Access Tokens, your tokens must have read or write access to the contents of repositories to be issues read or write Xet tokens respectively.

Security Considerations

• Xet tokens are time-limited and should be refreshed/swapped before expiration
• Store tokens securely and avoid logging them (both Hub authentication tokens and Xet tokens)
• Use read tokens when possible; only request write tokens when necessary

Diagram

sequenceDiagram
   autonumber
   actor C as Client
   participant H as Hugging Face Hub (https://huggingface.co)
   participant CAS as CAS API

   loop Repeat after token expiration
      C->>H: https://huggingface.co/api/{repo_type}s/{repo_id}/xet-{token_type}-token/{revision}
      H->>C: { casUrl, accessToken, exp }

      loop Client invoking CAS API's
         C->>CAS: CAS API's using accessToken and casUrl<br/>(Reconstruction, Xorb & Shard upload, Global Dedupe)
         CAS->>C: API Responses, returns code 401 Unauthorized if token expired
      end
   end