Data harvests

A data harvest is a ZIP archive of the current account's data. Fluxer builds it in the background while you poll its status, then you download it.

Every route except Download data harvest archive needs a user token. Bot and OAuth2 tokens get 403.

You can download a completed archive until download_url_expires_at, seven days after it is written. After that, Get data harvest download URL returns 400 HARVEST_EXPIRED and the status stays completed.

Archive contents

The archive has these files.

user.json with the account, its connections without credentials and its thread memberships
channels/{channel_id}/messages.json for each channel with a collected message, oldest first
payments/payment_history.json
integrations/oauth.json with the applications the account owns
account/security.json
the avatar and banner under assets/user/

Each message lists its attachments with a data package URL, and the archive doesn't include the files. When the harvest completes, Fluxer emails a download link if the account has an email address and the instance sends email.

Harvest status values

Value	Description
pending	Processing has not started
processing	An attempt is running
completed	The archive is ready for download
failed	The latest attempt failed

When an attempt fails, Fluxer retries it a limited number of times. The retry clears failed_at and error_message, resets progress to 0 and reports processing. After the last attempt fails, progress_step becomes Failed and the status stays failed.

Harvest creation object

Structure

Field	Type	Description
harvest_id	snowflake	The ID of the created harvest
status	string	The harvest status, always `pending`
created_at	ISO8601 timestamp	When the harvest was requested

Harvest status object

Structure

Field	Type	Description
harvest_id	snowflake	The ID of the harvest
status	string	The harvest status
created_at	ISO8601 timestamp	When the harvest was requested
started_at	?ISO8601 timestamp	When the current attempt began
completed_at	?ISO8601 timestamp	When the archive was written
failed_at	?ISO8601 timestamp	When the latest attempt failed
file_size	?string	The archive size in bytes as a decimal string
progress_percent	number	Progress from 0 through 100
progress_step	?string	The harvest progress step
error_message	?string	The latest failure description, for logs only
download_url_expires_at	?ISO8601 timestamp	When the archive stops being downloadable
expires_at	?ISO8601 timestamp	Same as `download_url_expires_at`

Harvest progress steps

Value	Description
Queued	The harvest was created
Starting harvest	An attempt began, with progress 0
Harvesting messages	Message collection started, with progress 5
Harvested N messages	N messages were collected. This step is skipped when there are none
Collecting user metadata	Account data is being collected, with progress 60
Downloading attachments and creating archive	The ZIP archive is being written, with progress 65
Completed	The archive is ready, with progress 100
Failed	The last attempt failed and Fluxer stopped retrying

Harvest download object

Structure

Field	Type	Description
download_url	string	The temporary archive download URL (1-2048 characters)
expires_at	ISO8601 timestamp	Seven days after the URL was issued

Harvest message filter object

The filter chooses which messages a harvest collects, and the rest of the archive is always included.

Structure

Field	Type	Description
scope?	string	Either `selected` or `inaccessible_only` (default `selected`)
include_dms?	boolean	Whether to include open one-to-one DMs (default true)
include_dms_closed?¹	boolean	Whether to include closed one-to-one DMs (default true)
include_group_dms?	boolean	Whether to include group DMs you are still in (default true)
include_guilds?	boolean	Whether to include guilds you are still in (default true)
guild_filter_mode?	string	Either `exclude` or `include_only` (default `exclude`)
excluded_guild_ids?	array[snowflake]	The guilds to leave out in `exclude` mode (max 500)
included_guild_ids?	array[snowflake]	The only guilds to include in `include_only` mode (max 500)
start_date?²	?ISO8601 timestamp	Inclusive lower bound on message time
end_date?²	?ISO8601 timestamp	Exclusive upper bound on message time

¹ Independent of include_dms

² Compared with the message ID's snowflake timestamp. When both are set, start_date must be earlier than end_date.

The selected scope needs at least one include_* field set to true. The inaccessible_only scope selects messages in guilds you left or were removed from and in group DMs you left. It ignores the include_* fields, guild_filter_mode and both guild lists, and start_date and end_date still apply. Both scopes skip your personal notes channel.

Request data harvest

POST/v1/users/@me/harvest

Creates an unfiltered harvest and returns a harvest creation object. The request has no body.

Response

Status	Body	Condition
200	harvest creation object	The harvest was requested

The rate limit is 5 requests per 30 minutes, on the user:data:harvest bucket, shared with Request filtered data harvest.

Request filtered data harvest

POST/v1/users/@me/harvest/filtered

Creates a harvest that collects only the messages the filter selects and returns a harvest creation object. With every field at its default, it collects the same messages as Request data harvest except those in guilds and group DMs you have left.

JSON body

A harvest message filter object.

Response

Status	Body	Condition
200	harvest creation object	The harvest was requested
400	error response	A field is invalid, `selected` has no `include_*` field set to true or `start_date` is not before `end_date`

The rate limit is 5 requests per 30 minutes, on the user:data:harvest bucket, shared with Request data harvest.

Get latest data harvest

GET/v1/users/@me/harvest/latest

Returns the newest harvest status object, or null when the account has no harvest.

Response

Status	Body	Condition
200	harvest status object \| null	The latest harvest was returned

The rate limit is 40 requests per 10 seconds, on the user:harvest:latest bucket.

Get data harvest status

GET/v1/users/@me/harvest/{harvestId}

Returns one harvest status object owned by the current account.

Response

Status	Body	Condition
200	harvest status object	The harvest was returned
404	error response	`UNKNOWN_HARVEST` when the harvest doesn't exist or isn't yours

The rate limit is 40 requests per 10 seconds, on the user:harvest:status bucket.

Get data harvest download URL

GET/v1/users/@me/harvest/{harvestId}/download

Issues a temporary download URL for a completed harvest and returns a harvest download object. By default the URL is a presigned object storage URL. When FLUXER_API_PRESIGNED_HARVEST_DOWNLOADS_ENABLED is false, it links to Download data harvest archive.

Response

Status	Body	Condition
200	harvest download object	A URL was issued
400	error response	`HARVEST_NOT_READY`, `HARVEST_FAILED` or `HARVEST_EXPIRED`
404	error response	`UNKNOWN_HARVEST` when the harvest doesn't exist or isn't yours

The rate limit is 10 requests per minute, on the user:harvest:download bucket.

Download data harvest archive

GET/v1/harvest-downloads/{harvestId}

Unauthenticated

Streams the archive of a completed harvest. Use the full URL from Get data harvest download URL, whose token authorizes the request. When presigned downloads are on, this route returns 404.

Query parameters

Field	Type	Description
token	string	The signed download token (1-2048 characters)

Request headers

Field	Type	Description
Range?	string	A byte range, answered with 206 and `Content-Range`

Response

Status	Body	Condition
200	archive bytes	The archive was streamed as `application/zip` called `fluxer-data-{harvestId}.zip`
206	archive bytes	The requested byte range was streamed
404	`Not Found`	The token is invalid or expired, or the harvest is not downloadable
416	`Range Not Satisfiable`	The requested byte range is outside the archive

The rate limit is 60 requests per minute, on the user:harvest:download_file bucket.