> For the complete documentation index, see [llms.txt](https://docs.vega365.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.vega365.com/tandem/recovering-from-recorder-failure/media-recovery-with-backfill.md).

# Media Recovery with Backfill

How the recovered recorder's media database catches up.

## Recovering from a failure

This page follows a single cluster through a complete cycle: both recorders healthy, the secondary fails, the secondary returns, backfill restores what it missed, and the cluster returns to normal.

The example uses **PRIMARYREC1** as the primary and **V33** as the secondary. The sequence is the same in the other direction — backfill is bidirectional, and a failed primary recovers from the secondary the same way.

{% embed url="<https://youtu.be/o6Jx9tz-cJ8>" %}
Backfill of a failed secondary, end to end (abbreviated).
{% endembed %}

### 1. Healthy

With both recorders running, the Clustering **Dashboard** shows the cluster **in sync**: the primary Active, the secondary Standby, both clustering agents running, and a recent configuration sync timestamp on each.

<figure><img src="https://92057524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fi0mCLda4cGtI3Jwb6IQA%2Fuploads%2FPOSuwFmA8jiZvJ1uziwD%2Fafter_failback.png?alt=media&amp;token=9add1118-920d-4b8a-852e-f17d80d33ec7" alt=""><figcaption><p>Clustering Dashboard — in sync</p></figcaption></figure>

The BackFill **Flow Dashboard** shows both servers online, each with its camera count and backfill enabled, and a green flow between them.

<figure><img src="https://92057524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fi0mCLda4cGtI3Jwb6IQA%2Fuploads%2FXDAelrutyvcwAlYIutxF%2Fafter_failback_bf.png?alt=media&amp;token=919f6f4b-334f-42e4-bd01-0220252622ae" alt=""><figcaption><p>Flow Dashboard — both servers online</p></figcaption></figure>

### 2. The secondary fails

When the secondary goes down, the Clustering Dashboard moves the cluster from **In sync** to **Degraded**, and the Degraded counter goes to 1 in this example.

<figure><img src="https://92057524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fi0mCLda4cGtI3Jwb6IQA%2Fuploads%2FxUm6QbUw87LKdOlSDvnl%2Fsec_failed_c_dashboard.png?alt=media&amp;token=fcef7a8d-d147-4118-ab32-672b6c1be59f" alt=""><figcaption><p>Clustering Dashboard — degraded</p></figcaption></figure>

On the failed member, the Tandem Clustering Agent shows Offline, and the configuration sync timestamp is replaced with a dash—nothing is syncing.

The Flow Dashboard shows the same event from the backfill side: one server online, one offline, the failed member's BackFill service **Offline**, its camera count at zero, and the flow between them red.

<figure><img src="https://92057524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fi0mCLda4cGtI3Jwb6IQA%2Fuploads%2FljXYYsUebhPirjosfJCU%2Fsec_failed_bf_dashboard.png?alt=media&amp;token=e841048b-901f-4948-8f2d-f0e2e1a433ae" alt=""><figcaption><p>Flow Dashboard — secondary offline</p></figcaption></figure>

{% hint style="info" %}
**Recording continues throughout.** The surviving member holds every camera and keeps recording. The second copy is lost for as long as the other member is down—and that is what backfill repairs.
{% endhint %}

### 3. The secondary returns and backfill runs

When the failed member comes back, backfill starts. Both servers move to **Syncing**: the source reports **Sending**, the recovering member reports **Syncing**, and the flow between them turns teal.

<figure><img src="https://92057524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fi0mCLda4cGtI3Jwb6IQA%2Fuploads%2FQVd31o2k4FhuCWCQXMHq%2Fbackfill_in_progress.png?alt=media&amp;token=8fcce62f-74d0-4761-be81-9b3d4457b4de" alt=""><figcaption><p>Backfill in progress</p></figcaption></figure>

The flow carries the live detail of the transfer:

| Indicator        | Meaning                                                                                             |
| ---------------- | --------------------------------------------------------------------------------------------------- |
| Throttle label   | The bandwidth limit in force for the current hour — **Unlimited** where no throttling row covers it |
| Progress and ETA | How far the current transfer has got, and how long is left                                          |
| Phase label      | What the transfer is doing right now                                                                |

#### How long backfill takes

Three things determine the duration:

* **The amount of content.** How long the member was down, at the recorded bitrate of the cameras in the cluster.
* **Bandwidth throttling.** The limits in force during the hours the job runs. See [Backfill Configuration](https://docs.vega365.com/tandem/configuration/on-the-management-client/backfill-configuration.md).
* **How long the live media database folder takes to fill and close at the source.** Content cannot be copied from the folder currently being written. That folder usually closes within about an hour, but where cameras record on motion only, and there is little motion to record, it can take several hours to fill.

The third is the one that surprises people. A cluster with quiet, motion-triggered cameras can sit waiting long after the bulk of the transfer is done — not because anything is wrong, but because the last portion of content has not yet been closed at the source.

### 4. Confirming what was restored

Once complete, verify completion in the BackFill **Configuration** tab.&#x20;

<figure><img src="https://92057524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fi0mCLda4cGtI3Jwb6IQA%2Fuploads%2F3rvWk5vfYFhZanr7Yz9I%2Fbf_log.png?alt=media&amp;token=f778edb2-5e9e-4283-a582-000ec118cc8b" alt=""><figcaption><p>Backfill History</p></figcaption></figure>

Use **Download Full List** for the complete history beyond the last ten.

Use From Time and To Time to confirm the outage is covered: the period they span should match the period the recorder was down. The recorder's lifetime **Backfills** count and **Data** total also increase after each job.

### 5. Back to normal

Once backfill completes, both servers return to online on the Flow Dashboard, and the cluster returns to **In sync** on the Clustering Dashboard, with a fresh configuration sync timestamp on both members. The two recorders again hold the same recordings.

<figure><img src="https://92057524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fi0mCLda4cGtI3Jwb6IQA%2Fuploads%2FbhvyardiASHLd8HewYWh%2Fafter_failback.png?alt=media&amp;token=e5c6318c-99e5-4ea5-ac4b-a1e12b1c7ab1" alt=""><figcaption><p>Normal State</p></figcaption></figure>

<figure><img src="https://92057524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fi0mCLda4cGtI3Jwb6IQA%2Fuploads%2FXT9aVTPnZczHP4U7DdrO%2Fafter_failback_bf.png?alt=media&amp;token=83ebe9a1-032b-4d92-957a-fbbe4f7274ee" alt=""><figcaption><p>Backfill is complete</p></figcaption></figure>

### If backfill is not successful

* **The recovering member has no free capacity.** Staged content has nowhere to go. See the staging drive requirements in [System Requirements](https://docs.vega365.com/tandem/system-requirements.md).
* **Backfill is disabled on the recovering member.** Check the **Enable Backfill** toggle in that recorder's settings — see [Backfill Configuration](https://docs.vega365.com/tandem/configuration/on-the-management-client/backfill-configuration.md)
* **Media encryption is enabled.** You can't copy encrypted media between members, and the gap is permanent. See [System Requirements](https://docs.vega365.com/tandem/system-requirements.md).
* **The transfer is slower than expected.** Check the throttling table for the hours the job is running in, and see [#how-long-backfill-takes](#how-long-backfill-takes "mention") above — a job waiting on the live folder at the source is working normally.
