Replace Copies with Hardlinks

Have you recently switched to a setup that supports hardlinks and Instant Moves (Atomic-Moves)? Would you like to replace duplicate files with hardlinks?

If your operating system supports it, you can use jdupes or fclones.

The latest binaries are available for Windows at the links above. For Mac or Linux, you can use a package manager like homebrew, apt, or pacman to install the latest version.

Jdupes

We won't cover every command in this guide.

If you want to learn what else jdupes can do, please read the usage manual.

Performance Considerations

This process can place a heavy load on your system for an extended time, depending on your library size. As duplicates are found and hardlinks are created, the process becomes more efficient because linked files and different-sized files are not repeatedly checked. This means later runs will finish faster.

You can speed this up significantly by using a hash database. This stores information about your files (including their signatures) across jdupes runs, which greatly increases the speed.

Simply add the following option before your directories with a path that is always available and persistent:

-y "/mnt/user/appdata/scripts/media_hash.db"

Note: We don't recommend using jdupes on cloud-based setups.

Basic Jdupes Usage

Here's the basic command structure:

jdupes [options] DIR1 DIR2

Dry Run Example

The example below performs a dry run and shows a summary at the end.

Adjust folder paths to match your directory structure.

Without Hash DatabaseWith Hash Database

jdupes -rMX onlyext:mp4,mkv,avi "/mnt/user/data/torrents/movies/" "/mnt/user/data/media/movies"

jdupes -rMX onlyext:mp4,mkv,avi -y "/mnt/user/appdata/scripts/media_hash.db" "/mnt/user/data/torrents/movies/" "/mnt/user/data/media/movies"

Hardlink All Duplicates

The example below will hardlink all duplicate files without prompting.

Adjust folder paths to match your directory structure.

Without Hash DatabaseWith Hash Database

jdupes -rLX onlyext:mp4,mkv,avi "/mnt/user/data/torrents/movies/" "/mnt/user/data/media/movies"

jdupes -rLX onlyext:mp4,mkv,avi -y "/mnt/user/appdata/scripts/media_hash.db" "/mnt/user/data/torrents/movies/" "/mnt/user/data/media/movies"

Important Warnings

Windows Hardlink Limit

Windows only allows a maximum of 1,023 hardlinks per file.

Quick Mode Risk

The -Q or --quick option only reads each file once, hashes it, and performs comparisons based solely on the hashes. There is a small but real risk of a hash collision, which is why the standard byte-for-byte comparison exists as a safety check. This option bypasses that safety check.

Do not use this option on any data where data loss is unacceptable. You have been warned!

fclones

There are several ways to run fclones. We're only going to show the option we've tested and had the best experience with.

Basic fclones Usage

Here's the basic command structure:

fclones [options] DIR1 DIR2

Usage Example

fclones --one-fs --hidden --follow-links "/mnt/user/data/torrents/movies/" "/mnt/user/data/media/movies"

fclones on unRAID

This is straightforward because two users on our Discord created a bash script to make this as simple as possible.

The All-in-One Script (Recommended)

Use option 1 in the following guide: How to run the unRAID mover for qBittorrent with the Mover Tuning that also will install fclones for you automatically.

How to Install the Standalone fclones on unRAID

We're going to install fclones with the help of a script that will place fclones in /usr/local/bin/.

In your unRAID Dashboard, go to the Settings tab and select User Scripts in the User Utilities section at the bottom.
At the bottom of the User Scripts page, click the ADD NEW SCRIPT button.
A popup will ask you to name the script. For this example, use fclones Auto Installer and click OK.
Click the cogwheel next to the new script in the list and select Edit Script.
Copy and paste the script below into the new window that opens, then click SAVE CHANGES.

unRAID fclones installer - This script will install fclones on unRAID or update the current version.
If you want to keep fclones up to date, you can choose when the script should run to stay current. If you only want to run it once or handle the updates manually, skip this step.

Click Apply.
Click RUN IN BACKGROUND to manually trigger the installation of the fclones Auto Installer.