Filepack is a content-addressed package format with self-describing metadata.
A package is a directory of files and a manifest containing their BLAKE3 hashes. File and directory hashes form a Merkle tree whose root hash, the package fingerprint, uniquely identifies the package. Packages can be signed with Ed25519 and verified against a manifest, fingerprint, or signature.
Packages may contain machine-readable metadata describing their content, allowing programmatic search, preview, playback, and conversion.
filepack is a command-line tool for creating, signing, and verifying
packages, and includes an HTTP server for package upload, download, and
display.
A demo server with a selection of example packages is available at filepack.com.
Filepack is currently experimental. The filepack interface and package format
may change at any time.
Documentation is available at filepack.com/book/.
See installation to get started and Why is metadata important? for some of the higher-level ideas behind filepack.
Filepack supports a number of subcommands, including filepack create to
create a manifest, filepack verify to verify a manifest, and filepack serve
to start an HTTP package server.
See filepack help for supported subcommands and filepack help SUBCOMMAND
for information about a particular subcommand.
Create a manifest.
Recommended lints can be enabled with:
filepack create --deny distributionVerify the contents of a directory against a manifest.
To verify the contents of DIR against DIR/manifest.filepack:
filepack verify DIRIf the current directory contains manifest.filepack, DIR can be omitted:
filepack verifyfilepack verify takes an optional --print flag, which prints the manifest
to standard output if verification succeeds. This can be used in a pipeline to
ensure that the manifest has been verified before proceeding:
filepack verify --print | jqStart HTTP server. The filepack server has limited authentication and no disk usage quota and should be considered experimental.
To serve HTTP on 0.0.0.0:80:
filepack serveTo serve HTTPS on 0.0.0.0:443 with an ACME TLS certificate:
filepack serve --https --domain filepack.exampleSee filepack serve --help for more details.
Files and packages can be uploaded with filepack upload and downloaded with
filepack download.
Packages can be deleted with filepack delete.
filepack manifests are named manifest.filepack and are placed alongside the
files they reference.
Manifests are encoded with deco, a compact binary serialization format, and may
be converted to JSON for inspection or manipulation with filepack manifest.
Manifests, when converted to JSON, are an object with three mandatory keys,
embedded, package, and signatures.
The value of the mandatory embedded key is an object mapping BLAKE3 hashes to
hex-encoded file contents. The package object only includes file names,
hashes, and sizes, but files may be embedded in the manifest archive in the
embedded map.
This is currently only used to embed metadata.filemeta.
The value of the mandatory package key is an object mapping path components
to directory entries. Directory entries may be subdirectories or files. Files
are objects with two fields: hash, the tagged hex BLAKE3 hash of the file,
and size, the length of the file in bytes.
Path components are UTF-8 and may not be . or .., contain the path
separators / or \, contain control characters, or begin with a Windows
drive prefix, such as C:.
The value of the mandatory signatures key is an array of signatures.
Signatures are tagged hex strings that include an Ed25519 key, the package
fingerprint the signature is made over, an optional timestamp, and the
signature itself.
Public keys are Curve25519 points and signatures are Ed25519 signatures made
over the hash of a serialized deco byte string containing the package
fingerprint, which commits to the content of package, and an optional
timestamp.
A manifest converted to JSON over a directory containing the files README.md
and src/main.rs, with a signature:
{
"embedded": {},
"package": {
"README.md": {
"hash": "hash1fc253b84551ce6b00e820a826ac18054dc7f63a318ce62f3175315f5c467a62a",
"size": 11883
},
"src": {
"main.rs": {
"hash": "hash11fa48b95ed335369d45b91af8138bdccd1413364bcdbfa6e9034e8a2cfd6e17f",
"size": 33
}
}
},
"signatures": ["…"]
}The signature is elided for brevity. Signatures are tagged hex strings containing a public key and signature.
Filepack packages may contain a file named metadata.filemeta describing the
package and its contents. If present, metadata.filemeta is also embedded in
the manifest.filepack archive.
Metadata is authored by creating a file named metadata.yaml in the root of a
new package. filepack create then loads metadata.yaml if present, checks
for validity and unknown fields, and writes the deco serialization to
metadata.filemeta in the package root.
metadata.yaml is retained as a human-readable reference and for amending
metadata, but metadata.filemeta is the authoritative source of metadata. For
consumption by scripts and tools, filepack metadata prints the contents of
metadata.filemeta as JSON.
Filepack metadata is intended to be a broadly useful machine and human readable description of the contents of a package, covering personal, distribution, and archival use-cases.
Metadata follows a fixed schema and is not user-extensible. Future versions of
filepack may define new metadata fields, causing verification errors if those
fields are present and invalid according to the new schema.
Please feel free to open an issue with ideas for new metadata fields.
filepack create supports optional lints to check your package for issues:
filepack create --deny distributionThe distribution lint group checks for issues which can cause problems if the
package is intended for distribution, such as non-portable paths that are
illegal on Windows, paths which would conflict on case-insensitive file
systems, and inclusion of junk files such as .DS_Store.
Lint groups and the lints they contain may be printed with:
filepack lintsIndividual lints may be allowed with --allow:
filepack create --deny distribution --allow windows-reserved-filenamefilepack supports the generation of
Curve25519 public/private keypairs,
and the creation and verification of
EdDSA signatures over manifests.
Keypairs are generated with:
filepack keygenWhich creates master.public and master.private files in the keychain
subdirectory of the filepack data directory.
Generated public keys can be printed with:
filepack keySignatures are created with:
filepack signWhich signs the manifest in the current directory with your master key and adds
the signature to the manifest's signatures array. Signatures are made over a
fingerprint hash, recursively calculated from the contents of the manifest.
Signatures embedded in a manifest are verified whenever a manifest is verified. The presence of a signature by a particular public key can be asserted with:
filepack verify --key PUBLIC_KEYWhich will fail if a valid signature for PUBLIC_KEY over the manifest
contents is not present.
Filepack signatures are made over the package fingerprint, which is the hash of a deco object which commits to the files and directories contained in the manifest.
Fingerprints are BLAKE3 hashes, constructed such that it is impossible to produce packages which are different but which have the same fingerprint.
Fingerprints may be used as a globally unique identifier. If two packages have the same fingerprint, they have the same content.
The [DIRECTORY] argument in examples can be omitted, in which case it
defaults to the current directory.
To create a filepack manifest:
filepack create [DIRECTORY]This creates manifest.filepack containing hashes and file sizes of all files
in the DIRECTORY and subdirectories.
To enable linting, use the --deny flag with a lint or lint group:
filepack create --deny <LINT> [DIRECTORY]To view all lints:
filepack lintsPlease feel free to open issues requesting new lints.
To create a package with metadata describing the package, create a file named
metadata.yaml in the package directory, for example:
title: The Necronomicon
creator: Abdul Alhazred
description: >-
The Old Ones, their history, and the rites by which they may be summoned.
language: laThen, to create the package:
filepack create [DIRECTORY]Which will validate metadata if present and write it to metadata.filemeta, as
well as include it in manifest.filepack.
See the book for more information.
To sign a package when creating it, first create a new public and private key pair:
filepack keygenThis creates a new public key, master.public, with corresponding private key,
master.private, in the filepack keychain directory.
The filepack info command will print the path to the filepack data directory,
keychain directory, and any public keys in the keychain directory.
Your public key can be printed with:
filepack keyTo sign a new package:
filepack create --sign [DIRECTORY]To add a signature to an existing package:
filepack signTo print the fingerprint of a package:
filepack fingerprint [DIRECTORY]Manifests contain only hashes and not file content, and so can be published anywhere, including places where technical or legal limitations would preclude publication of the content itself.
Package content can be verified with:
filepack verify [DIRECTORY]Any extra files, missing files, or modified files will produce an error.
This verifies files against hashes stored in the manifest.
Running filepack verify on its own can detect accidental corruption, but not
intentional modification, since an attacker could modify package content, and
then modify the manifest to make it match the modified content and thus pass
verification.
However, if the manifest was obtained from a trusted source, verification will catch any modifications, intentional or otherwise, even if the package content was obtained from an untrusted source.
To detect intentional modification, you can either verify a package against a known-good fingerprint:
filepack verify --fingerprint <FINGERPRINT> [DIRECTORY]This verifies both the contents of the package and that the manifest has the expected fingerprint. The former protects against accidental corruption, and the latter against intentional modification.
Packages can also be verified by checking for a signature by a known public key. This also protects against intentional modification, as long as the private key corresponding to the public key is protected, and only known to you, or someone you trust.
To verify that a package has a signature from a particular public key:
filepack verify --key <PUBLIC_KEY> [DIRECTORY]The --key option can be repeated to require signatures from multiple public
keys.
Create a filepack manifest with:
filepack create <PACKAGE>This will create <PACKAGE>/manifest.filepack
To later verify the package against the manifest:
filepack verify <PACKAGE>Because the manifest contains cryptographic hashes, accidental corruption to
the files or manifest will always be detected by filepack verify.
This is not the case with intentional, malicious corruption, since an attacker could modify the files and replace the manifest hashes with the hashes of the modified files.
Because an attacker could modify the files and replace the manifest hashes with the hashes of the modified files, you must ensure that the manifest has not been tampered with.
This can be accomplished in a number of ways, either by saving the manifest to a secure location, saving the package fingerprint, or signing the package.
To save the manifest in a secure location, use the --manifest option to save
the manifest somewhere other than the package:
filepack create <PACKAGE> --manifest <MANIFEST>Then, verify the package against the saved manifest:
filepack verify <PACKAGE> --manifest <MANIFEST>Because the manifest was protected, any modification to the package will be detected. This has the advantage that not only will any modifications be detected, but which files were modified can also be detected.
Create the manifest in the package root with:
filepack create <PACKAGE>Print the package fingerprint:
filepack fingerprint <PACKAGE>Save the fingerprint in a secure location.
Then, verify the package against the saved fingerprint:
filepack verify <PACKAGE> --fingerprint <FINGERPRINT>Because the fingerprint was protected, any modification to the package will be detected. This has the advantage that you only have to save a small text string, but the disadvantage that while any modifications will be detected, you will not be able to determine which files have changed.
Create the manifest in the package root and sign it with your master key:
filepack create <PACKAGE> --signThen, verify the package and its signature:
filepack verify <PACKAGE> --key masterAny modification to the package or manifest will invalidate the signature, which will be detected. This has the advantage of not needing to save the manifest or fingerprint of packages you want to verify. However, you will need to generate and secure your private key.
To check the authenticity of a package created by someone else, get their public key and verify that the package contains a signature by that key:
filepack verify <PACKAGE> --key <KEY>