Repository navigation
Conversation
Introduces an 'av1_parser' workspace crate that owns all generic AV1 OBU parsing and I/O (ObuReader, IvfWriter, ObuWriter, LEB128, IVF container handling). All existing HEVC functionality is preserved. Commands that process bitstreams (extract-rpu, inject-rpu, convert, remove) now auto-detect the input format: .av1 / .ivf -> new AV1 code path everything else -> existing HEVC code path (unchanged) DoVi-specific AV1 helpers (is_dovi_rpu_obu, extract_dovi_t35_payload, build_dovi_obu) live in src/dovi/av1.rs and use the dolby_vision crate's existing AV1/EMDF API.
The simplified OBU-by-OBU replacement only worked when the input already contained existing DoVi OBUs. Replace it with the proven temporal-unit-aware approach: - IVF path: build_output_frame_av1 inserts RPU right after the OBU_TEMPORAL_DELIMITER and strips any pre-existing DoVi OBUs - Raw AV1 path: accumulate OBUs per temporal unit, inject one RPU per unit regardless of whether an existing OBU is present
…remover Switch inject_ivf_av1 to accept &mut IvfWriter<W> and call write_frame() instead of raw write_ivf_frame_header + write_all, making the IVF injection path consistent with how remover.rs uses IvfWriter throughout.
Owner
|
I'll need to take some time to create a separate crate so that I can maintain it properly. |
|
Thx for it, it works pretty well, i hope it will be in stable at some time. |
Introduces an 'av1_parser' workspace crate that owns all generic AV1 OBU parsing and I/O (ObuReader, IvfWriter, ObuWriter, LEB128, IVF container handling). All existing HEVC functionality is preserved. Commands that process bitstreams (extract-rpu, inject-rpu, convert, remove) now auto-detect the input format: .av1 / .ivf -> new AV1 code path everything else -> existing HEVC code path (unchanged) DoVi-specific AV1 helpers (is_dovi_rpu_obu, extract_dovi_t35_payload, build_dovi_obu) live in src/dovi/av1.rs and use the dolby_vision crate's existing AV1/EMDF API.
The simplified OBU-by-OBU replacement only worked when the input already contained existing DoVi OBUs. Replace it with the proven temporal-unit-aware approach: - IVF path: build_output_frame_av1 inserts RPU right after the OBU_TEMPORAL_DELIMITER and strips any pre-existing DoVi OBUs - Raw AV1 path: accumulate OBUs per temporal unit, inject one RPU per unit regardless of whether an existing OBU is present
…remover Switch inject_ivf_av1 to accept &mut IvfWriter<W> and call write_frame() instead of raw write_ivf_frame_header + write_all, making the IVF injection path consistent with how remover.rs uses IvfWriter throughout.
Three fixes on top of the AV1 support: * trailing_bits(). The Dolby Vision metadata OBU ended right after the byte-aligned T.35 payload, so strict AV1 parsers rejected the unit with 'trailing_one_bit out of range: 0' - FFmpeg's cbs_av1 refuses to read the whole stream. Append the byte-aligned trailing bits (0x80), which leaves the T.35 payload itself untouched and therefore keeps the Dolby round-trip byte identical. * Insert position. The RPU OBU was written directly behind the temporal delimiter, ahead of the sequence header. mkvmerge then treats it as part of the AV1 codec configuration record. It now goes immediately before the first frame OBU, after the sequence header and any static metadata, for both the IVF and the raw path. The raw path buffers the whole temporal unit for that and shares the IVF path's writer. * Codec detection. Recognising AV1 by file extension alone missed piped input, which is how the DDVT scripts feed elementary streams. Detection now sniffs the leading bytes (IVF signature, plausible leading OBU, HEVC start code) and works for stdin as well, by handing the buffered head to whichever path takes over. extract-rpu, convert and remove all use it; DoviProcessor gained read_write_from_reader for the HEVC side. Raw AV1 extraction also counts temporal units rather than RPU OBUs, so --limit means the same thing for both containers. Tests: 39 unit tests pass, 61/73 integration tests - the 12 failures are the 'help' snapshots, which fail identically on unmodified upstream under core.autocrlf.
metadata_insert_index, the trailing bits constant and the codec sniffing are not Dolby Vision specific, and hdr10plus_tool needs exactly the same logic. Keeping one implementation in the shared crate means both tools cannot drift apart. src/dovi/av1.rs keeps only what touches the RPU.
extract-rpu, convert and remove accept an .mkv for HEVC because hevc_parser brings a Matroska reader along. For AV1 the same call failed with 'No HEVC video track found', so the scripts had to demux first even when a container would have done. av1_parser gains a small reader on top of matroska-demuxer: a Matroska block holds exactly the OBUs of one temporal unit, so nothing beyond parsing the OBUs out of the block is needed. matroska_video_codec() says which of the two codecs a container carries, which is what the three commands dispatch on. Matroska input is unwrapped into a raw elementary stream, as on the HEVC side, so the default output of an AV1 track is raw AV1 rather than a copy of the container extension. The per-OBU conversion moved into a helper that the IVF, raw and Matroska paths share. Verified on a 3054 frame AV1 Matroska: extract-rpu finds all 3054 RPUs over 27 scenes, remove strips them and leaves HDR10+ untouched. Tests unchanged at 38 unit and 61/73 integration - the 12 failures are the help snapshots that fail the same way on unmodified upstream.
The AV1-in-Matroska mapping stores a temporal unit without its delimiter, so the OBUs handed out by MatroskaAv1Reader had none. Writing a raw AV1 stream from them produced a file with no temporal unit boundaries left: remuxing it gave a single frame instead of the whole video. The reader now prepends the delimiter when a block does not start with one, which fixes every consumer at once - the extractors never cared, but convert and remove write streams. Caught by remuxing the output and comparing frame counts: 1 before, 3054 after, matching the source.
* extract-rpu on an AV1 stream without Dolby Vision wrote an empty RPU
file and exited 0, which reads as success to anything scripting it. The
HEVC path fails with "No RPU was found in input file"; the AV1 paths,
raw and Matroska, now do the same, before any file is created.
* Command descriptions, the inject-rpu output help and the injector's
refusal named only HEVC. Injection from Matroska or a pipe is refused as
before - the HEVC path refuses it too - but the sentence now says what
is accepted.
Checked from the outside with FFmpeg's CBS parser (trace_headers), the OBU
order of every temporal unit, and round trips, for raw OBU, IVF, Matroska
and piped input; convert -m 0 and -m 2 give the RPUs extract-rpu -m gives.
The failing unit tests are the same twelve that fail on unmodified
upstream on Windows (".exe" in usage lines).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The metadata OBUs this branch wrote ended right after the T.35 payload, without trailing_bits(), so strict parsers reject them: dav1d reports a malformed ITU-T T.35 message, FFmpeg's av1_frame_merge refuses the unit outright. They also sat ahead of the sequence header, where mkvmerge copies them into the av1C record. Both were fixed on feat/hevc-av1, along with codec detection on piped input, Matroska input for extract and remove, and the smaller things in its last commit; this brings that state here unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
|
Superseded by #424, which moves AV1 into a separate crate as you asked. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds AV1 bitstream support to
dovi_toolalongside the existing HEVC pipeline — no HEVC code is removed or modified.av1_parserworkspace crate owns all AV1 I/O:ObuReader,IvfWriter,ObuWriter, and frame-level read/write helpersis_dovi_rpu_obu,extract_dovi_t35_payload,build_dovi_obu) live insrc/dovi/av1.rsextract-rpuandremovealso take Matroska and piped input; injection, as for HEVC, needs a raw or IVF filetrailing_bits()(AV1 spec 5.3.4) and sit after the sequence header, right before the first frame; the output passes FFmpeg's CBS parser (-bsf:v trace_headers)Commands extended
extract-rpu.bininject-rpuremoveconvertDesign
av1_parsercrate is responsible for all AV1 I/ORelates to #387