Media over QUIC - Transfork

3.1.1. Broadcast

A Broadcast is a collection of tracks from a single producer, identified by a unique name within the session.¶

Each subscription is scoped to a single Broadcast and Track within it. A publisher may advertise available broadcasts via an ANNOUNCE message or an out-of-band mechanism.¶

The MoqTransport draft refers to this as "track namespace". I couldn't help but bikeshed.¶

3.1.2. Track

A Track is a series of Groups within a Broadcast, identified by a unique name within the Broadcast.¶

Each subscription is scoped to a single Track and starts/ends at a Group boundary. A subscriber chooses the priority of each subscription, dictating which Track arrives first during congestion.¶

There is currently no way to discover tracks within a broadcast; it must be negotiated out-of-band.¶

3.1.3. Group

A Group is an ordered stream of Frames within a Track.¶

A Group may be served via a QUIC stream or datagram, depending on the subscription. If the Group is dropped for any reason, a subscriber may FETCH it again starting at a given byte offset.¶

3.1.4. Frame

A Frame is a payload of bytes within a Group.¶

Frames currently only provides framing, hence the name. Framing is useful in some applications but can also be redundant or increase memory usage, as the size must be known upfront.¶

Frame will be removed in a future version of the draft and delegated to a higher layer. However, it's useful when being compared to an Object in MoqTransport.¶

4.2.1. Session

The Session stream contains all messages that are session level.¶

The client MUST open a single Session Stream immediately after establishing the QUIC/WebTransport session. The client sends a SESSION_CLIENT message and the server replies with a SESSION_SERVER message.¶

Afterwards, both endpoints MAY send SESSION_INFO messages containing information about the session. The endpoint SHOULD send an updated SESSION_INFO message, such as after a significant change in the session bitrate.¶

The session remains active until the Session Stream is closed or reset by either endpoint.¶

4.2.2. Announce

A publisher can open an Announce Stream to advertise a broadcast. This is optional, as the application determine the broadcast name out-of-band.¶

The publisher MUST start the stream with an ANNOUNCE message. The subscriber MUST reply with an ANNOUNCE_OK message or reset the stream. The announcement is active until the stream is closed or reset by either endpoint.¶

There is currently no expectation that a relay will forward an ANNOUNCE message downstream. A future draft may introduce a mechanism to discover broadcasts matching a prefix.¶

4.2.3. Subscribe

A subscriber can open a Subscribe Stream to request a named track within a broadcast. The Stream Type indicates if the subscription is served via QUIC streams or datagrams.¶

The SUBSCRIBE message contains a requested Broadcast and Track. It also contains prioritization information and a range of Groups, all of which MAY be updated by the subscriber via a SUBSCRIBE_UPDATE message. The subscription is active until the either endpoint closes or resets the stream.¶

The subscriber MUST start a Info Stream with a SUBSCRIBE message followed by any number SUBSCRIBE_UPDATE messages. The publisher MUST reply with an INFO message followed by any number of GROUP_DROPPED messages (streams only). A publisher MAY reset the stream at any point if it is unable to serve the subscription.¶

When QUIC streams are used, the publisher MUST transmit a complete Group Stream or a GROUP_DROPPED message for each Group within the subscription range. This means the publisher MUST transmit a GROUP_DROPPED if a Group Stream is reset. The subscriber SHOULD close the subscription when all GROUP and GROUP_DROP messages have been received, and the publisher MAY close the subscription after all messages have been acknowledged.¶

When QUIC datagrams are used, the publisher MAY transmit a Group/Frame Datagram for each Group within the subscription range. The publisher MUST NOT transmit a GROUP_DROPPED message. The publisher SHOULD close the subscription after all Group/Frame Datagrams have been transmitted, and the subscriber MAY close the subscription after all Groups have been received.¶

4.2.4. Fetch

A subscriber can open a Fetch Stream to receive a single Group at a specified offset. This is primarily used to recover from an abrupt stream termination, causing the truncation of a Group.¶

The subscriber MUST start a Fetch Stream with a FETCH message. The publisher MUST reply with a GROUP message followed by the rest of a Group Stream, except starting at the specified offset.¶

The fetch is active until both endpoints close the stream, or either endpoint resets the stream.¶

4.2.5. Info

A subscriber can open an Info Stream to request information about a track. This is not often necessary as SUBSCRIBE will trigger an INFO reply.¶

The subscriber MUST start the stream with a INFO_REQUEST message. The publisher MUST reply with an INFO message or reset the stream. Both endpoints MUST close the stream afterwards.¶

4.3.1. Group

A publisher creates Group Streams in response to a Subscribe Stream.¶

A Group Stream MUST start with a GROUP message and MAY be followed by any number of FRAME messages. An application MAY use zero length Group or Frames to signal gaps.¶

Both the publisher and subscriber MAY reset the stream at any time. When a Group stream is reset, the publisher MUST send a GROUP_DROP message on the corresponding Subscribe stream. A future version of this draft may utilize reliable reset instead.¶

4.4.1. Group Datagram

A Group Datagram consists of a GROUP message followed by one or more FRAME messages. These contents are identical to a Group Stream.¶

4.4.2. Frame Datagram

A Frame Datagram consists of a GROUP message followed by the Frame Payload. This saves 1-2 bytes by skipping the FRAME header when there's a only single Frame, as the datagram itself contains a length.¶

6.1.1. Group of Pictures

A simple application can ignore the complexity of P/B frames and focus on I-Frames. This is the optimal approach for many encoding configurations.¶

Each I-Frame begins a Group of Pictures (GoP). A GoP is a set of frames that MAY depend on each other and MUST NOT depend on other GoPs. Each frame has a decode order (DTS) and a frame MUST NOT depend on frames with a higher DTS.¶

This perfectly maps to a QUIC stream, as they too are independent and ordered. The easiest way to use MoqTransfork is to send each GoP as a GROUP with each frame as a FRAME, hence the names.¶

Given the nature of QUIC streams and GoPs, all actions are done at Group boundaries. Each SUBSCRIBE starts at a Group to ensure that it starts with an I-Frame. A FETCH can be used to start at a byte offset instead, assuming some of the Group has already been received. Each Group is delivered in decode order ensuring that all frames are decodable (no artifacts).¶

A subscriber can choose the Group Order based on the desired user experience:¶

• SUBSCRIBE order=DESC: Transmits new Groups first to allow skipping, intended for low-latency live streams.¶

• SUBSCRIBE order=ASC: Transmits old Groups first to avoid skipping, intended for VOD and reliable live streams.¶

A publisher or subscriber can skip the remainder of a Group by resetting a Group Stream or by issuing a SUBSCRIBE_UPDATE. This truncates the Group and a subscriber can issue a FETCH if it wants to resume at a byte offset.¶

6.1.2. Layers

An advanced application can subdivide a GoP into layers.¶

The most comprehensive way to do this is with Scalable Video Coding (SVC). There is a base layer and one or more enhancement layers that depend on lower layers. For example, a 4K stream could be broken into 4K, 1080p, and 360p (base) layers. However, SVC has limited support and is complex to encode.¶

Another approach is to use temporal scalability via something like B-pyramids. The frames within a GoP are sub-divided based on their dependencies, intentionally creating a hierarchy. For example, even frames could be prevented from referencing odd frames, creating a base 30fps layer and an enhancement 60fps layer. This is effectively a custom SVC scheme, however it's limited to time (can't change resolution) and the enhancement layer will be significantly smaller than the base layer.¶

The purpose of these layers is to support degrading the quality of the broadcast. A subscriber could limit bandwidth usage by choose to only receive the base layer or a subset of the enhancements layers. During congestion, the base layer can be prioritized while the enhancement layers can be deprioritized/dropped. However, the cost is a small increase in bitrate (10%) as limiting potential references can only hurt the compression ratio.¶

When using MoqTransfork, each layer is delivered as a separate Track. This allows the subscriber to choose which layers to receive and how to prioritize them in SUBSCRIBE. It also enables layers to be prioritized within the layer application, for example Alice's base layer is more important than Bob's enhancement layer.¶

The application is responsible for determining the relationship between layers, since they're unrelated tracks as MoqTransport is concerned. The application could use a catalog to advertise the layers and how to synchronize them, for example based on the Group Sequence.¶

6.1.3. Non-Reference Frames

While not explicitly stated, I believe the complexity in MoqTransport stems almost entirely from a single use-case: the ability to drop individual non-reference frames in the middle of a group.¶

A non-reference frame cannot be referenced by other frames and is effectively a leaf node in the dependency graph. In theory, non-reference frames can be dropped during congestion without affecting the decodability of other frames. This idea sounds good on paper but it's a trap.¶

Good encoders try to avoid non-reference frames because they fundamentally reduce the compression ratio; they are dead-end bits that can't be used for prediction. You can configure the encoder to produce non-reference frames but the result is a higher bitrate for the ability to drop a small amount during congestion. But the main problem is the complexity introduced into the transport, as each frame must be transmitted as an individual QUIC stream based on a dependency graph that is not available to relays, and difficult for both broadcasters and viewers to parse.¶

The ability to drop individual non-reference frames in the middle of a group is an explicit non-goal for MoqTransfork. An alternative is to put them into a separate layer, such that the tail of the layer could be dropped.¶

6.2.1. Frames

Audio samples are very small and for the sake of compression, are grouped into a frame. This depends on the codec and the sample rate but each frame is typically 10-50ms of audio.¶

Audio frames are independent and ordered, again making them a good fit for QUIC streams. Each audio frame can be transmitted as a GROUP with a single FRAME.¶

6.2.2. Groups

It may also be desirable to group audio frames into larger units.¶

For example, if an application wants reliable playback and A/V synchronization, then audio and video could be aligned. An application could then subscribe to video and audio starting at group X for both tracks, instead of trying to maintain a mapping between the two based on timestamp. This is quite common in HLS/DASH as there's no reason to subdivide audio segments at frame boundaries.¶

This can be accomplished with MoqTransfork by using multiple audio FRAMEs within a GROUP. This limits the ability to drop individual audio frames during congestion (tail drop only) but that's up to the application.¶

6.2.3. FEC

Real-time audio applications often use Forward Error Correction (FEC) to conceal packet loss. Audio frames are a good candidate for FEC given that they are small and independent.¶

In an ideal world, FEC would be performed by QUIC based on the properties of the hop. However this is not currently not supported and FEC is left to the application.¶

In MoqTransfork, each FEC packet is transmitted as a separate GROUP with a single FRAME. A real-time subscriber issues a SUBSCRIBE with an aggressive Group Expires value in the milliseconds range. The publisher will drop any Groups that have not been transmitted or acknowledged within this time frame, potentially causing them to be lost.¶

The subscriber can choose if it wants to use datagrams or streams for a subscription. I recommend using streams if the RTT is smaller than Group Expires, as it gives retransmissions a chance even when FEC is used.¶

6.3.1. Catalog

Originally part of the transport itself, the catalog is a list of all tracks within a broadcast. It's since been delegated to the application and is now just another track with a well-known name.¶

The proposed MoQ catalog format supports live updates. It does this by encoding a base JSON blob and then applying JSON patches over time. If the number of deltas becomes too large, the producer can start over with a new base JSON blob.¶

In MoqTransfork, the base and all deltas are a single GROUP. The base is the first FRAME and all deltas are subsequent FRAMEs. The producer can create a new GROUP to start over, repeating the process.¶

6.3.2. Timeline

Another track that is commonly pitched is a timeline track. This records the presentation timestamp of each Group, giving a VOD viewer to seek to a specific time.¶

The timeline track is a single Group containing a Frame for each timestamp. The live nature of the timeline track is great for DVR applications while being concise enough for VOD. Timed metadata would use a similar approach or perhaps leverage this track.¶

6.3.3. Interaction

Another common use-case is to transmit user interactions, such as controller inputs or chat messages. It's up to the application to determine the format and encoding of these messages.¶

Let's take controller input as an example. The application needs to determine its loss vs latency tolerance, as reordering or dropping inputs will lead to a poor user experience.¶

• If you don't want loss, then use a single GROUP with a FRAME per input.¶

• If you don't want latency, then use a GROUP per input with a single FRAME.¶

• If you want a hybrid, then use form of clustering inputs into GROUPs and FRAMEs based on time.¶

A publisher could monitor the session RTT or stream acknowledgements to get a sense of the latency and create Groups accordingly. However, this only applies to the first hop and won't be applicable when relays are involved.¶

6.4.1. Real-Time

Real-time latency is accomplished by prioritizing the most important media during congestion and skipping the rest.¶

This is slightly different from other media protocols which instead opt to drop packets. The end result is similar, but prioritization means utilizing all available bandwidth as determined by the congestion controller. A subscriber or publisher can reset groups to avoid wasting bandwidth on old data.¶

A real-time viewer could issue:¶

SUBSCRIBE track=audio priority=0 order=DESC group_expires=100ms
SUBSCRIBE track=video priority=1 order=DESC group_expires=100ms

In this example, audio is higher priority than video, and newer groups are higher priority than older groups. Suppose a viewer fell behind after a burst of congestion and has to decide which groups to deliver next. This configuration would result in the transmission order:¶

GROUP track=audio sequence=102
GROUP track=audio sequence=101
GROUP track=audio sequence=100
GROUP track=video sequence=5
GROUP track=video sequence=4

The user experience depends on the amount of congestion:¶

• If there's no congestion, all audio and video is delivered.¶

• If there's moderate congestion, the tail of the old video group is dropped.¶

• If there's severe congestion, all video will be late/dropped and some audio groups/frames will be dropped.¶

The value of group_expires is optional. In this example it means that the publisher automatically resets each group 100ms after they are no longer the latest. It's recommended to use the maximum jitter buffer size.¶

6.4.2. Unreliable Live

Unreliable live is a term I made up. Basically we want low latency, but we don't need it at all costs and we're willing to skip some video to achieve it. This is useful for broadcasts where latency is important but so is picture quality.¶

An unreliable live viewer could issue:¶

SUBSCRIBE track=audio priority=0 order=ASC
SUBSCRIBE track=video priority=1 order=DESC group_expires=3s

This example is different from the real-time one in that audio is fully reliable and delivered in order. Of course this is optional and up to the application, as it will result in buffering during significant congestion. If the viewer goes through a tunnel and then comes back online, they won't miss any audio.¶

A key difference is that our jitter buffer is much larger for video, 3s in this example. The player will tolerate up to 3s of latency before it starts skipping past video frames. Note that the group_expires value can be increased during buffering by issuing a SUBSCRIBE_UPDATE.¶

6.4.3. Reliable Live

Reliable live is another term I made up. This is when we have a live stream but primarily care about picture quality. A good example is a sports game where you want to see every frame.¶

A reliable live viewer could issue:¶

SUBSCRIBE track=audio priority=0 order=ASC
SUBSCRIBE track=video priority=0 order=ASC

This will deliver both audio and video in order, and with the same priority. The viewer won't miss any content unless the publisher resets a group. However, this can result in buffering during congestion and provides a similar user experience to HLS/DASH.¶

6.4.4. VOD / DVR

Video on Demand (VOD) and Digital Video Recorder (DVR) both involve seeking backwards in a live stream. MoqTransfork can serve this use-case too, don't worry.¶

A VOD viewer could issue:¶

SUBSCRIBE track=audio priority=0 order=ASC start=345 end=396
SUBSCRIBE track=video priority=0 order=ASC start=123 end=134

The application is responsible for determining the group sequence numbers based on the desired timestamp. This could be done via a timeline track or out-of-band.¶

A subscriber will need a specific end or else it will download too much data at once, as old media is transmitted at network speed and not encode speed. It will need to issue an updated SUBSCRIBE to expand the range as playback continues and the buffer depletes. A subscriber could use SUBSCRIBE_UPDATE, however there are race conditions involved.¶

A DVR player does the same thing but can automatically support joining the live stream. It's perfectly valid to specify a end in the future and it will behave like reliable live viewer once it reaches the live playhead.¶

Alternatively, a DVR player could prefetch the live playhead by issuing a parallel SUBSCRIBE at a lower priority. This would allow playback to immediately continue after clicking the "Go Live" button, canceling or deprioritizing the VOD subscription.¶

SUBSCRIBE track=video priority=0 order=ASC start=123 end=134
SUBSCRIBE track=video priority=1 order=DESC

6.4.5. Upstream

All of these separate viewers could be watching the same broadcast. How is a relay supposed to fetch the content from upstream?¶

MoqTransfork addresses this by providing a Default Track Priority and Default Group Order in the INFO message. This is the intended behavior for the first hop and dictates which viewers are preferred.¶

For example, suppose the producer chooses:¶

INFO track=audio default_priority=0 default_order=DESC
INFO track=video default_priority=1 default_order=DESC

If Alice is watching a VOD and issues:¶

SUBSCRIBE track=audio priority=0 order=ASC
SUBSCRIBE track=video priority=0 order=ASC

If Bob is watching real-time and issues:¶

SUBSCRIBE track=audio priority=0 order=DESC
SUBSCRIBE track=video priority=1 order=DESC

For any congestion on the first mile, then the relay will improve Bob's experience by following the producer's preference. However any congestion on the last mile will always use the viewer's preference.¶

A relay should use the default priority/order only when there's a conflict. If viewers have the same priority/order, then the relay should use the viewer's preference and it can always issue a SUBSCRIBE_UPDATE when this changes.¶

6.5.1. ABR

Virtually all mass fanout use-cases rely on Adaptive Bitrate (ABR) streaming. The idea is to encode the same content at multiple bitrates and resolutions, allowing the viewer to choose based on their unique situation.¶

MoqTransfork unsurprisingly supports this via multiple Tracks, but relies on the application to determine the relationship between them. This is often done via a catalog track that details each track's name, bitrate, resolution, and codec. This includes how a group in one track corresponds to a group in another track. A common approach is to use the same Group Sequence number for all tracks, or perhaps utilize a timeline track to map between Group Sequences and presentation timestamps.¶

The viewer may limit the available tracks based on capabilities or preferences. For example, the device may not support the 4K track since it uses AV1, or the screen size may be too small to justify the bandwidth. This is easy enough to support; just ignore these tracks in the catalog.¶

The primary reason to use ABR is to adapt to changing network conditions. The viewer learns about the estimated bandwidth via the SESSION_INFO message, or by measuring network traffic and can then choose the appropriate track based on bitrate.¶

Transitioning between tracks can be done seamlessly by utilizing prioritization. For example, suppose a viewer is watching the 360p track and wants to switch to 1080p at group 69.¶

A real-time or unreliable live viewer could issue:¶

SUBSCRIBE_UPDATE track=360p  priority=1 order=DESC end=69
SUBSCRIBE        track=1080p priority=0 order=DESC start=69

A reliable live or VOD viewer could issue:¶

SUBSCRIBE_UPDATE track=360p  priority=0 order=ASC end=69
SUBSCRIBE        track=1080p priority=1 order=ASC start=69

The difference between them is whether to prioritize the old track or the new track. In both scenarios, the subscription will seamlessly switch at group 69 even if it's seconds in the future. The same behavior can be used to switch down.¶

6.5.2. SVC

We touched on SVC before, but it's worth mentioning as an alternative to ABR. I want to see it used more often but I doubt it will be.¶

Instead of choosing the track based on the bitrate, the viewer subscribes to them all:¶

SUBSCRIBE track=360p  priority=0 order=DESC
SUBSCRIBE track=1080p priority=1 order=DESC
SUBSCRIBE track=4k    priority=2 order=DESC

During congestion, the 4k enhancement layer will be deprioritized followed by the 1080p enhancement layer. This is a more efficient use of bandwidth than ABR, but it requires more complex encoding.¶

6.6.1. Discovery

The first step to joining a conference is to discover the available broadcasts.¶

There is currently no discovery mechanism in MoqTransfork. However, an application can build one on top of a MoqTransfork track (of course!).¶

For example, suppose we have a conference room called room.12345. An index service could produce a room.12345 track that lists all broadcasts within the room. When Alice joins and ANNOUNCES room.12345.alice, the index service could update the room.12345 track to add a new FRAME +alice. The same can be done to remove her when she leaves.¶

6.6.2. Participants

Extending the idea that audio is more important than video, we can prioritize tracks regardless of the source. This works because SUBSCRIBE priority is scoped to the session and not the broadcast.¶

SUBSCRIBE track=alice.audio priority=1
SUBSCRIBE track=frank.audio priority=1
SUBSCRIBE track=alice.video priority=3
SUBSCRIBE track=frank.video priority=3

When Alice starts talking or is focused, we can actually issue a SUBSCRIBE_UPDATE to increase her priority:¶

SUBSCRIBE_UPDATE track=alice.audio priority=0
SUBSCRIBE_UPDATE track=alice.video priority=2

Note that audio is still more important than video, but Alice is now more important than Frank. (poor Frank)¶

This concept can further be extended to work with SVC or ABR:¶

SUBSCRIBE track=alice.360p priority=1
SUBSCRIBE track=frank.360p priority=2
SUBSCRIBE track=alice.720p priority=3
SUBSCRIBE track=frank.720p priority=4