This specification specifies an Hypothetical Render Model (HRM) that constrains the presentation complexity of
documents that conform to any of the TTML Profiles for Internet Media Subtitles and Captions ([IMSC]).
The model is not intended as a specification of the processing requirements for implementations. For instance, while the
model defines a glyph buffer for the purpose of limiting the number of glyphs displayed at any given point in time, it neither
requires the implementation of such a buffer, nor models the sub-pixel glyph positioning and anti-aliased glyph rendering
that can be used to produce text output.
Furthermore, the model is not intended to constrain readability complexity.
Status of This Document
This section describes the status of this
document at the time of its publication. A list of current W3C
publications and the latest revision of this technical report can be found
in the W3C technical reports index at
https://www.w3.org/TR/.
For this specification to exit the CR stage, at least 1 of each, and at least 2 of one, of the following
independent factors of verifications SHALL be demonstrated:
content producing implementation
validating implementation
The Working Group has identified the following at
risk features:
Publication as a Candidate Recommendation does not
imply endorsement by W3C and its Members. A Candidate Recommendation Snapshot has received
wide review, is intended to
gather
implementation experience,
and has commitments from Working Group members to
royalty-free licensing
for implementations.
This Candidate Recommendation is not expected to advance to Proposed
Recommendation any earlier than 01 March 2023.
This document was produced by a group
operating under the
W3C Patent
Policy.
W3C maintains a
public list of any patent disclosures
made in connection with the deliverables of
the group; that page also includes
instructions for disclosing a patent. An individual who has actual
knowledge of a patent which the individual believes contains
Essential Claim(s)
must disclose the information in accordance with
section 6 of the W3C Patent Policy.
error. A failure to conform to the constraints defined by this specification.
IMSC Document Instance. A Document Instance that conforms to any profile defined in any edition of
[IMSC].
4. Conformance
As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.
The key word SHALL in this document
is to be interpreted as described in
BCP 14
[RFC2119] [RFC8174]
when, and only when, they appear in all capitals, as shown here.
The objective of the HRM is to allow subtitle and caption authors
and providers to verify that the content they provide does not exceed
defined complexity levels, so that playback systems can render the content
synchronized with the author-specified display times.
Playback systems include desktop computers, mobile devices and
home theatre devices.
Note
The HRM is not a new concept: it has been included in all
versions and editions of [IMSC] and has remained substantially
unchanged. It is refactored herein to simplify document maintenance.
IMSC Document Instances are typically authored by a first party
and rendered by a second party. Unless both parties agree on the maximum
complexity of a IMSC Document Instance, it is likely that:
an IMSC Document Instance authored by a first party will
exceed the capabilities of the presentation processor of a second
party, resulting in an incomplete presentation where some subtitles or
captions might be missing or might be presented too late;
a first party authors only extremely simple IMSC Document
Instances in an attempt to ensure a complete presentation across all
renderers, resulting in a lower quality presentation;
a second party over-provisions their presentation processor
in order to ensure a complete presentation of all IMSC Document
Instances, resulting in increased run-time resource usage, code
complexity, etc.; or
one presentation processor becomes the de-facto reference to
determine whether the presentation of a IMSC Document Instance
will succeed, at the expense of other renderers and consistency of
presentation.
As illustrated in Figure 1, by defining a method (the
HRM) to compute a proxy for the complexity of an IMSC Document
Instance and specifying a complexity limit based on such proxy:
implementers can design their presentation processors to maximize the
likelihood that they will be able to render correctly all IMSC Document
Instances that do not exceed this limit; and
audiences can expect subtitle and caption presentation to match
the authorial intention.
5.3 Why is the HRM needed to limit complexity?
The HRM supplements the syntactic and structural constraints imposed in
[IMSC] by imposing constraints on the contents of the presentation.
Because of the temporal and spatial variability of subtitles and
captions across types of content, territories and languages, it is not
possible to limit the complexity of an IMSC Document Instance using
only average values.
An average-based constraint of 840 characters per
minute could be met in multiple ways, with different rendering
complexities. Contrast two potential approaches:
In the first, 5
characters are presented for a fraction of a second, followed by 835
characters that are then presented for over 59 seconds. This generates a
high rendering complexity for the 835 characters, since there is only a
brief time available to paint them.
In the second, 210 characters must be
painted every 15 seconds, giving 15 seconds to prepare for the next
presentation. This has a much lower rendering complexity.
The HRM achieves a more accurate representation of the complexity of an
IMSC Document Instance at any given time by taking into account
its past complexity in addition to its instantaneous complexity. The
same approach is commonly used in video to limit bitstream complexity,
e.g., the Hypothetical Reference Decoder (HRD) specified in
[iso14496-10].
5.4 How does the HRM measure and limit complexity?
The HRM defines a simple model for the rendering of subtitles and captions, and uses the time it takes to
render subtitles and captions according to that model as a proxy for the complexity of the subtitles and captions.
Rendering includes drawing region backgrounds, rendering and copying text, and decoding and copying images.
Complexity is then limited by requiring that the time to render one subtitle or caption is shorter than the time
elapsed since the previous subtitle or caption.
This simple model requires only a static analysis of the IMSC Document Instance, requires no fetching of
external resources and does not require the IMSC Document Instance to be actually rendered. Several
simplifying assumptions are made to achieve this. For example, the model assumes that each character is
drawn independently, and accounts for that assumption being, in many cases, false, by assigning different render
speeds for different scripts. In general the model is not intended to capture the actual time that an
implementation takes to render subtitles and captions, but rather scale with it: a document that is twice as
complex according to the model would require roughly twice as many resources to actually render.
5.5 Where is the HRM used?
The HRM is typically used prior to distribution of the IMSC Document Instance to the end-user, as an
integral part of authoring and as a quality check before distribution.
When the HRM is used, the consequences of an IMSC Document Instance exceeding the HRM limits depends
on the context:
an authoring system might, for example, flag the specific point in
time where the HRM limits were exceeded;
the ingest component of a streaming platform might outright reject
the IMSC Document Instance.
The HRM is not intended to be used when the IMSC Document
Instance is presented to end-users since:
end-users are not concerned with a technical complexity measure,
just as they are not concerned with video bit-rate, but instead with
whether the presentation is successful;
non-empty ISDs are processed using a simple double buffering model: while a non-empty ISD
En is being painted into a Back Buffer, the previous non-empty ISD Em is available
for display in a Front Buffer; and
empty ISDs merely disconnect the Front Buffer from the display when presented,
i.e., presenting an empty ISD is equivalent to presenting nothing.
The model specifies a (hypothetical) time required for completely painting a non-empty ISD
as a proxy for complexity. Painting includes clearing the Back Buffer, drawing region backgrounds,
rendering and copying glyphs, and decoding and copying images. Complexity is then limited by requiring that
painting of non-empty ISD En begins no earlier than the presentation time of the previous non-empty
non-empty ISD Em and completes by the presentation time of En.
In contrast, there is no complexity involved connecting and disconnecting the Front Buffer from the display, and
thus no complexity associated with empty ISDs.
Whenever applicable, constraints are specified relative to Root
Container Region dimensions, allowing subtitle sequences to be authored independently of
Related Video Object
resolution.
To enable scenarios where the same glyphs are used in multiple successive Intermediate Synchronic Documents, e.g. to convey a CEA-608/708-style roll-up (see
[CEA-608] and [CEA-708]), the Glyph Buffers Gn and Gn-1 store rendered glyphs across Intermediate Synchronic Documents, allowing glyphs to be copied into the Presentation
Buffer instead of rendered, a more costly operation.
Similarly, Decoded Image Buffers Dn and Dn-1 store decoded images across Intermediate Synchronic Documents, allowing images to be copied into the Presentation
Buffer instead of decoded.
paint, according to stacking order, all background pixels for each region;
paint all pixels for background colors associated with text or image subtitle content; and
paint the text or image subtitle content.
The Presentation Compositor SHALL start rendering En:
at the presentation time of Em, where m is the largest non-zero value that is both less than
n and is such that Em is not an empty ISD, if the presentation time of En minus
that of Em is less than IPD; or
at the presentation time of En minus IPD, otherwise.
Note
The Presentation Compositor never begins rendering an ISD more than IPD ahead of its
presentation time.
Figure 3
illustrates the rendering and presentation of Intermediate Synchronic Documents, where the hatched
areas indicate time spent drawing the associated Intermediate Synchronic Document. For example, the
Presentation Compositor begins rendering E1 at the presentation time of E0 since
E1 is not an empty ISD. In contrast, the Presentation Compositor begins rendering
E5 at the presentation time of E5 minus IPD since (i) both E3 and
E4 are empty ISDs and the presentation time of E5 minus that of E2 is
greater than IPD. Furthermore, E2 remains in the Front Buffer until the presentation time of
E5 but is not presented while E3 and E4 are presented, during which time the
Front Buffer is not available for display. Finally, the Presentation Compositor begins rendering E0
at the presentation time of E0 minus IPD since E0 is the first Intermediate
Synchronic Document.
This feature is expected to be removed before advancement to Proposed Recommendation if sufficient implementation experience cannot be demonstrated.
The contents of the Back Buffer SHALL be transferred instantaneously to the Front Buffer at the presentation
time of a non-empty ISD En, making the latter available for display.
The Front Buffer SHALL be:
disconnected from the display while an empty ISD is being presented; and
connected to the display, otherwise.
Note
It is possible for the contents of the Front Buffer to never be displayed. This can happen, for
example, if the Back Buffer is copied twice to Front Buffer between two consecutive video frame boundaries of the
Related Video Object.
It SHALL be an error for the Presentation Compositor to fail to complete painting pixels for
non-empty ISD En before its presentation time.
Unless specified otherwise, the following table SHALL specify values for IPD and BDraw.
BDraw effectively sets a limit on fillings regions - for example, assuming that the Root Container
Region is ultimately rendered at 1920×1080 resolution, a BDraw of 12 s-1 would correspond to a fill rate of
1920×1080×12/s=23.7×220pixels s-1.
NBG(Ri) counts the number of tts:backgroundColor attributes specified span elements.
In a common scenario illustrated below, this results in the complexity of painting (relatively small) span backgrounds to be equal to painting the background of (relatively much larger) region that essentially fills the root container.
This can be addressed by excluding span from the NBG(Ri) computation, and instead including tts:backgroundColor in the list of glyph properties at https://www.w3.org/TR/ttml-imsc1.1/#paint-text.
Note
An element and its parent that satisfy the criteria above and share identical computed values of
tts:backgroundColor are counted as two distinct elements for the purpose of computing NBG(Ri).
Note
The set element is not included in the computation of NBG(Ri). While it can affect the
computed values of tts:backgroundColor, it is removed during Intermediate Synchronic Document
construction.
For each presented image, the Presentation Compositor SHALL either:
if an identical image is present in Decoded Image Buffer Dn, copy the image from Decoded Image Buffer
Dn to the Back Buffer using the Image Copier; or
if an identical image is present in Decoded Image Buffer Dn-1, i.e. an identical image was present in
Intermediate Synchronic Document En-1, copy using the Image Copier the image from Decoded Image Buffer
Dn-1 to both the Decoded Image Buffer Dn and the Back Buffer; or
otherwise, decode the image, using the Image Decoder, into the Back Buffer and Decoded
Image Buffer Dn.
Two images SHALL be identical if and only if they reference the same encoded image source.
NSIZ(Ii) SHALL be the number of pixels of presented image Ii.
The contents of the Decoded Image Buffer DnSHALL be transferred instantaneously to Decoded Image Buffer
Dn-1 at the presentation time of Intermediate Synchronic Document En.
The total size occupied by images stored in Decoded Image Buffers Dn or Dn-1SHALL be the sum of
their Normalized Image Area.
The size of Decoded Image Buffers Dn or Dn-1SHALL be the Normalized Decoded Image Buffer Size
(NDIBS).
Unless specified otherwise, the following table SHALL specify ICpy, IDec, and NDBIS.
Parameter
Initial value
Normalized image copy performance factor (ICpy)
6
Image Decoding rate (IDec)
1 × 220 pixels s-1
Normalized Decoded Image Buffer Size (NDIBS)
0.9885
10. Paint Text
In the context of this section, a glyph is a tuple consisting of (i) one character and (ii) the computed values
of the following style properties:
tts:color
tts:fontFamily
tts:fontSize
tts:fontStyle
tts:fontWeight
tts:textDecoration
tts:textOutline
tts:textShadow
Note
In the case where a property is prohibited in a profile of [IMSC], the computed value of the property
specified in [ttml2] can be used.
Note
The Hypothetical Render Model defines a one-to-one mapping between characters and glyphs
(using the definition of glyph from this document).
While a one-to-one mapping between code points and glyphs (using the definition of glyph from [i18n-glossary])
is common in some scripts (such as the Latin script),
the actual relationship is more complex.
Some scripts, such as Arabic, use different glyphs for a given character,
depending on its position in a word.
Some scripts require combining marks or use a sequence of code points to form a glyph.
Cases exist where a given sequence of code points can have different glyph representations depending on context.
This complexity is accounted for by reducing the performance of the glyph buffer
for scripts where a one-to-one mapping is not the general rule (see GCpy below).
if an identical glyph is present in Glyph Buffer Gn, copy the glyph from Glyph Buffer
Gn to the Back Buffer using the Glyph Copier; or
if an identical glyph is present in Glyph Buffer Gn-1, i.e. an identical glyph was present in
Intermediate Synchronic Document En-1, copy using the Glyph Copier the glyph from Glyph Buffer
Gn-1 to both the Glyph Buffer Gn and the Back Buffer; or
otherwise render using the Glyph Renderer the glyph into the Back Buffer and Glyph Buffer
Gn.
Figure 4
Example of Presentation Compositor Behavior for Text Rendering
NRGA(Gi) does not take into account decorations (e.g. underline), effects (e.g. outline) or actual
typographical glyph aspect ratio. An implementation can determine an actual buffer size needs based on worst-case glyph size
complexity.
The contents of the Glyph Buffer GnSHALL be copied instantaneously to Glyph Buffer Gn-1 at the
presentation time of Intermediate Synchronic Document En.
It SHALL be an error for the sum of NRGA(gi) over all glyphs Glyph Buffer Gn
to be larger than the Normalized Glyph Buffer Size (NGBS).
Unless specified otherwise, the following table specifies values of GCpy, Ren and NGBS.
Normalized glyph copy performance factor (GCpy)
Script property, as defined at [UAX24], for the character of
gi
GCpy
Latin, Greek, Cyrillic, Hebrew or Common
12
any other value
3
Text rendering performance factor Ren(Gi)
Script property, as defined at [UAX24], for the character of
gi
Ren(Gi)
Han, Katakana, Hiragana, Bopomofo or Hangul
0.6
any other value
1.2
Normalized Glyph Buffer Size (NGBS)
1
Note
While DURT(En) is not affected, the choice of font by the presentation processor can increase actual rendering
complexity at time of presentation. For instance, a cursive font might select different glyphs for a given grapheme (in order to maintain joining or for the start/end of the
word) even in the Latin script. Conversely the rendering of scripts that fall in the any other
value category can in practice achieve performance comparable to, say, the Latin script.
A. Accessibility Considerations
This section is non-normative.
A.1 Impact of non-conformance
In a system where IMSC Document Instances are expected to conform to the Hypothetical Render Model, an IMSC
Document Instance that does not conform to the Hypothetical Render Model might negatively impact accessibility
during presentation of the IMSC Document Instance and its associated content.
A.2 User customisation of presentation
This specification does not attempt to model any additional complexity for presentation processors
that might arise due to the user customisation of presentation,
for example as described by [media-accessibility-reqs];
such user customisation is not defined by [IMSC].
Implementers of presentation processors
that support user customisation of presentation should ensure that those processors are able
to present IMSC Document Instances that conform to the Hypothetical Render Model,
even if the customisation effectively increases the complexity of presentation.
B. Privacy and Security Considerations
This section is non-normative.
B.1 General
This specification has no inherent security or privacy implications.
The algorithm defined within this specification is used for static analysis of a resource.
This specification does not define any protocol or interface for obtaining such a resource,
and it does not define any interface for exposing the results of the analysis.
No personal or sensitive information is processed as part of the algorithm,
other than any such information that might happen to be part of the IMSC Document Instance being analysed.
No information is exposed by the algorithm to any origin.
No scripts are loaded or processed as part of the algorithm
and no links to external resources are dereferenced.
B.2 Implementation considerations
Implementers of this specification should capture and meet privacy and security requirements for their intended application.
For example, an implementation could, when reporting on an error encountered during processing of an IMSC Document Instance,
include a section of the content of an IMSC Document Instance to elaborate the error.
If that content could include sensitive or personal information,
the implementation should ensure that any such output is provided using appropriately secure protocols.
No such reporting is defined or required by this specification.
C. Error Reporting and Exception Handling
This section is non-normative.
C.1 Error Reporting
This specification does not define how, or even if, errors should be reported.
For example, an implementation could stop on the first error encountered,
or continue to process the IMSC Document Instance
and report every error.
Or an implementation could exit with an appropriate status code
without reporting any details at all.
C.2 Exception Handling
This specification does not define any runtime exceptions,
or how such exceptions should be handled.
In order to allow short (less than 100 ms) gaps between subtitles, which is common practice, the complexity
of presenting empty ISDs has been reduced to zero: instead of being drawn into the Back Buffer, an
empty ISD merely disconnects the Front Buffer from the display while it is presented.
