|
|
BIFS Tutorial - Part VTime-dependent Nodes |
|
|
|
BIFS Tutorial - Part VTime-dependent Nodes |
|
In part III, we have used the Movietexture node to include a video in the scene, but we have left out a major functionality of this node: starting and stoping the playback of the movie at desired times. We have also used the TimeSensor node to generate events, but haven't tried to stop or start it interactively. A node whose behaviour depend on the current time ("scene time") is said to be a time-dependent node. These are TimeSensor, MovieTexture, AudioBuffer, AudioSource, AudioClip and AnimationStream. To understand these nodes we have to explain a bit more how timing works in MPEG-4, and what are the differences between the MPEG-4 and the VRML timing models.
VRML follows a 'download-and-play' model. The VRML content is first downloaded to the terminal before it is displayed to the user. Although MPEG-4 may look similar when playing local content or downloaded content (HTTP/FTP...), it is not designed along this model, but originally built for broadcast environments, which means a scene can be displayed to the user even though all information is not already present at the terminal. Actually, a terminal may not even know what "all information" means. The MPEG-4 model has then been a bit modified to allow video-on-demand kind of application, hence follows what is called a 'streaming' model.
We have seen the notion of ElementaryStream, an entity that carries data for a media object. We have also seen the notions of Access Unit (AU) and presentation time or CompositionTimeStamp (CTS), which allows to "author in time" the scene. An elementary stream is nothing but a succession of AUs, each AU having its own and UNIQUE CTS (that is, 2 consecutive AUs of an ES may not have the same CTS). The CTS is then used to present the decoded data to the user in time. The timing of ES data can then be known during all its lifetime. The second step is to synchronize streams composing a single object (in the case of scalable content), to ensure inter-stream synchronization, otherwise a decoder could get out-of-order, hence unusable, data. This is done by defining time dependencies between ESs to ensure that timestamps of different streams are interpreted in a single time base. This single time base is called the Object Time Base or OTB, and the stream picked up as the time reference is called "clock". Usually one object has its own OTB, but objects may share OTBs to achieve inter-object synchronization (this is usually the case of a video object and its related audio object). In order to estimate network jitters, the clock may also carry specific timing informations called Object Clock References (OCRs), very much like RTCP timestamps in the RTP protocol.
Objects may share a single clock, but don't forget this clock is used to synchronize the object data for presentation (the famous CTS). Imagine you design a scene with a video, and setup the video to loop. If the video object clock is actually the BIFS ES, what happens at the end of the video (time T)? The video is restarted from time 0. But the BIFS stream is still playing, the clock is still running and indicates time T, thus the video data will arrive too late for presentation and will be discarded. Conclusion: you will not see the video loop, but very likely (depending on implementations) only see the last frame of the video at best. This is EXTREMELY important in the case of MP4 files, where by default all streams (or tracks) shall be presented synchronized. If you do not explicetly assign clock references, you will sure get some weird content! So never forget to check your clock dependencies when authoring content with multiple time base !!
In MPEG-4 as in VRML, the scene has a timing model needed to use the TimeSensor node. This is sometimes refered to as "simulation time", but "scene time" is more accurate. In VRML, the scene time origin t=0 is January, 1st 1970 0h00 (content uses absolute time), whereas in MPEG-4 the scene time origin t=0 is the CTS of the first BIFS Access Unit (content uses relative time).
We will study the behavior of time-dependent nodes with the MovieTexture node, but all other time-dependent nodes behave the same. The MovieTexture node is:
|
field |
field type |
event type |
default value |
description |
| loop | SFBool | exposedField | FALSE | Indicates that the video object shall restart once playback is over |
| speed | SFFloat | exposedField | 1.0 | Indicates playback speed |
| startTime | SFTime | exposedField | 0 | Indicates the scene time at which playback shall start |
| stopTime | SFTime | exposedField | 0 | Indicates the scene time at which playback shall stop |
| url | MFString | exposedField | [] | Object Identifier of the video |
| repeatS | SFBool | field | TRUE | indicates that texture shall be repeated horizontally to fill the shape |
| repeatT | SFBool | field | TRUE | indicates that texture shall be repeated vertically to fill the shape |
| duration_changed | SFTime | eventOut | Duration of the video object | |
| isActive | SFBool | eventOut | Indicates whether playback is active or not |
As we have seen, the scene time in MPEG-4 is relative to the first BIFS access unit recieved. Hence the startTime and stopTime fields are coded as relative offsets to the scene time. If the node is inserted in an AU at time t=3s with a startTime of 2 seconds encoded in the bitstream, the actual value of startTime will be offseted of the AU CTS (current scene time), hence 5 seconds.
The behavior of time-dependent nodes is the same in MPEG-4 as in VRML
For time-dependent nodes the notions of activation and deactivation are defined. A time-dependent node always has the 3 following fields: exposedFields startTime, stopTime, and eventOut isActive. It may also have an exposedField loop used to restart playback (but AudioSource node doesn't). These values are used to compute the state of the node. Unlike any other nodes, an active time-dependent node may ignore modification on some of its fields (that is, the node will not take into account the modification and won't generate related events until next activation).
When a time-dependent node has the loop field, the notion of cycle is defined. For TimeSensor, the cycle is given by the cycleInterval field. For MovieTexture and AudioClip, the cycle is defined as the duration of the object - if the object duration is unknown, the node has no associated cycle (infinite cycle duration). For AudioBuffer, the cycle duration is given by the length field.
When active at the end of a cycle and the loop field is false, the node is deactivated (playback doesn't restart).
When active at the end of a cycle and the loop field is true, the node keeps being active (playback restarts).
If startTime >= stopTime, the node stays activated according to the loop field. Otherwise (startTime < stopTime) the node will be deactivated (playback stops) when the scene time reaches stopTime.
A time-dependent node only triggers an isActive event when changing state: when becoming active, isActive=TRUE is generated, and when becoming inactive, isActive=FALSE is generated.
A time-dependent node is inactive until the scene time reaches startTime. When scene time >= startTime, the node becomes active (playback starts) and triggers an isActive=TRUE eventOut.
An active time-dependent node becomes inactive when scene time becomes greater than stopTime IF stopTime > startTime. If stopTime <= startTime, the stopTime field is ignored (not used for deactivation detection).
An active time-dependent node becomes inactive when the cycle end is reached and the loop field is FALSE. For AudioSource where no loop field is present, the node becomes inactive when the media is over.
If an active time-dependent node recieves a loop=FALSE event, the node remains active until the current cycle is over or scene time becomes greater than stopTime if stopTime>startTime.
Any modification of the startTime field of an active time-dependent node is ignored. If the stopTime field of an active time-dependent node is modified and the new value is less than or equal to startTime, the modification is ignored (the field value is modified but the node react as if the value was not modified).
If the stopTime field of an active time-dependent node is modified at t=T0 and the new value is so that startTime < stopTime <= T0, then the node becomes inactive.
Note: with time-dependent nodes triggering playback of a media object, the media object will start playing from its current clock time. If the clock is not running, it will thus restart from the begining. Consequently, these nodes cannot be used to seek a media object.
Let's take the video example of part III (video1.xmt), set the loop field to true and start the movie at 2 seconds and stop it at 30 seconds. The MovieTexture node is then
<texture><MovieTexture url="'10'" loop="true" startTime="2" stopTime="30"/></texture>
The final scene is video2.xmt, video2.bt, video2.mp4
Note what happens at the end of the movie. Some player will restart the whole presentation, and some others will just restart the movie and display the last frame(s). Also note that the first frame of the movie should be displayed while the MovieTexture node is inactive.
Now let's take the above example and set the clock reference of the video stream to be itself. This is done by setting the OCR_ES_ID field in the ESDescriptor of the video stream:
<ES_Descriptor ES_ID="VideoStream" binaryID="3" OCR_ES_ID="VideoStream">
</ES_Descriptor>
The final scene is video3.xmt, video3.bt, video3.mp4
Note that now the movie is correctly looping and stops when stopTime is reached.
Exercise 15 : Rewrite the above examples with audio instead of video using the AudioClip node. Note that considerations on clock dependencies still apply.
Exercise 16 : Using touch sensors and conditionals, write a simple scene with a play button and a stop button controling a movie.
We have seen in this part the different notions of timing in the MPEG-4 standard, how to control time-dependent nodes and how important object clock dependencies are. You have now all the tools needed to write complex interactive animations and effects to enhance the user experience.
|
[ Home ] [ Interactivity] [ Time-dependent Nodes ] [ Controling Media Objects ] |
