Repository navigation
Improve handling of HLS/DASH/Progressive HTTP streaming data #858
Description
Followup issue of #810
Currently
Streamis dependent on service specific stuff (like YT's itag). This is a big problem because sooner or later this class will blow up. It's already confusing enough and this PR makes it worse by adding stuff for the delivery/transmission-method (currently the Stream class can only contain progressive HTTP streaming data).It's also currently hard to say which service uses which type of streaming and it's not possible to use different delivery methods for the same stream (not sure if this is possible/useful in the first place...).
Also when looking at the PR for NewPipe: We can't just create a specific playbackresolver/mediasource foreach stream type or service, that will be unmaintainable in the long run.
My ideas how to resolve these problems:
What should be fixed anyway
- isVideoOnly inside VideoStream should be removed. It's either a
VideoStreamor aAudioStreamor aVideoAudioStreamwhich contains the data of both.- Move the YT specific stuff out of the stream classes. Example resolution could be a
YouTubeVideoStreamwhich extendsVideoStream. However this may be solved (see below) in an even better way.- I'm not sure how SubtitleStream is supposed to fit into the below constructs, there may be changes required. As far as I have seen subtitles can be delivered e.g. via DASH or in other ways.
Idea 1
Move everything that is delivery specific from
Stream-like classes to a separateStreamDeliveryData-like classes:
Example proposal:
Idea 2
If - due to currently unknown reasons to me - it's not possible to do Idea 1 we could - instead of nesting (as done in Idea 1)
StreamDeliveryDataintoStream- simply create them side by side and derive the corresponding implementations from them.
Example proposal:
Overview how the delivery methods work:
Activity
litetex commentedon Jun 10, 2022
If anyone is interested in the current progress / what I have done so far, take a look at:
https://github.com/litetex/NewPipeExtractor/tree/delivery-methods-without-design-flaws
Warning: It's currently not compiling 😆
AudricV commentedon Sep 14, 2025
Finally, after so much time, here is my new stream API proposal, some elements are not entirely fixed (see the reasoning above with more details and thoughts):
Distinguishing between streaming protocols and delivery sources
Progressive media, DASH manifests or HLS ones are commonly served via HTTPs URLs, but could be also provided directly as strings (for example, if I remember properly, DASH manifests of Facebook videos are provided in service's HTML pages) or via other protocols (P2P for instance). That's why the
DeliverySourcenotion has been introduced.As multiple streaming protocols exist and some services use custom playback protocols (especially YouTube), it makes sense to add a list of standard protocols (DASH, HLS, PROGRESSIVE (i.e. a raw media file source)) and the ability to distinguish custom protocols (such as YouTube's UMP and SABR ones). Custom protocols should be represented in custom
Streamimplementations in my opinion, if services requires to do so.Muxed streams versus demuxed streams
Streams can be either muxed (i.e. composed of multiple streams) or demuxed. We only have the muxed notion for video streams with audio, this is limited (but applies to the majority of services the extractor currently supports), as we may also get subtitles.
I propose to have instead a generic notion of a muxed stream, which would support multiple video streams, multiple audio streams and multiple subtitles (or even just multiple streams regardless of their type, I'm uncertain on this). When one field would be
null, it means an extractor doesn't know whether there are tracks of this type in the muxed stream. Empty lists would mean there is no stream of this type in this muxed stream.This approach could be useful for DASH manifests and HLS master playlists (playlists containing multiple resolutions) for which we do not have enough metadata. However, I am not 100% sure if this is a good approach for these manifests, as they can contain muxed streams too (DASH and HLS support video merged with audio). A direct subclass of Stream named
ManifestStreams(or a better name of course) may be a better approach.YouTube specific stream types
YouTube streams can be grouped into 6 stream types (OTF for VOD videos is not included as it has been almost (if not completely) entirely removed, so we can drop its support):
For VOD videos only:
For VOD, live and post-live (i.e. when livestreams are ended but they haven't been reencoded as regular videos yet) videos:
For both stream types above, YouTube's biggest clients almost do not use them anymore and when they do, they use the UMP versions. Livestreams' captions are still requested separately by some clients right now. Stream URLs for adaptive streams are also being removed for some clients such as desktop website client, i.e. the InnerTube
WEBclient (ID 1), except for (post-)livestreams and videos withsignatureCiphers.With this API approach, legacy streams would be returned as
MuxedStreams and adaptive streams as UMP streams and in a SABR stream.Demuxed streams classes in my hiearchy diagram would be for VOD streams and (post-)live streams (their specific fields like
targetDurationSecormaxDvrDurationSec- this one is not returned by the extractor would be just null for VOD videos).Itagis a new version ofItagItem. It contains YouTube specific info only: xtags, itag ID, init and index ranges. Generic info such as bitrates, audio compression type (such as DRC), audio loudness, video width, video height, audio language, audio channels or stream expiration time, would only be present in the genericStreamclasses.More properties to generic stream types
A lot of information currently in
ItagItemshould be deduplicated and generalized intoStream, as it is not YouTube specific. Non service stream classes currently lack several properties which can be useful for some clients. Some of them are described below, but it is not an exhaustive list or a final version.Common
bitrate(oraverageBitrate): a long representing for demuxed streams the average bitrate of the stream. I don't know if it would make sense for muxed streams, but if so, then it should be the sum of all active streams (but would it make sense for muxed streams for which you can only get a part of available streams at a time, such as YouTube SABR streams, so this value can change? I don't think that's the case)Video streams (demuxed)
definitionRange: an enum representing the different range types (HDRorSDR)projectionType: an enum representing the different projection types (2D,MESH, ...) (I am not sure how to describe this part as I am not an expert in video projection)Audio streams (demuxed)
isDrc: likely a boolean, to describe whether the audio has dynamic range compression (unless there are multiple audio compression types, then it would be namedcompressionTypeand have a enum representing the different compression types and no compression)loudnessNormalization: the value of loudness normalization, as a double (in decibels? This is what YouTube uses)isAutomaticallyGenerated: whether the audio track has been automatically generated, so a booleanaudioChannels: an short representing audio channels countSubtitles (demuxed)
translations: a list ofSubtitlesStreamrepresenting translations from the currentSubtitlesStream.@litetex If you can, I really want to get your feedback on this.
theScrabi commentedon Sep 15, 2025
@Profpatsch @theScrabi: NewPlayer's design should align with this.
@AudricV you should geht familiar with NewPlayer as well.