Donggyu Lee* ย ยทย Youngbin Ki* ย ยทย Jeonghun Kang ย ยทย Taehwan Kim
Ulsan National Institute of Science and Technology (UNIST)
๐ KDD 2026 ย ยทย Datasets & Benchmarks Track ย ยทย *equal contribution
SVHighlights videos average 2 hours โ roughly 30โ60ร longer than existing highlight detection benchmarks.
While highlight detection for long-form videos is of great practical importance, most existing methods remain limited to short-form content, largely due to the absence of a suitable benchmark. To bridge this gap, we introduce SVHighlights, to the best of our knowledge, the first benchmark for highlight detection in extremely long sports videos, each exceeding one hour in duration, across multiple sports categories. SVHighlights is constructed from pairs of full-length sports videos and their corresponding official highlight videos using a dataset generation pipeline, enabling scalable and cost-effective label generation without conventional per-clip saliency annotation. The benchmark comprises 320 videos spanning a wide range of sports, with an average duration of 2.00 hours and a total of 640.18 hours, substantially exceeding previous highlight detection datasets. Beyond the lack of benchmarks, existing methods also face fundamental challenges on long videos: models trained on short clips of only a few minutes fail to generalize to hour-long content, and their clip-level scoring lacks the broader context needed to identify highlights in long-form videos. To address these challenges and provide a strong baseline for SVHighlights, we present TF-SELECTOR, a training-free segment-based approach that divides each video into context-aware segments by merging adjacent shots sharing the same semantic content, and predicts segment-level saliency scores using a large language model (LLM) with multimodal inputs including visual captions, transcripts, and audio volume. Extensive experiments demonstrate that TF-SELECTOR achieves superior performance across most evaluation metrics compared to Video Temporal Grounding (VTG)-tuned baselines, with improvements of +3.12 in HIT@1, +4.06 in HIT@K, and +2.95 in IoU. These results establish SVHighlights as a challenging testbed for long-form highlight detection and demonstrate that a simple segment-based strategy can effectively scale to hour-long videos.
- [2026.06.05] ๐ The paper is now available on arXiv.
- [2026.05.25] ๐ค Released the TF-SELECTOR model code.
- [2026.05.23] ๐ Released the SVHighlights dataset (annotations & features) on Hugging Face and the preprocessing code.
- [2026.05.17] ๐ SVHighlights was accepted to KDD 2026 (Datasets & Benchmarks Track).
SVHighlights pairs 320 full-length sports broadcasts with their official highlight videos, spanning 8 sports โ american football, baseball, basketball, ice hockey, racing, rugby, soccer, and volleyball (40 videos each).
Rather than relying on manual per-clip annotation, we align each official highlight video to its full-length broadcast. Every 1-second highlight clip is matched to the most similar full-video frame via a pixel-level PSNR score, a post-processing step enforces temporal consistency, and a filtering step removes mismatched frames.
Each video places the official highlight video (left) next to the full-video frames aligned by our pipeline (right) โ for every highlight second, a 1-second clip cut from the full broadcast at the frame our highlight alignment matched.
๐ American Footballamerican_football.mp4 |
โพ Baseballbaseball.mp4 |
๐ Basketballbasketball.mp4 |
๐ Ice Hockeyice_hockey.mp4 |
๐ Racingrace.mp4 |
๐ Rugbyrugby.mp4 |
โฝ Soccersoccer.mp4 |
๐ Volleyballvolleyball.mp4 |
TF-SELECTOR (Training-Free Segment-based Extremely Long video highlight detECTOR) is a training-free baseline with three stages:
- Context-aware segmentation โ shots are detected with a shot-boundary detector and adjacent shots that share the same spoken sentence (from an ASR model) are merged into coherent segments.
- Segment captioning โ a visionโlanguage model (VLM) generates a textual description for each segment.
- Segment-level scoring โ a large language model (LLM) predicts a saliency score per segment from the caption, transcript, and audio volume; the score is then assigned to every clip in the segment.
The code is split into two parts:
benchmark/โ SVHighlights dataset construction (paper Section 3): video trimming, highlight alignment, filtering, and label generation, plus the alignment-quality evaluation tools.tf_selector/โ TF-SELECTOR baseline (paper Section 4): preprocessing (shot-boundary detection, speech recognition, context-aware segmentation, audio-volume extraction) and inference (segment captioning with a VLM, LLM-based saliency scoring, and prediction parsing into the per-clip saliency-score format consumed byeval.py).
See each directory's README.md for the full pipeline and usage instructions.
Highlight-detection predictions are scored with eval.py โ see
Evaluation.
We do not redistribute the original videos. In line with the dataset's
terms of use, we release only the video URLs (video_list.csv), the extracted
features, and the annotation labels, as a Hugging Face dataset:
Both the directory-style annotations and the features are released as tarballs
so they download as a few large Xet objects rather than thousands of small
files โ this avoids Hugging Face's anonymous API rate limit on snapshot
downloads. The four annotations/*.tar cover all 8 sports together; features
are split per sport (features/<sport>.tar).
# 1) Download the dataset (use --max-workers 1 to stay under the HF rate limit)
huggingface-cli download idong1004/SVHighlights \
--repo-type dataset --local-dir ./data --max-workers 1
# 2) Extract the annotation tarballs in place
for a in alignment segments whisper shots; do
tar -xf data/annotations/${a}.tar -C data/annotations/ && rm data/annotations/${a}.tar
done
# 3) Extract the per-sport feature tarballs in place
for s in american_football baseball basketball ice_hockey race rugby soccer volleyball; do
tar -xf data/features/${s}.tar -C data/features/ && rm data/features/${s}.tar
doneData layout (after extraction)
data/
โโโ metadata/
โ โโโ video_list.csv # video URLs + per-video trim boundaries
โโโ annotations/
โ โโโ alignment/ # benchmark/align.py (from alignment.tar)
โ โโโ all_filtered_frame_idx.json # benchmark/filter_frames.py + manual filtering
โ โโโ label.json # benchmark/labeling.py
โ โโโ shots/ # tf_selector/shot_boundary.py (from shots.tar)
โ โโโ whisper/ # tf_selector/transcribe.py (from whisper.tar)
โ โโโ segments/ # tf_selector/segment.py (from segments.tar)
โ โโโ segment_caption.json # tf_selector/segment_captioning.py (VLM)
โ โโโ volume.json # tf_selector/volume.py
โ โโโ minmax_volume.json # tf_selector/volume_minmax.py
โโโ features/
โโโ <sport>/ # extracted from features/<sport>.tar
โโโ vid_clip/ # HERO video CLIP features
โโโ vid_slowfast/ # HERO video SlowFast features
โโโ txt_clip/ # HERO query CLIP features
โโโ aud_pann/ # PANN audio features
Videos are named <sport>_<idx>.mp4, where <sport> is one of:
american_football, baseball, basketball, ice_hockey, race, rugby,
soccer, volleyball.
Annotations
| Artifact | Produced by | Description |
|---|---|---|
annotations/alignment/ |
benchmark/align.py |
Per-video raw PSNR alignment between highlight clips and full-video frames. |
annotations/all_filtered_frame_idx.json |
benchmark/filter_frames.py + manual filtering |
Final aligned frame index per highlight clip. |
annotations/label.json |
benchmark/labeling.py |
Per-clip {0, 1} highlight labels (one score per 2-second clip) โ the evaluation ground truth. |
annotations/shots/ |
tf_selector/shot_boundary.py |
Per-video shot-boundary detection (start/end frame index per shot). |
annotations/whisper/ |
tf_selector/transcribe.py |
Word-level WhisperX transcripts. |
annotations/segments/ |
tf_selector/segment.py |
Context-aware segments (2-minute maximum length). |
annotations/segment_caption.json |
tf_selector/segment_captioning.py |
Per-segment captions from the InternVL2_5-8B VLM (input to the LLM scorer). |
annotations/volume.json |
tf_selector/volume.py |
Per-clip audio loudness (dBFS) for every video. |
annotations/minmax_volume.json |
tf_selector/volume_minmax.py |
Min-max normalized per-clip audio volume in [0, 1]. |
Features (QVHighlights-style)
| Feature | Directory | Extractor |
|---|---|---|
| Video โ CLIP | features/<sport>/vid_clip/ |
HERO Video Feature Extractor |
| Video โ SlowFast | features/<sport>/vid_slowfast/ |
HERO Video Feature Extractor |
| Query โ CLIP | features/<sport>/txt_clip/ |
HERO Video Feature Extractor |
| Audio โ PANN | features/<sport>/aud_pann/ |
audioset_tagging_cnn |
The query feature is the CLIP text embedding of the fixed query "Highlight of this {sport} video". Video and query features were extracted with the HERO Video Feature Extractor; audio features with PANN / audioset_tagging_cnn.
benchmark/ and tf_selector/ run in a single conda environment (named
svhighlights here, Python 3.9).
conda create -n svhighlights python=3.9 -y
conda activate svhighlights
# ffmpeg / ffprobe โ used by almost every script
conda install -c conda-forge ffmpeg -y
# Python dependencies (covers benchmark/ and tf_selector/ โ preprocessing + model inference)
pip install -r requirements.txt
# PyTorch with CUDA 12.1 โ required by tf_selector/ model inference
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121
# External packages installed from source:
pip install git+https://github.com/openai/CLIP.git # benchmark/eval_clip_similarity.py
git clone https://github.com/soCzech/TransNetV2 # tf_selector/shot_boundary.pyFor GPU use, install the PyTorch build that matches your CUDA version (the pipeline was run with PyTorch 2.3.1 / CUDA 12.1). The two external packages above are each needed by only one script โ skip either you do not plan to run.
Tested with Python 3.9, PyTorch 2.3.1 (CUDA 12.1), and ffmpeg 4.2.9.
Note โ
tf_selector/main.pyandtf_selector/segment_captioning.pylog in to the Hugging Face Hub at startup vialogin(token=os.environ.get('HF_TOKEN')); export your access token before running, e.g.export HF_TOKEN=hf_.... Intf_selector/parse.py, replace thepath/to/frame/{vid}placeholder with the directory that contains your per-video frames.
The code is split into two pipelines โ see each directory for the full step-by-step commands and CLI arguments.
- Dataset construction โ
benchmark/: video trimming, highlight alignment, PSNR filtering, and per-clip label generation (paper Section 3). - TF-SELECTOR baseline โ
tf_selector/: preprocessing (shot boundaries, ASR, context-aware segmentation, audio volume, VLM segment captioning) and training-free inference (LLM saliency scoring + prediction parsing) (paper Section 4). All preprocessing outputs are released underannotations/, so these steps are optional.
Every script accepts --sports to restrict processing to a subset of sports
and -h for the full argument list.
To evaluate a highlight-detection model on SVHighlights, produce a prediction
file with a per-clip saliency score for every video, then run eval.py.
Prediction file โ a JSON list with one entry per video:
[
{
"vid": "american_football_1",
"pred_saliency_scores": [0.895, 0.880, 0.863, ...]
},
...
]vidโ video id (<sport>_<idx>)pred_saliency_scoresโ one predicted score per 2-second clip, in clip order (each video is divided into non-overlapping 2-second clips)
Ground-truth file โ annotations/label.json from the dataset: a JSON list
of {"vid", "saliency_scores"}, where saliency_scores are the per-clip
{0, 1} highlight labels.
Run:
python eval.py \
--submission_path path/to/predictions.json \
--gt_path annotations/label.json \
--save_path path/to/results.jsoneval.py reports four metrics, computed per sport and over all:
| Metric | Description |
|---|---|
HL-mAP |
Clip-level mean Average Precision |
HL-Hit1 |
Whether the top-scored clip is a ground-truth highlight |
HL-Hitk |
Hit rate among the top-K clips (K = number of GT highlight clips) |
HL-IoU |
Temporal overlap between the top-K predictions and GT highlights |
We do not redistribute the original sports broadcasts. For research that needs the raw visual content, we share per-video frames sampled at 1 frame per second (1 fps) on request, subject to a research-only, no-redistribution agreement.
Approved requests receive a Google Drive link with a 7-day access window. For
any issue with the form, please email leedongkyu2019@unist.ac.kr or open an
issue on this repository.
If you find SVHighlights useful in your research, please cite:
@inproceedings{Lee_2026,
title={SVHighlights: Towards Extremely Long Sport Video Highlight Detection},
url={http://dx.doi.org/10.1145/3770855.3817564},
DOI={10.1145/3770855.3817564},
booktitle={Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2},
publisher={ACM},
author={Lee, Donggyu and Ki, Youngbin and Kang, Jeonghun and Kim, Taehwan},
year={2026},
month=Aug,
pages={9254โ9265}
}


