Skip to content

About

[KDD 2026 Datasets & Benchmarks Track] SVHighlights: a benchmark for highlight detection in extremely long sports videos

Resources

Stars

11 stars

Watchers

0 watching

Forks

Latest commit

ย 

History

36 Commits

Folders and files

NameName
Last commit message
Last commit date
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

๐ŸŽฌ SVHighlights: Towards Extremely Long Sport Video Highlight Detection

arXiv Project Page ๐Ÿค— Dataset License: Apache 2.0 Conference Python

Donggyu Lee* ย ยทย  Youngbin Ki* ย ยทย  Jeonghun Kang ย ยทย  Taehwan Kim

Ulsan National Institute of Science and Technology (UNIST)

๐Ÿ“ KDD 2026 ย ยทย  Datasets & Benchmarks Track ย ยทย  *equal contribution

SVHighlights videos average 2 hours โ€” roughly 30โ€“60ร— longer than existing highlight detection benchmarks.


๐Ÿ“ Abstract

While highlight detection for long-form videos is of great practical importance, most existing methods remain limited to short-form content, largely due to the absence of a suitable benchmark. To bridge this gap, we introduce SVHighlights, to the best of our knowledge, the first benchmark for highlight detection in extremely long sports videos, each exceeding one hour in duration, across multiple sports categories. SVHighlights is constructed from pairs of full-length sports videos and their corresponding official highlight videos using a dataset generation pipeline, enabling scalable and cost-effective label generation without conventional per-clip saliency annotation. The benchmark comprises 320 videos spanning a wide range of sports, with an average duration of 2.00 hours and a total of 640.18 hours, substantially exceeding previous highlight detection datasets. Beyond the lack of benchmarks, existing methods also face fundamental challenges on long videos: models trained on short clips of only a few minutes fail to generalize to hour-long content, and their clip-level scoring lacks the broader context needed to identify highlights in long-form videos. To address these challenges and provide a strong baseline for SVHighlights, we present TF-SELECTOR, a training-free segment-based approach that divides each video into context-aware segments by merging adjacent shots sharing the same semantic content, and predicts segment-level saliency scores using a large language model (LLM) with multimodal inputs including visual captions, transcripts, and audio volume. Extensive experiments demonstrate that TF-SELECTOR achieves superior performance across most evaluation metrics compared to Video Temporal Grounding (VTG)-tuned baselines, with improvements of +3.12 in HIT@1, +4.06 in HIT@K, and +2.95 in IoU. These results establish SVHighlights as a challenging testbed for long-form highlight detection and demonstrate that a simple segment-based strategy can effectively scale to hour-long videos.

๐Ÿ”ฅ Update

  • [2026.06.05] ๐Ÿ“„ The paper is now available on arXiv.
  • [2026.05.25] ๐Ÿค– Released the TF-SELECTOR model code.
  • [2026.05.23] ๐Ÿš€ Released the SVHighlights dataset (annotations & features) on Hugging Face and the preprocessing code.
  • [2026.05.17] ๐ŸŽ‰ SVHighlights was accepted to KDD 2026 (Datasets & Benchmarks Track).

๐Ÿ“บ SVHighlights

SVHighlights pairs 320 full-length sports broadcasts with their official highlight videos, spanning 8 sports โ€” american football, baseball, basketball, ice hockey, racing, rugby, soccer, and volleyball (40 videos each).

Highlight alignment

Rather than relying on manual per-clip annotation, we align each official highlight video to its full-length broadcast. Every 1-second highlight clip is matched to the most similar full-video frame via a pixel-level PSNR score, a post-processing step enforces temporal consistency, and a filtering step removes mismatched frames.

Qualitative comparison

Each video places the official highlight video (left) next to the full-video frames aligned by our pipeline (right) โ€” for every highlight second, a 1-second clip cut from the full broadcast at the frame our highlight alignment matched.

๐Ÿˆ American Football
american_football.mp4
โšพ Baseball
baseball.mp4
๐Ÿ€ Basketball
basketball.mp4
๐Ÿ’ Ice Hockey
ice_hockey.mp4
๐Ÿ Racing
race.mp4
๐Ÿ‰ Rugby
rugby.mp4
โšฝ Soccer
soccer.mp4
๐Ÿ Volleyball
volleyball.mp4

๐Ÿค– TF-SELECTOR

TF-SELECTOR (Training-Free Segment-based Extremely Long video highlight detECTOR) is a training-free baseline with three stages:

  1. Context-aware segmentation โ€” shots are detected with a shot-boundary detector and adjacent shots that share the same spoken sentence (from an ASR model) are merged into coherent segments.
  2. Segment captioning โ€” a visionโ€“language model (VLM) generates a textual description for each segment.
  3. Segment-level scoring โ€” a large language model (LLM) predicts a saliency score per segment from the caption, transcript, and audio volume; the score is then assigned to every clip in the segment.

๐Ÿ—‚๏ธ Repository Structure

The code is split into two parts:

  • benchmark/ โ€” SVHighlights dataset construction (paper Section 3): video trimming, highlight alignment, filtering, and label generation, plus the alignment-quality evaluation tools.
  • tf_selector/ โ€” TF-SELECTOR baseline (paper Section 4): preprocessing (shot-boundary detection, speech recognition, context-aware segmentation, audio-volume extraction) and inference (segment captioning with a VLM, LLM-based saliency scoring, and prediction parsing into the per-clip saliency-score format consumed by eval.py).

See each directory's README.md for the full pipeline and usage instructions. Highlight-detection predictions are scored with eval.py โ€” see Evaluation.

โฌ‡๏ธ Download

We do not redistribute the original videos. In line with the dataset's terms of use, we release only the video URLs (video_list.csv), the extracted features, and the annotation labels, as a Hugging Face dataset:

๐Ÿค— https://huggingface.co/datasets/idong1004/SVHighlights

Both the directory-style annotations and the features are released as tarballs so they download as a few large Xet objects rather than thousands of small files โ€” this avoids Hugging Face's anonymous API rate limit on snapshot downloads. The four annotations/*.tar cover all 8 sports together; features are split per sport (features/<sport>.tar).

# 1) Download the dataset (use --max-workers 1 to stay under the HF rate limit)
huggingface-cli download idong1004/SVHighlights \
  --repo-type dataset --local-dir ./data --max-workers 1

# 2) Extract the annotation tarballs in place
for a in alignment segments whisper shots; do
  tar -xf data/annotations/${a}.tar -C data/annotations/ && rm data/annotations/${a}.tar
done

# 3) Extract the per-sport feature tarballs in place
for s in american_football baseball basketball ice_hockey race rugby soccer volleyball; do
  tar -xf data/features/${s}.tar -C data/features/ && rm data/features/${s}.tar
done

Data layout (after extraction)

data/
โ”œโ”€โ”€ metadata/
โ”‚   โ””โ”€โ”€ video_list.csv       # video URLs + per-video trim boundaries
โ”œโ”€โ”€ annotations/
โ”‚   โ”œโ”€โ”€ alignment/            # benchmark/align.py            (from alignment.tar)
โ”‚   โ”œโ”€โ”€ all_filtered_frame_idx.json   # benchmark/filter_frames.py + manual filtering
โ”‚   โ”œโ”€โ”€ label.json            # benchmark/labeling.py
โ”‚   โ”œโ”€โ”€ shots/                # tf_selector/shot_boundary.py  (from shots.tar)
โ”‚   โ”œโ”€โ”€ whisper/              # tf_selector/transcribe.py     (from whisper.tar)
โ”‚   โ”œโ”€โ”€ segments/             # tf_selector/segment.py        (from segments.tar)
โ”‚   โ”œโ”€โ”€ segment_caption.json  # tf_selector/segment_captioning.py (VLM)
โ”‚   โ”œโ”€โ”€ volume.json           # tf_selector/volume.py
โ”‚   โ””โ”€โ”€ minmax_volume.json    # tf_selector/volume_minmax.py
โ””โ”€โ”€ features/
    โ””โ”€โ”€ <sport>/                 # extracted from features/<sport>.tar
        โ”œโ”€โ”€ vid_clip/            # HERO video CLIP features
        โ”œโ”€โ”€ vid_slowfast/        # HERO video SlowFast features
        โ”œโ”€โ”€ txt_clip/            # HERO query CLIP features
        โ””โ”€โ”€ aud_pann/            # PANN audio features

Videos are named <sport>_<idx>.mp4, where <sport> is one of: american_football, baseball, basketball, ice_hockey, race, rugby, soccer, volleyball.

Annotations

Artifact Produced by Description
annotations/alignment/ benchmark/align.py Per-video raw PSNR alignment between highlight clips and full-video frames.
annotations/all_filtered_frame_idx.json benchmark/filter_frames.py + manual filtering Final aligned frame index per highlight clip.
annotations/label.json benchmark/labeling.py Per-clip {0, 1} highlight labels (one score per 2-second clip) โ€” the evaluation ground truth.
annotations/shots/ tf_selector/shot_boundary.py Per-video shot-boundary detection (start/end frame index per shot).
annotations/whisper/ tf_selector/transcribe.py Word-level WhisperX transcripts.
annotations/segments/ tf_selector/segment.py Context-aware segments (2-minute maximum length).
annotations/segment_caption.json tf_selector/segment_captioning.py Per-segment captions from the InternVL2_5-8B VLM (input to the LLM scorer).
annotations/volume.json tf_selector/volume.py Per-clip audio loudness (dBFS) for every video.
annotations/minmax_volume.json tf_selector/volume_minmax.py Min-max normalized per-clip audio volume in [0, 1].

Features (QVHighlights-style)

Feature Directory Extractor
Video โ€” CLIP features/<sport>/vid_clip/ HERO Video Feature Extractor
Video โ€” SlowFast features/<sport>/vid_slowfast/ HERO Video Feature Extractor
Query โ€” CLIP features/<sport>/txt_clip/ HERO Video Feature Extractor
Audio โ€” PANN features/<sport>/aud_pann/ audioset_tagging_cnn

The query feature is the CLIP text embedding of the fixed query "Highlight of this {sport} video". Video and query features were extracted with the HERO Video Feature Extractor; audio features with PANN / audioset_tagging_cnn.

๐ŸŽฎ Environment Setup

benchmark/ and tf_selector/ run in a single conda environment (named svhighlights here, Python 3.9).

conda create -n svhighlights python=3.9 -y
conda activate svhighlights

# ffmpeg / ffprobe โ€” used by almost every script
conda install -c conda-forge ffmpeg -y

# Python dependencies (covers benchmark/ and tf_selector/ โ€” preprocessing + model inference)
pip install -r requirements.txt

# PyTorch with CUDA 12.1 โ€” required by tf_selector/ model inference
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121

# External packages installed from source:
pip install git+https://github.com/openai/CLIP.git        # benchmark/eval_clip_similarity.py
git clone https://github.com/soCzech/TransNetV2           # tf_selector/shot_boundary.py

For GPU use, install the PyTorch build that matches your CUDA version (the pipeline was run with PyTorch 2.3.1 / CUDA 12.1). The two external packages above are each needed by only one script โ€” skip either you do not plan to run.

Tested with Python 3.9, PyTorch 2.3.1 (CUDA 12.1), and ffmpeg 4.2.9.

Note โ€” tf_selector/main.py and tf_selector/segment_captioning.py log in to the Hugging Face Hub at startup via login(token=os.environ.get('HF_TOKEN')); export your access token before running, e.g. export HF_TOKEN=hf_.... In tf_selector/parse.py, replace the path/to/frame/{vid} placeholder with the directory that contains your per-video frames.

๐Ÿš€ Usage

The code is split into two pipelines โ€” see each directory for the full step-by-step commands and CLI arguments.

  • Dataset construction โ€” benchmark/: video trimming, highlight alignment, PSNR filtering, and per-clip label generation (paper Section 3).
  • TF-SELECTOR baseline โ€” tf_selector/: preprocessing (shot boundaries, ASR, context-aware segmentation, audio volume, VLM segment captioning) and training-free inference (LLM saliency scoring + prediction parsing) (paper Section 4). All preprocessing outputs are released under annotations/, so these steps are optional.

Every script accepts --sports to restrict processing to a subset of sports and -h for the full argument list.

๐Ÿ“Š Evaluation

To evaluate a highlight-detection model on SVHighlights, produce a prediction file with a per-clip saliency score for every video, then run eval.py.

Prediction file โ€” a JSON list with one entry per video:

[
  {
    "vid": "american_football_1",
    "pred_saliency_scores": [0.895, 0.880, 0.863, ...]
  },
  ...
]
  • vid โ€” video id (<sport>_<idx>)
  • pred_saliency_scores โ€” one predicted score per 2-second clip, in clip order (each video is divided into non-overlapping 2-second clips)

Ground-truth file โ€” annotations/label.json from the dataset: a JSON list of {"vid", "saliency_scores"}, where saliency_scores are the per-clip {0, 1} highlight labels.

Run:

python eval.py \
    --submission_path path/to/predictions.json \
    --gt_path        annotations/label.json \
    --save_path      path/to/results.json

eval.py reports four metrics, computed per sport and over all:

Metric Description
HL-mAP Clip-level mean Average Precision
HL-Hit1 Whether the top-scored clip is a ground-truth highlight
HL-Hitk Hit rate among the top-K clips (K = number of GT highlight clips)
HL-IoU Temporal overlap between the top-K predictions and GT highlights

๐Ÿ“ฉ Request Frame Data

We do not redistribute the original sports broadcasts. For research that needs the raw visual content, we share per-video frames sampled at 1 frame per second (1 fps) on request, subject to a research-only, no-redistribution agreement.

๐Ÿ“ฉ Request access to the frame data

Approved requests receive a Google Drive link with a 7-day access window. For any issue with the form, please email leedongkyu2019@unist.ac.kr or open an issue on this repository.

๐Ÿ“š Citation

If you find SVHighlights useful in your research, please cite:

@inproceedings{Lee_2026,
  title={SVHighlights: Towards Extremely Long Sport Video Highlight Detection},
  url={http://dx.doi.org/10.1145/3770855.3817564},
  DOI={10.1145/3770855.3817564},
  booktitle={Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2},
  publisher={ACM},
  author={Lee, Donggyu and Ki, Youngbin and Kang, Jeonghun and Kim, Taehwan},
  year={2026},
  month=Aug,
  pages={9254โ€“9265}
}

About

[KDD 2026 Datasets & Benchmarks Track] SVHighlights: a benchmark for highlight detection in extremely long sports videos

Resources

Stars

11 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages