Skip to content

[Feature] Wait for shared-filesystem output of completed Slurm jobs - #1104

Merged
jan-janssen merged 6 commits into
mainfrom
fix-slurm-output-grace-period
Nov 17, 2026
Merged

jan-janssen merged 6 commits into
mainfrom
fix-slurm-output-grace-period

Conversation

Copilot AI commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Fixes #1102

Problem

When a Slurm job left squeue and its _o.h5 file wasn't yet visible on the shared filesystem, the task was failed with "terminated without producing output", even though the job had completed successfully.

Change

  • When a Slurm job is no longer in the queue, query sacct for its state.
  • COMPLETED: don't fail; allow a 60 s grace period for the output file to appear (it is loaded as soon as it does). If it never appears, the task fails after the grace period.
  • Still queued/running states: don't fail.
  • Any other state (TIMEOUT, FAILED, OOM, ...) or unknown: fail promptly as before.
  • Flux behavior is unchanged.

Testing

Added unit tests (mocked queue/accounting) for delayed output, failed jobs and grace-period expiry; the targeted file-backend tests pass. A live Slurm reproduction wasn't possible as no Slurm is available in this environment.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d365f87d-5d8b-457c-b8b6-f55552a764f4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Co-authored-by: jan-janssen <3854739+jan-janssen@users.noreply.github.com>
@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.36364% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.16%. Comparing base (ab93e8b) to head (feb972e).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
src/executorlib/standalone/command_pysqa.py 86.36% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1104      +/-   ##
==========================================
- Coverage   94.24%   94.16%   -0.09%     
==========================================
  Files          39       39              
  Lines        2190     2210      +20     
==========================================
+ Hits         2064     2081      +17     
- Misses        126      129       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jan-janssen

Copy link
Copy Markdown
Member

@copilot Can the same analysis be extended to support the flux job scheduler and not only the slurm scheduler? In particular it would be great to use the pysqa package which should already provide an abstraction layer for checking the status of the job.

Co-authored-by: jan-janssen <3854739+jan-janssen@users.noreply.github.com>

Copilot AI commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@copilot Can the same analysis be extended to support the flux job scheduler and not only the slurm scheduler? In particular it would be great to use the pysqa package which should already provide an ...

Extended the shared output grace period to Flux using pysqa’s finished status; error still fails immediately. Added coverage for delayed output and bounded expiry. Targeted tests and mypy pass. Commit: 35167ab.

@jan-janssen jan-janssen changed the title Wait for shared-filesystem output of completed Slurm jobs [Feature] Wait for shared-filesystem output of completed Slurm jobs Nov 13, 2026
@jan-janssen
jan-janssen marked this pull request as ready for review November 17, 2026 23:07
@jan-janssen
jan-janssen merged commit c5f255a into main Nov 17, 2026
31 of 34 checks passed
@jan-janssen
jan-janssen deleted the fix-slurm-output-grace-period branch November 17, 2026 23:07
@jan-janssen
jan-janssen restored the fix-slurm-output-grace-period branch November 17, 2026 23:07
@jan-janssen
jan-janssen deleted the fix-slurm-output-grace-period branch November 17, 2026 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Avoid failing Slurm tasks before shared-filesystem output appears

2 participants