Skip to content

fix(sagemaker): preserve generated text in streaming responses - #23626

Open
dsilvadeepal wants to merge 1 commit into
run-llama:mainfrom
dsilvadeepal:fix/sagemaker-streaming-text
Open

dsilvadeepal wants to merge 1 commit into
run-llama:mainfrom
dsilvadeepal:fix/sagemaker-streaming-text

Conversation

@dsilvadeepal

@dsilvadeepal dsilvadeepal commented Oct 20, 2026 •

Copy link
Copy Markdown

Fixes #22221.

IOHandler.deserialize_streaming_output uses character-set stripping to remove the JSON envelope. This drops valid leading letters and trailing brackets/quotes; for example, data engineering becomes engineering.

Use exact prefix/suffix removal while retaining the existing fragment decoding behavior. Add regression coverage for full responses and fragments, including a mocked SageMakerLLM.stream_complete test that verifies cumulative text and deltas. No dependency changes.

Validation against main 619d746:

  • Before fix: 15 failed, 6 passed.
  • After fix: all 21 package tests passed; package coverage 79%.
  • Ruff lint/format and git diff --check passed.
  • Tests use synthetic payloads and a mocked AWS client; no live AWS validation.

This does not add buffering for arbitrary byte splits inside JSON escapes, UTF-8 characters, or the envelope.

Related earlier unmerged attempts: #22222, #22226, #22260, #22440.

AI assistance: Codex helped diagnose, implement, and test this change. I reviewed the changes and tested them locally before submission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: SageMaker LLM streaming silently corrupts generated text (lstrip/rstrip used as affix removal)

1 participant