[dogancanbakir]

Fixes #2703

Input starting with a UTF-16 BOM (Windows PowerShell output) is decoded to UTF-8, and a UTF-8 BOM is stripped, for -l, -stream and stdin.

Summary by CodeRabbit

• Bug Fixes
  • Input files and standard input now correctly handle UTF-16 text with a byte-order mark and UTF-8 text with a byte-order mark. UTF-16 input is converted to UTF-8, while input without a byte-order mark continues to work as before.
  • Streaming and non-streaming file input now provide consistent handling of byte-order marks.

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Essentials
• Run ID: d2db3067-715f-4a05-ae83-7df1e45e4c68

Commits

Reviewing files that changed from the base of the PR and between bab476e and 53b37d9.

Files selected for processing: 2

• runner/runner.go
• runner/targetline_test.go

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

Walkthrough

The runner now decodes BOM-marked UTF-8 and UTF-16 input for files, streaming reads, glob-matched files, and stdin. Tests verify UTF-8, UTF-16LE, and UTF-16BE target files.

Changes

Input decoding

Layer / File(s)	Summary
Decoder and file iterator runner/runner.go	Added decodeInput for UTF-8 BOM removal and UTF-16LE/BE conversion. Added inputFileLines for decoded file iteration and read-error handling.
Runner integration and validation runner/runner.go, runner/targetline_test.go	Updated file, glob, and stdin input paths to use decoded readers. Added tests for UTF-8, UTF-16LE, and UTF-16BE target files in file-loading and streaming modes.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

• projectdiscovery/httpx#2703 — The change decodes UTF-16 input with a BOM for file, streaming, and stdin paths.

Possibly related PRs

• projectdiscovery/httpx#2691 — Both changes affect target-line parsing from runner file and stdin input paths.
• projectdiscovery/httpx#2668 — Both changes use the runner’s file and streaming input flow before target-line parsing.

Suggested reviewers: mazzz3r, mzack9999

Merge Risk: Minimal · up to 53b37

The reported fixture problem does not block the BOM input change. No actionable merge-blocking risk remains after normal checks.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title "fix(runner): decode input with a byte order mark" matches the main change. The runner now decodes UTF-16 input with a BOM to UTF-8 and strips the UTF-8 BOM for file loading, streaming, and …
Linked Issues check		Issue #2703 requires UTF-16 input files to be decoded so target lines can be processed. decodeInput uses unicode.BOMOverride for UTF-16 BOM detection and UTF-8 BOM removal. loadAndCloseFile appl…
Out of Scope Changes check		The changes stay within issue #2703 and the stated input coverage. The new streaming iterator is required to decode streamed files and preserve read errors. The added tests verify the input decoding b…
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…

Full details: Title check

Explanation

The title "fix(runner): decode input with a byte order mark" matches the main change. The runner now decodes UTF-16 input with a BOM to UTF-8 and strips the UTF-8 BOM for file loading, streaming, and stdin. The title is specific and concise, and it identifies the primary change from the developer's perspective. The summary also covers the added test for BOM handling, which the title does not need to mention.

Full details: Linked Issues check

Explanation

Issue #2703 requires UTF-16 input files to be decoded so target lines can be processed. decodeInput uses unicode.BOMOverride for UTF-16 BOM detection and UTF-8 BOM removal. loadAndCloseFile applies it to -l input. inputFileLines applies it to streamed files. The stdin path also applies it. TestRunner_readsInputWithBOM covers UTF-16LE, UTF-16BE, and UTF-8 BOM input for -l and -stream.

Full details: Out of Scope Changes check

Explanation

The changes stay within issue #2703 and the stated input coverage. The new streaming iterator is required to decode streamed files and preserve read errors. The added tests verify the input decoding behavior. No unrelated product behavior is changed.

Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 2 files.

Finishing Touches

Generate docstrings

• Commit to this branch
• Create a new PR

Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{A rabbit reads bytes by the moonlit trail
UTF-8 and UTF-16 now tell the tale
BOMs hop softly out of sight
Two target lines arrive just right
Files and stdin share the way
The runner bounds through dawn-lit hay}