Skip to content

fix(core): pass str paths to fsspec in SimpleDirectoryReader - #23664

Open
muthukkumaranb wants to merge 1 commit into
run-llama:mainfrom
muthukkumaranb:fix-dirfs-pathlike
Open

muthukkumaranb wants to merge 1 commit into
run-llama:mainfrom
muthukkumaranb:fix-dirfs-pathlike

Conversation

@muthukkumaranb

Copy link
Copy Markdown

Description

SimpleDirectoryReader passes Path/PurePosixPath objects directly to fs.open(). Most fsspec
filesystems accept these, but DirFileSystem._join() only handles str, list and dict, so it tries
to iterate the path and fails with TypeError: 'PurePosixPath' object is not iterable.

This PR converts the path to str at the three fs.open() call sites (read_file_content,
load_file, aload_file). Other filesystems are unaffected because they already accept plain strings.

Fixes #23663

New Package?

  • Yes
  • No

Version Bump?

  • Yes
  • No (llama-index-core is exempt)

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • I added new unit tests to cover this change

Added a dir_fs fixture (a MemoryFileSystem wrapped in a DirFileSystem) and three tests covering
load_data, aload_data and read_file_content. All three failed before the fix with the TypeError
and pass after it.

uv run -- pytest tests/readers in llama-index-core: 34 passed.
ruff format --check and ruff check pass on the changed files.

Suggested Checklist:

  • I have performed a self-review of my own code
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective
  • New and existing unit tests pass locally with my changes

AI disclosure: I used an AI assistant (Claude) to help locate the call sites and draft the
regression tests. I reproduced the bug myself, reviewed every line, and ran the tests listed above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: SimpleDirectoryReader fails with fsspec DirFileSystem: 'PurePosixPath' object is not iterable

1 participant