Support for iglob

[ymoisan]

Hi,

How easy would it be to get a generator back from glob instead of a list ? iglob ? Some arg to the current fsspec glob implementation ?

In the context of pipelines (I'm thinking kerchunk) that could be an interesting feature.

Thanx !

[martindurant]

cf fsspec/s3fs#670

[martindurant]

Really it's find and/or walk that needs to be the generator. I would be supportive of ifind and iglob methods. It's doable, but would need some careful refactoring, because various implementations override some of the methods.

If the size of the glob output a bottleneck for you? Or is it the upfront time spent listing files? I would be surprised if either is significant compared to kerchunk-scanning each file.

[ymoisan]

I don't have issues with lists now really. I'm just thinking if we want a process to kerchunk a very large number of files it would be nice to pipe files directly to kerchunk without the memory overhead of a potentially huge list. So it's not a matter of time generating the list; rather just be able to pipeline things. Makes sense ?

[martindurant]

Yes, it is reasonable, but I would be tempted to defer until it's really a problem. I imagine if we were processing such a large number of files (>>million), we would meet other limits sooner.