Small files aren't compacted based on the block size

[tchaton]

What happened:

I have started a JuiceFS filesystem

docker run --network host -v /teamspace/studios/this_studio/data:/data redis

juicefs format \
  --block-size 4194304 \
  --storage s3 \
  --bucket https://lai-yabbering-purple-59op-km69.s3.us-east-1.amazonaws.com \
  --access-key ... \
  --secret-key ... \
  --session-token ... \
  redis://localhost:6379/1 \
  mystor6

sudo juicefs mount \
  --buffer-size 41943040 \
  --writeback \
  --prefetch 4 \
  --attr-cache=1 \
  --entry-cache=1 \
  --dir-entry-cache=1 \
  --cache-dir /teamspace/studios/this_studios/juicefs_cache \
  -d redis://localhost:6379/1 \
  /mnt/myjfs

import os
from tqdm import tqdm

FOLDER_DIR = "/mnt/myjfs"
NUM_ASSETS = 1000  # Reduce for testing
NUM_FILES = 1000
DATA = b"A" * (4 * 1024)  # 4 KB of data

open_files = {}

# Open all files first
for asset_idx in tqdm(range(NUM_ASSETS)):
    asset_dir = os.path.join(FOLDER_DIR, str(asset_idx))
    os.makedirs(asset_dir, exist_ok=True)

    for file_idx in range(NUM_FILES):
        file_path = os.path.join(asset_dir, f"{file_idx}.csv")
        open_files[file_path] = os.open(file_path, os.O_RDWR | os.O_CREAT)

    # Write to all files
    for file_path, fd in tqdm(open_files.items()):
        os.write(fd, DATA)

    # Close all files at the same time
    for fd in open_files.values():
        os.close(fd)

    open_files = {}


print("File writing complete.")

I was writing a lot of very tiny files to it.

But it seems the files aren't compacted.

What you expected to happen:

I want to be able to specify a minimum block size, so files gets compacted together, not just splitted if they are larger than the block size.

So I can read those files back and get the gain of fetching the full block size.

Ideally, I would like to control the compaction to ensure things are closely grouped together.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version:
• Cloud provider or hardware configuration running JuiceFS:
• OS (e.g cat /etc/os-release):
• Kernel (e.g. uname -a):
• Object storage (cloud provider and region, or self maintained):
• Metadata engine info (version, cloud provider managed or self maintained):
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others:

[jiefenghuang]

Different small files will not be compacted together

[tchaton]

Hey @jiefenghuang. I think you should have considered this issue before closing it without providing a clear explication on why this isn't or won't be supported.

Compacting small files together is the only way to make general data in the cloud fast.

Right now, if I want to use JuiceFS to backup github repos, etc... It is plain slow and not a suitable solution.

Anyhow, sad to see the lack of consideration for fundamental issue with the project.

[jiefenghuang]

You can refer to our official documentation for more information. In your scenario, you can utilize writeback caching for acceleration, along with adjusting other parameters for optimization.

Additionally, the issue of massive amounts of small files is a fundamental challenge in object storage, which is typically addressed by most cloud service providers offering object storage solutions.

[tchaton]

Hey @jiefenghuang. I referred to the docs. I was just curious why this wasn't something worth exploring.

writeback helps accelerating write a little bit but nothing life changing. Also, it doesn't solve the issue. Small files are still not compacted.

Ideally, I could specify writeback delay of 5 or 10 seconds, and all the files written during that time could be compacted together. Furthermore, even N seconds, files could be re-compacted to accelerate overall performance of the filesystem.

Additionally, the issue of massive amounts of small files is a fundamental challenge in object storage, which is typically addressed by most cloud service providers offering object storage solutions.

I know. I built LitData (https://github.com/Lightning-AI/litdata) to solve this for ML model training in the cloud. But I was hoping to find a mount solution tackling this for any sizes. I hope it was JuiceFS but it turns out it isn't.

Anyhow thanks.

[jiefenghuang]

This is not a common scenario. Including this would result in heavy logic garbage collection of data in object storage, and the merge strategy would also add significant complexity. These issues contradict the current architecture of the community edition of JuiceFS, which separates data storage into object storage. Addressing this matter directly within the object storage is more straightforward, and most cloud object storage services have already implemented numerous optimizations. Therefore, this is indeed not a priority issue. I will move the issue to question board for further discussion.