database is locked with sqlite

[mwqgithub]

What happened:

We were writing some files to the mount point.
A line that says database is locked appeared in the log:
May 18 01:51:16 ip-172-31-4-1 juicefs[6074]: juicefs[6074] : Read transaction succeeded after 4 tries (15.108270368s), last error: database is locked [sql.go:768].
And the access to the mount point became slow.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

It only happened once, so far we don't know how to reproduce it.

Anything else we need to know?

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version:
  juicefs version 1.1.0+2023-09-04.08c4ae6
• Cloud provider or hardware configuration running JuiceFS:
  aws
• OS (e.g cat /etc/os-release):
• Kernel (e.g. uname -a):
• Object storage (cloud provider and region, or self maintained):
  s3
• Metadata engine info (version, cloud provider managed or self maintained):
  a sqlite database located at an EBS data volume attached to the instance
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others:

[SandyXSD]

Usually it's caused by heavy write workloads. JuiceFS will retry the transaction as expected.

[jeffstearns]

I understand and agree. However, this can also happen when the filesystem is idle. It can be quite severe.

I observe this bug on Debian GNU/Linux 12 (bookworm) 6.8.12-8. Object storage is S3. Metadata engine is SQLite3 on the local machine.

I use JuiceFS for Proxmox backup. It is not used by any other process for any other purpose.

After installation and reboot, performance is excellent. This continues for about a week of use. Then each database access becomes very slow -- maybe 3000 mSec. Given that a filesystem operation may require tens of database accesses, a filesystem operation will time out after 50 seconds.

juicefs stats confirm that the bottleneck is the database, not S3.

The local filesystem and host are not the bottleneck. I/O performance is excellent, latency is very low, i/o queues are empty, and SMART status is perfect. The bottleneck is within sqlite3 itself.

This is further confirmed by running manual sqlite3 queries against the database. Even simple queries like .tables will time out due to database locks. I must set the sqlite3 query timeout to several thousand mSec in order to overcome this.

So the filesystem is idle, but the sqlite3 database has its knickers in a knot. Killing the juicefs process frees the database locks and returns sqlite performance to normal.

It's clear that JuiceFS is interacting with SQLite3 in a way that imposes long-running database locks even the filesystem is idle.

Update: SQLite3 has quirky semantics regarding transactions and locks. Since JuiceFS uses a generic SQL interface, I imagine that the problem lies in a mismatch between generic SQL semantics and the quirks of SQLite3 locks.

[davies]

When this happens, can you download the list of goroutines to see what the juicefs process is doing with sqlite?

[mwqgithub]

Could you please explain what you mean by a 'heavy' workload? Are you referring to high bandwidth usage or a high level of concurrency? In our case, we are writing about 260GB of data into JFS using a single thread. Would this be considered a heavy workload? Could this issue be related to S3 throttling?

[davies]

The heavy workload means concurrent access to SQLite. SQLite allow single write transaction at a time (locked), so concurrent metadata transaction of JuiceFS will be retried as normal.

[mwqgithub]

I see, is there anything we can do about this? Are there any configurations or parameters that could be helpful in this situation? Or should we switch to another type of metadata server?

[zhijian-pro]

I suggest using Redis as the metadata engine.

[mwqgithub]

Thank you for your advice.