Repository navigation
Pre-work
DescriptionI have an issue where ping times are in excess of 1000ms. I had UptimeKuma to check if the service is alive. Recently, there was a steady climb in response times to which there was a peak response of 17000ms before UptimeKuma registered it offline. I have also noticed that load times for the web UI takes a significant amount of time compared to other services running in docker.
I would like to note that the system has lost performance and this is likely the cause for the high response times. Nothing but Docker and Speedtest Tracker are running on the LXC. There are about 250-300 speedtests in the database. I am running this in a Proxmox Debian 12 LXC with docker installed. This is fairly standard when it comes to running Docker processes in an LXC from what I can tell. Here's the specs of the LXC: Expected BehaviorThe response time would be lowered to an absolute max of 250ms. Most services running under the same conditions run with single digit response times, but there are single spikes every now and then. Steps to ReproduceI don't think this is something only I am experiencing running this in an LXC. So I think just a regular install would work. If the response time is not what I am experiencing, try running in an LXC with the same specs listed above.
Deployment EnvironmentDocker Compose Application Informationphp not installed? Runs in a docker environment.What browser(s) are you seeing the problem on?Brave, Chrome, Insomnia HTTP Requests, UptimeKuma Pings Logs |
Replies: 15 comments
|
Just for my understanding about the problem. What are you pinging in these tests? Is the response time from the healthcheck endpoint on |
|
This data shows me pinging the web GUI at the
I am investigating to see if it is a network configuration issue, however I am not seeing that it is the case, as I can ping the server just fine with <1ms of latency This appears to me to be an issue with the web server, but I could be wrong. |
|
Let me know if you find anything as you dig deeper, I'm not seeing the same issues when pinging the server vs. the application. 
|
|
After some investigation, it appears that the entire LXC container is running slow. I'm still checking to see if it is my issue or yours. I have done reboots of the LXC and the node associated with it. The issue is still present. There are no random large files or anything. The LXC has plenty of resources available, so nothing is being bottlenecked. Nothing is even being used. from this assumption I can only think it's something with the web server not responding quickly. I can no longer connect to it via the web interface on a browser to see the data. I believe this issue occurred running v1.8.0. I am currently updating the docker compose file, and will update you on the results. |
In the past I've seen instances where LXC containers have had poor performance. Could be unrelated but this user found a fix for their issue: #1672 |
|
I think that issue was because the computer disabled his PCIe NIC due to ASPM. I have disabled it anyway to see if it fixed the issue, but it did not work. This is definitely something with the service, as the speed of the LXC has returned, but the pings have not gone to a normal level.
If you have any ideas, let me know. I'm out of ideas. It will sit and wait for a 200 code, and then load the contents of the data immediately. It is a little strange to me, it isn't buffering like a normal slow website would. I am still unable to connect to it via a browser. The problem seems intermittent as well, as randomly I will get a 2 second health check, then I will get a 20 second one. I am stumped at the moment. Edit: More times than not, it will not be a short ping. 98% of the time it would be 20+ seconds. |
|
Do you know the last version where the API response times were still fast? We haven't changed anything to the API in a long time that might cause this delay. |
|
I don't know what version the API times were fast on. I've only ever ran 1.8.0 and it slowly got worse as time progressed. I just ran a Docker compose pull, but the issue still persists. |
|
@Jacob8ch would you be willing to test a different image? |
|
Sure, I'd be willing to do that. |
I've been testing a new image which is plug-n-play with the LSIO one to use it:
So far @svenvg93 and I have been seeing pretty significant improvements in application performance across the board, we're seeing API response times comparing the two images from ~160ms (LSIO) to ~30ms (mine). |
|
👋 This issue has been automatically marked as stale due to inactivity. If this issue is still relevant, please comment to keep it open. Without activity, it will be closed in 7 days. |
|
After running this, my problem is still present. I have reinstalled it on a new LXC and this issue is starting to come back again on the new container. Perhaps there is some background process, memory leak, or cache that isn't cleared that is slowing the machine down? |
|
Generally speaking I haven't supported LXC and there has been a history of issues with performance problems. I'm converting this to a discussion so the community can provide additional support/recommendations. |
|
It's been some time. This issue was caused by a failing Ceph cluster in my PVE instance. I replaced a couple of drives and the problem was solved. This ended up not being your fault, but I am glad this came to light. Thanks for the speedy responses, and sorry to bug you again. |
It's been some time. This issue was caused by a failing Ceph cluster in my PVE instance. I replaced a couple of drives and the problem was solved. This ended up not being your fault, but I am glad this came to light. Thanks for the speedy responses, and sorry to bug you again.