Support

Akeeba Backup for Joomla!

#32106 – Used memory not being released

Posted in ‘Akeeba Backup for Joomla!’
This is a public ticket. Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.
Monday, 09 December 2019 09:58 CST
I have had a problem for a while with server memory usage increasing and have suspected this is due to Akeeba Backup, but have multiple sites running on a busy server so it's been tricky to diagnose. Today I set up a new server and can confirm that Akeeba Backup causes memory usage to increase considerably, and not reduce after it completes.

I created a new DigitalOcean droplet with 1GB RAM, running Ubuntu 18.04.3 (LTS) x64. I restored an Akeeba Backup from my live system into a brand new environment on this droplet. The database is a little more than 1GB, thanks mostly to one table of 900MB.

I ran a full backup using Akeeba Backup from the Joomla administrator. The backup worked fine. During this period, as can be seen from the attached graph, memory usage went from about 53% to a peak of 83%, and since settled to 75% but has not reduced in the 2 hours subsequently.

Not sure how easy this is to resolve! Anything else I can do to help investigate?

Cheers
-- Mike --
Custom Fields
Joomla! version (in x.y.z format) 3.9.11
PHP version (in x.y.z format) 7.3.12
Akeeba Backup version (x.y.z format) 6.6.1
 
mikeprince
Tuesday, 10 December 2019 00:01 CST
To put thing in perspective we're talking about 225MB. This is not a lot of memory and can be explained in various ways. Without having collected data from the server prior and after the backup or have htop open throughout the backup I can't tell you for sure what is happening on your server but I can tell you what is happening to my Ubuntu 18.04.3 (desktop and server) installations when I run a backup and after the backup is complete.

There are three primary places where memory can go, neither of which is a leak and is eventually put to good use by the system.

Most likely the majority or totality of the memory used is the filesystem cache. All UNIX systems try to utilize as much memory as possible to cache data from the filesystem in memory. If the system requires memory for something else they are going to use free memory first, then release memory from filesystem cache. That is to say, the filesystem cache is not being freed up on a timeout (it doesn't "expire"). It's only freed up when the memory needs to be used. This amount of memory appears in top as "buff/cache".

The other thing that may use up memory is Apache. Depending on which Apache MPM you are using it may be launching new threads / processes in anticipation of handling more requests. Usually, Apache MPM's configurations have a low and high threshold i.e. the minimum number of processes / threads to have in reserve and the maximum to launch. They may also have conditions for releasing processes / threads. It is possible that after a period of server activity you end up with some processes / threads in memory, ready to serve requests. Since it's a web server I'd say that's a fairly good use of memory.

Finally, if you are using PHP-FPM (FastCGI Process Manager) it also keeps its own pool of PHP processes. Like Apache, it has a low and max threshold and conditions for launching and releasing processes. By default it's set to start with a minimum of 3 processes, go up to 10 and keep at least 5 in memory.

Between all three of these you can easily account for 225MB.

Most important metric of your server IS NOT the memory used. It is swap. As long as Swap used is low and steady during normal server operation you have nothing to worry about. You need to worry and in fact worry a hell of a lot if you see used swap starting to grow under server load. This causes a very well known and documented "slow death" of Apache since requests are being backed up in Apache's queue which causes more swap to occur which makes processing them slower, which causes even more requests to get backed up... you get the idea. It's a positive feedback loop that leads to total system collapse (typically you can't even SSH into the poor server).

As to why you found yourself with higher memory usage during backup: again, it's not a lot of memory and it is perfectly understandable. You have a PHP process that is trying to access the entirety of your site's files and database contents. That's a lot of disk reads and writes and the system uses up as much filesystem cache as possible, with a very small reserve of around 128MB of free memory for emergencies (the kernel would like to be able to cope with sudden I/O spikes without having to scavenge for memory, thank you very much). Not to mention that the backup process itself also uses some memory with a typical spike between the memory used for the PHP binary and its modules, Joomla itself and its extensions and Akeeba Backup totaling near 100MB (out of which Akeeba Backup typically maxes out under 40MB). Once the backup stops the memory used up by the backup script itself is freed, the disk cache is not. There's your memory difference. It's totally fine and expected.

FYI you'd experience exactly the same thing using tar and mysqldump. File reads and writes are file reads and writes, no matter what triggered them.


Nicholas K. Dionysopoulos

Lead Developer and Director



Greek: native

English: excellent

French: basic



Please keep in mind my timezone and cultural differences when reading my replies. Thank you!



nicholas
Tuesday, 10 December 2019 03:00 CST
Thanks for taking the time to explain so clearly. I have had problems in the past with gradual system slowdowns, eventually resulting in the need to reboot, and with difficulty SSHing in, so have been trying to get to the bottom of this. I'll try to monitor swap more carefully as Apache 'thrashing' as you described seems a likely cause.
 
mikeprince
Tuesday, 10 December 2019 07:05 CST
I can guarantee that this is the reason. The slow death spiral you describe happens when Apache starts swapping like hell.

This is completely outside of scope of our support but I can give you some good pointers.

First read the Apache Performance Tuning page in Apache's documentation. It gives you a concise list of all the things you need to look into and how they affect Apache's performance.

The next must-read article is 3 small tweaks to make Apache fly which goes into minute detail about how much memory will be used by Apache and how to tune down MaxClients to prevent the slow death due to extreme swapping. In fact, if you read the last (Real-world example) paragraph in the MaxClients section I'm sure you'll recognize it to be the very problem you described having intermittently!

Armed with this knowledge and trawling through your logs you will be able to find out what is the best approach to follow. Decrease some Apache threading limits or add more RAM? Is the PHP memory limit and / or execution time unrealistically high, allowing for pages going rogue? Also ask yourself if you need all the services running on your server (Ubuntu enables quite too much by default if you ask me). Then start considering how you can make your site lighter. Is caching plausible or do you have a mostly personalized site like ours? Does using Redis or memcached make more sense than file caching for your use case? Can you get rid of a slow-loading, framework-based template and use a nimbler, speedier and less memory intensive hand-built one? Are all modules and plugins on your site necessary? Can you move your backups to a different time of the day, away from high-traffic periods and / or your site being crawled by search engines?

I hope that helps!


Nicholas K. Dionysopoulos

Lead Developer and Director



Greek: native

English: excellent

French: basic



Please keep in mind my timezone and cultural differences when reading my replies. Thank you!



nicholas
Thursday, 09 January 2020 17:17 CST
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.
system
This ticket is closed, therefore read-only. You can no longer reply to it. If you need to provide more information, please open a new ticket and mention this ticket's number.

Support Information

Working hours: Typically we work Monday to Friday, 9am to 7pm Cyprus timezone (EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets, but we cannot respond to them, outside of our working hours.

Upcoming unavailability: We will be closed December 23rd, 2019 19:00 to January 3rd, 2020 10:00 Cyprus timezone (EEST) due to the Holiday Season. You will not be able to file new support tickets or reply to existing ones.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!

Cookies Notification - Action required

This website uses cookies to provide user authentication and improve your user experience. Please indicate whether you consent to our site placing these cookies on your device. You can change your preference later, from the controls which will be made available to you at the bottom of every page of our site.