Support

Akeeba Backup for Joomla!

#28245 Problem with memory exhausting during CRON backup

Posted in ‘Akeeba Backup for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Akeeba Backup version
n/a

Latest post by nicholas on Tuesday, 22 August 2017 06:11 CDT

drf
I've had this problem for a while, unfortately other tasks have had priority over this.
The server this is running on is cPanel based.
I run a daily backup locally.
If I run the backup via cron using the akeeba script, it completes.
If I run the backup via cron using the frontend backup system, it fails.
In both cases I can see that it eats up ALL physical memory when it runs (2 GB ram).
The site has quite a lot of images on it, as well as a few videos, in the form of an image gallery and a video gallery (in regards to large files).
Each backup archive takes up around 2 GB.

drf
I just tried monitoring it a bit closer and manually invoking the backup, and I've activated the debug log level as well, as soon as it starts backup up DBs, it goes from using almost nothing to using 1.6 GB ram. This keeps climbing when going on with files.
It is using MySQL (PDO) on the system.

tampe125
Akeeba Staff
Hello,

what's an approximate size of your database?
Can you please attach a log of the failed backup?

Davide Tampellini

Developer and Support Staff

๐Ÿ‡ฎ๐Ÿ‡นItalian: native ๐Ÿ‡ฌ๐Ÿ‡งEnglish: good โ€ข ๐Ÿ• My time zone is Europe / Rome (UTC +1)
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

drf
cPanel says that the MySQL folder takes up 290.72 Mbytes, this is including the piwik, and yourls databases that are also running on this host.
These are also backed up as external databases. The site database takes up 223345 when exported to an 'sql' file.
It looks like the backup doesn't fail after all, it had finished, I just saw that it displayed some 'red' indicator in the dashboard at a point.
I have the APC User cache enabled in PHP, and I tried disabling that in Joomla and PHP and run the backup again, just to see if it had any effect, it didn't change the ram usage.

tampe125
Akeeba Staff
The backup process is an intensive memory process, since we have to copy (read and write) the whole site plus the database.
If it's taking too many resources, you can slow down the process by tweaking the execution time. Please set the following:
Minimum execution time: 5 seconds
Maximum execution time: 3 seconds
Runtime bias: 50%

Yes, minimum is greater than maximum, that's not a typo. In this way the backup process will be slower.

Just a side note: looking at the log file, you are backing a lot of log tables (visits and click actions). Do you really need them? If not, you can exclude them from the backup.

Davide Tampellini

Developer and Support Staff

๐Ÿ‡ฎ๐Ÿ‡นItalian: native ๐Ÿ‡ฌ๐Ÿ‡งEnglish: good โ€ข ๐Ÿ• My time zone is Europe / Rome (UTC +1)
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

drf
I don't see much improvement, but come on 2 GB RAM for backup??? I mean even with a 200 MB database, it can hold the entire database 5-7 times over.
It also takes it forever to release the memory, but that is probably not an akeeba problem.
I've included a graph of memory consumption, and as you can see from it, I did a test this morning, and by now it still hasn't released the memory.

tampe125
Akeeba Staff
This is definitively a server issue, since Akeeba Backup runs fine with about 60 - 80Mb of available memory. Since you have almost full control over the server, do you have any details about the who is using that amount of memory?
It's the PHP process or the MySQL one? By the way, is the database on the same server or it's on an external machine?
At the end of the day, both the CLI script and the frontend method are using the same engine, it only changes the way they're invoked; to be honest usually it's the CLI script that consumes more resources since everything is done in one shot, there isn't the redirect "cooldown" period.
Do you have any system plugin installed, other Joomla stock ones? You could try to disable them one by one to and see if there's any improvement.

Davide Tampellini

Developer and Support Staff

๐Ÿ‡ฎ๐Ÿ‡นItalian: native ๐Ÿ‡ฌ๐Ÿ‡งEnglish: good โ€ข ๐Ÿ• My time zone is Europe / Rome (UTC +1)
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

drf
Ok, we've found out that the memory is eaten by diskcache, it is not memory allocations from the script.
It's based on Cloudlinux and cPanel, and they seem unable to find a way of slowing the script down, and not let it allocate everything for diskcache (which it then doesn't release in time, placing the site in error 500).

tampe125
Akeeba Staff
I'm sorry but this clearly is a server setup issue and it's not related to our software: I'm sorry but we can't help you any further.

Davide Tampellini

Developer and Support Staff

๐Ÿ‡ฎ๐Ÿ‡นItalian: native ๐Ÿ‡ฌ๐Ÿ‡งEnglish: good โ€ข ๐Ÿ• My time zone is Europe / Rome (UTC +1)
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

drf
Always interesting to be a customer and be caught in the middle between to suppliers that you are actually paying real money, and they each blame each other, leaving the customer that pays bills both ways as the looser. Classical IT supplier problems.

nicholas
Akeeba Staff
Manager
You seem to have a problem outside the scope of our support. Our software does not allocate memory on its own right. The problem you are experiencing seems to be with the wrong way CloudLinux addresses disk cache.

Since the introduction of UNIX in the late 1970s the disk cache has never counted against the virtual memory size. In fact, UNIX' mantra is "the RAM is sitting there using up power so let's use it to speed up the disk". As it's been written over and again "The Linux disk cache is very unobtrusive. It uses spare memory to greatly increase disk access speeds, and without taking any memory away from applications. A fully used store of ram on Linux is efficient hardware use, not a warning sign".

If you read the link above you'll see that the outcome of the first experiment on normal Linux is that the amount of disk cache memory allocated doesn't matter. When an application requests memory the operating system ditches disk cache in favor of applications. That's also the reason why the operating system does not expose an API for non-privileged (userland) applications to control the disk cache. Keep that last bit in mind.

Not on CloudLinux though. Someone decided to negate how Linux works by having disk cache count as application memory and not letting the operating system do what it has evolved to do best over 50 years, i.e. free up disk cache aggressively to yield more memory to the applications. Despite this abnormality, it STILL doesn't provide a userland API to control the disk cache.

The lack of a userland API means that userland applications (like PHP itself, on top of which Akeeba Backup, Joomla etc are scripted) do not have a way to communicate to the system that they do not want so much disk cache. You should ask your host what is the best way to set an upper limit to the maximum amount of disk cache allocated by CloudLinux. This is what you are paying your host for. Please ask them to fix this problem. It's ultimately and solely in their purview.

What you are paying us for is support strictly regarding our software. I can give you a workaround strictly concerning our software but it's not a solution (the problem is with the server, not our software) and it's a kludge. We thought that letting you know that the problem is not in our software and you shouldn't try a wrong workaround is more valuable than giving such a wrong workaround.

Anyway, if you insist on deploying a wrong workaround we will provide it for you with the stern warning that it's NOT what a reasonable person who knows at least the basics of how servers work should ever do. Should you choose to follow this workaround you are to relieve us from any liability whatsoever. We DO NOT want you to follow this advice, it's wrong. Here it goes. Instead of having one backup profile create many, let's say thirty, different backup profiles. One of them should be Full Site but have all files and folders excluded. The other 29 will be files only and each one include a portion of your site's contents. Restoring this site requires extracting the 29 Files Only backups first, then extracting and restoring the Full Site backup that only has the database. The backups should run with an adequate time space between them to allow the broken implementation of disk caching in CloudLinux to free up memory.

But, please, do me a favor and DO NOT follow this advice! If we were just to get your money and don't care about you, as you implied, we could have just given you this absolutely bad workaround and call it a day. Instead, I implore you to be reasonable and ask your host to fix the server issue because that's what you are paying them to do.

Now, here's the thing. I'd much rather prefer that we kept the discussion rational and helped you figure out the situation with your host, especially since you are most likely going to talk to some hapless first level support drone who has never seen the inside of a data center in photos, let alone know how to configure a server. That's what we do with the vast majority of our clients. Couple of tickets later they get to talk to a real engineer at the host and the problem is solved. As you can notice if you re-read this ticket, we always go above and beyond what we are strictly paid to do.

However, as soon as you start yelling "I paid you money" things get legal, liabilities are assessed and we have to invoke the Terms of Service (TOS), it being the legal agreement between you and our company, one which you agreed to unreservedly and explicitly when subscribing (i.e. before you paid us), to protect us from you. So, I have to regrettably inform you that per the Support Policy section of the Terms of Service your ticket is out of scope. As a result I have to close it without any further reply and, I'm afraid, do the same to any other tickets filed from you about or around the same issue.

Have a good day and good luck with your host!

Nicholas K. Dionysopoulos

Lead Developer and Director

๐Ÿ‡ฌ๐Ÿ‡ทGreek: native ๐Ÿ‡ฌ๐Ÿ‡งEnglish: excellent ๐Ÿ‡ซ๐Ÿ‡ทFrench: basic โ€ข ๐Ÿ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!