Support

Admin Tools

#31543 Admin Tools not playing nice with Amazon RDS (Aurora MySQL)

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Admin Tools version
n/a

Latest post by on Monday, 19 August 2019 17:17 CDT

bjmorgan
Our webserver lives within the AWS infrastructure. We recently migrated the database from the LAMP stack on our EC2 instance to an RDS instance (Aurora SQL). The website itself has seen a boost in performance with the exception of Akeeba Backup and Admin Tools.

Admin Tools requested to run the Wizard again, which regretfully and stupidly, I did. Yes, I know I shouldn't have after the fact, but what's done is done.

After running the Wizard, the connection to the site (administrator & public) would hang indefinitely and the connections to the database would bloat. The Joomla default .htaccess configuration was active. JCH Optimized was temporarily disabled, and the cache cleared. The only resolution was to manually disable Admin Tools via the terminal. With Admin Tools disabled, the site performs optimally.

Even after subsequent reinstallations, Admin Tools doesn't seem to be working well with RDS and the site hangs when trying to save settings from within the component itself.

Akeeba Backup has similar problems. It will not proceed past the "Backup Database" step. Since AWS provides regular backups of our RDS database, I can simply change the Akeeba Backup configuration to backup site files and skip the database, however, this doesn't address the problem.

I suspect it may be a database connectivity problem between Akeeba and RDS, but need some advice as to what parameters I should be investigating. The site itself works fine under RDS.

dlb
The problems that most people run into after re-running the First Run Wizard are that it can change the Secret URL Parameter and the password and user for the Password Protect Administrator. From the overall tone of your ticket, my guess is that's not the problem. It does raise a possibility, check the Security Exceptions Log. Any error that is triggered from WAF should leave a track there. I can't tell you what to look for.

Akeeba Backup should be getting the database information from configuration.php, the same place Joomla! gets it. It is possible to override that information in the Configuration settings of the profile. I've never seen an instance where that override was necessary. If you have connection information in the override section, please remove it and see if that fixes the problem.

Please go to the Joomla! Global Settings screen and test your email setup to make sure your site can send emails. If you have Admin Tools set to send you a notice for certain actions, and the email within the site doesn't work, Admin Tools will appear to hang while Joomla! tries to send the email.


Dale L. Brackin
Support Specialist


us.gifEnglish: native


Please keep in mind my timezone and cultural differences when reading my replies. Thank you!


????
My time zone is EST (UTC -5) (click here to see my current time in Philadelphia, PA)

bjmorgan
eMail works fine.

It's definitely a database connectivity/resources problem. I temporarily switched back to the localhost database (from an export of the current RDS database) and Akeeba Backup did just fine.

I've uninstalled Admin Tools for the time being. As soon as I get Backup to work with RDS, I'm thinking Admin Tools will follow suit.

I've also spun up a second, more powerful RDS instance and Akeeba Backup hangs on THAT one as well.

B.J.

nicholas
Akeeba Staff
Manager
As Dr. House said in TV series "Patients lie". His reasoning is that a patient will try to filter and explain their symptoms through the narrow lens of what they think they are suffering from. This is true for site owners who ask for assistance and yours is one of these cases. The database connection has nothing to do with your issues.

Admin Tools connects to the database through Joomla's database connection. In fact, we call \JFactory::getDbo() to get Joomla's database object and use it to connect to the database. Same for Akeeba Backup's interface.

The backup engine does pretty much the same. There's an internal class following the Decorator pattern and abstracts Joomla's database connection object which is to say that it merely passes everything back to Joomla's database connection object.

That is to say, your database connectivity is a red herring.

After running the Admin Tools Quick Setup Wizard you are resetting the Use IP Workarounds setting. If you have a load balancer in front of your EC2 instance OR if you are using Varnish, NginX or any other reverse proxy in front of Apache then you definitely need to enable this. Otherwise all requests will appear as coming from the same IP address. This will cause all requests to be reported as security exceptions and blocked which leads to increased database writes, a deliberate slow-down of permanently blocked IPs and that explains what you are seeing. It has nothing to do with the database connection (Amazon RDS for all intents and purposes to the outside world is a remote MySQL server, that's their unique selling point -- you can plug it in any existing application).

Regarding Akeeba Backup please file a new support ticket in the correct category and send us the log. I cannot address your issue based on what I already know is bad speculation rooted on an arbitrary and false premise.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

bjmorgan
I'm inclined to agree. I am confident that there's no problem with Joomla and Akeeba connecting to the database, I'm just wondering if there's a database configuration problem that would result in the site hanging or expending database resources once connected.

I have no load balancers for this single instance (no reverse proxies, either).

So let me go through my procedures with my observed outcomes, and, yes, it maybe chasing the rabbit down the wrong hole, but it's all I've got to work with right now.

Install Admin Tools with Localhost Database
- Export RDS database to localhost
- Install Admin Tools while configured to use localhost
- Run Quick Wizard Setup with Use IP Workarounds on, Save & close
- Site Runs (on localhost database)
- Export localhost database back to RDS
- Configure site to use RDS database
- Site hangs (RDS)

Adjust settings in localhost
- Manually adjust configuration.php to use localhost
- Site Runs (localhost)
- Adjust Admin Tools settings in localhost (turn off all WAF settings), Save & Close
- Site Runs (localhost)
- Export localhost back to RDS, switch configuration to RDS
- Site hangs (RDS)

Remove package from localhost for clean installation on RDS database
- Switch back to localhost via configuration.php
- Site Runs (localhost)
- Remove Admin Tools package while running DB from localhost
- Site Runs (localhost)
- Export localhost back to RDS, switch configuration to RDS
- Site Runs (RDS)

Admin Tools installation with RDS database
- Install Admin Tools while running RDS database
- Run Config Wizard (no WAF settings enabled)
- Site hangs at SAVE & CLOSE
- Manually disable main.php
- Site Runs (RDS)
- Uninstall Admin Tools package
- Reinstall Admin Tools package
- Run Config Wizard (Use IP Workarounds enabled)
- Site hangs at SAVE & CLOSE

So, I don't know what it is, but I know what the behavior is when using the localhost MySQL server vs. the RDS MySQL server.

nicholas
Akeeba Staff
Manager
When you say "localhost" do you mean that your site is running on your computer / laptop?

And no, Admin Tools does not listen to any port. It's not its own thing, it's a Joomla extension. Your web server listens to HTTP, it hands down the request to PHP which loads Joomla which handles the request and loads Admin Tools.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

bjmorgan
The LAMP stack on the EC2 instance. The MySQL server on the EC2 instance is what I'm referring to as the localhost.

bjmorgan
We've been running Admin Tools for years with no problems on this EC2 instance serving as a LAMP server. The only recent change was to move the database over to an RDS instance. Therefore, that's what I focused on as being the culprit.

nicholas
Akeeba Staff
Manager
Sorry for the delay. I had to run some tests.

I have to ask exactly how you are connecting Joomla to Amazon RDS Aurora MySQL. Have you set up a default MySQL connection in PHP? If so, that explains what you see.

The only way I could reproduce your issue is misconfiguring my Aurora MySQL instance to prevent TCP/IP connections. If you used Amazon's new interface (default) for creating an Aurora MySQL instance then the security group it created doesn't allow inbound TCP/IP traffic on port 3306. This has some side-effects. It's quite complicated how it works, but because Joomla tries to disconnect and reconnect its database after we use it in our decorated class it ends up trying to use the credentials in configuration.php which might cause PHP to hang for 30" while the OS TCP/IP stack is trying in vain to contact the remote MySQL server. That matches exactly what you describe happening.

Editing the Security Group and adding a new Inbound rule for port 3306 and an appropriate source matching your EC2 deployment this issue goes away. In fact, you can even add an inbound rule with your IP as the source to connect to the RDS instance remotely.

Doing the latter I even managed to have a site on my computer connected to RDS. It's slow as molasses due to network latency but Admin Tools works great and Akeeba Backup takes a restorable backup perfectly, if very slowly.

So, it looks like the problem has to do with how you have configured your RDS instance. There is nothing on our side of the code which would prevent you from using Amazon RDS Aurora MySQL in lieu of MySQL proper.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

bjmorgan
Currently the RDC instance is within a VPC and only allows connections to port 3306 from the private IP of the EC2 instance. It is not public facing.

nicholas
Akeeba Staff
Manager
I can tell you that it's definitely a configuration issue on the Amazon side of things. I have a site which connects to RDS and both Admin Tools and Akeeba Backup work reliably. I cannot tell you what could be the problem on Amazon's side because I neither have access to your AWS configuration nor is it under the scope of our support. My ability to help stops at determining that the issue is not down to a bug in our software. Sorry :(

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

bjmorgan
Yes, I figured it might be an RDC issue, but wanted to make sure. But why would the site perform optimally without Admin Tools while on RDC? I’ve added the EC2’s public Elastic IP to the RDC security group as test, but there were no changes in outcomes. I’ll try spinning up another RDC instance and make it public facing (I can’t modify the current RDC instance to have a public IP)

nicholas
Akeeba Staff
Manager
If that doesn't work for you let me know and I'll try to set up the whole EC2 + RDS combination sometime tomorrow or over the weekend (I need to finish up some releases and some other work first).

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

bjmorgan
I'll take a look into it more when I have some time, in the meantime at least I have a work around of keeping Admin Tools disabled and just backing up site files. I have a feeling it will will be mostly poking and prodding in the EC2 & RDS security group settings, MySQL configuration of the RDS instance, and server-side settings (PHP, Apache). I may try spinning up a fresh EC2/RDS combo as well and building it up from scratch.

However, I've got some other deadlines to meet at the moment as well (plus a week long vacation), so I'll let you know what I find.

nicholas
Akeeba Staff
Manager
No problem. Take your time. There was a magnitude 5.2 earthquake where I live earlier today. My day went a bit sideways to say the least. I don't think I'll have a realistic chance to run any tests over the weekend. Ping me when you are back from vacation if you have any ideas, I'll be interested to hear what exactly was going on.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

bjmorgan
Oh, be careful and safe! We've had our own spate of quakes in Southern California recently. They are no joke.

nicholas
Akeeba Staff
Manager
Thank you! Be safe too. I saw the images from the aftermath of the quakes in California.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

System Task
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!