Support

Admin Tools

#34918 Error connecting to the server: 403 - updating from 6.0.3 to 6.0.4

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Admin Tools version
n/a

Latest post by on Saturday, 24 April 2021 20:17 CDT

adiehm

I've checked the update key is in place.

I've saved the options to ensure the key is saved in the Joomla updates table. 

I've tried unpublishing "Installer - Admin Tools Professional" and "Installer - Akeeba Backup Professional"
( I read about doing that on your site some where)

The only thing that seems to work is to install 6.0.4 manually
(i'm not keen to do that on 100+ sites)

There has to be a better solution

nicholas
Akeeba Staff
Manager

Hello and thank you for contacting us about your update issue.

There is no problem with your account or our site. You have hit a Joomla bug. Please note that all of the following information is already in our documentation.

Before we begin please understand that extension updates are discovered, downloaded and installed by Joomla itself, using code that's under the absolute and total control of the Joomla project. Akeeba Ltd can not and does not have any influence on that code. In simple terms, your issue is a Joomla bug which is NOT influenced by anything we have done, currently do or can ever do. We can only offer a workaround and an explanation.

Workaround

IMPORTANT! Please follow all of these steps in the exact order provided below, without skipping any one of them even if you think it is unnecessary. Each step and the order we tell you to follow is the result of dozens of hours poring over Joomla's code, understanding how it works and figuring out the shortest workaround to many weird things Joomla does.

For each one of your sites:

  • Go to the backend of your site
  • Go to our software's main page e.g. click on Components and then on Akeeba Backup or Admin Tools or Akeeba Ticket System Professional depending on which of our software you have.
  • Click on Options in the top right corner. This opens Joomla's com_config component showing the settings for our software.
  • Clear your Download ID.
  • Click on Save & Close.
  • Click on Options in the top right corner again.
  • Enter your Download ID.
  • Click on Save & Close.
  • If you have more than one Professional version of our software installed repeat the process from the beginning until here for each one of: Akeeba Backup Professional, Admin Tools Professional, Akeeba Ticket System Professional.
  • Go to Extensions, Manage, Update.
  • Click on Clear Cache.
  • Go to System, Clear Cache
  • From the dropdown select Administrator
  • Click on Clear All even if the list is empty. Joomla has a hidden query cache which screws up updates if you don't clear it.
  • Go to Extensions, Manage, Update.
  • Click on Clear Cache again.
  • Click on Find Updates.
  • You should now be able to perform the updates to our software


Explanation

Joomla handles the extensions updates on its own, using code inside Joomla itself. This feature was originally written to only support free extensions. Back in Joomla! 3.2 support for paid extensions was added. The following is a technical explanation of how Joomla finds updates and how it determines the download URL when you choose to update a paid extension.

How updates work

Joomla cannot read the Download ID from the component's configuration. Whenever you click on Save & Close in our software's Options page you come back to the main page of our software. There, we check if the Download ID in the Options has changed. If it has, we find the row in the #__update_sites table for our software and add

dlid=THE_DOWNLOAD_ID_YOU_ENTERED

in the extra_query column.

When Joomla tries to find updates it reads the contents of the URL in the location column of the #__update_sites table. This is a static XML file listing all published versions of an extension. Joomla finds the latest version compatible with your Joomla version. Then it checks the latest version with the version currently installed on your site. If the version is different (not necessarily newer, mind you!) it decides there is an update available. At this point it creates a new row in the #__updates table and copies the extra_query column from the #__update_sites table's row to the new row in the #__updates table. At this point Joomla HAS NOT YET determined the download URL for the new version.

When you select an extension and click on the Update button Joomla reads the corresponding row of the #__updates table. It goes ahead and reads the contents of the detailsurl of that row. This is the same XML file it read to determine if there's an update available. Now it reads the downloadurl information from the XML file which contains the download URL for the new version without the Download ID. At this point Joomla combines this download URL with extra_query column of the #__updates table's row to create the real download URL. Moreover, it calls the XXX event to let any plugins in the "installer" group to modify the download URL. It then tries to download the new version's ZIP file from this modified download URL.

Why you get a 403

As you can understand from the explanation above, if the extra_query column in the #__update_sites table is empty or contains an invalid Download ID when Joomla tries to find updates then the download URL will contain no or an invalid Download ID. As a result our site will forbid downloading the update with a 403 Forbidden error message. Joomla suppresses all information except the 403 error code.

The only way to correct this issue is to replace the extra_query in the #__update_sites table, clear all updates, have Joomla find all updates again (so the #__updates rows have the updated extra_query) and then retry to apply the updates.

Why do you need the complicated workaround

This brings us back to Joomla doing some really weird things with updates.

For starters, it does not read the Download ID from the component's Options, it uses the extra_query column in the #__updates table. The only way to work around that and have our component replace the extra_query column in the #__updates table is to clear and re-enter the Download ID which triggers our code that applies the extra_query.

The second weird thing is that the extra_query column appears twice, once in the #__update_sites and once in the #__updates table. Therefore we need to replace the #__updates table records. The only way to do that is through Joomla's extension updater's Clear Cache which empties the #__updates_table.

However, Joomla does something which makes no sense. It caches the database queries! Even though we replaced the extra_query column in #__update_sites and cleared the #__updates table (i.e. the updates cache) Joomla will STILL see the old, non-working updates unless we go to Clear Cache and clear the administrator cache. Even funnier, the database query cache which is part of the "system" cache may not even show up in the Clean Cache page, totally misleading the user! That's why we ask you to blindly click on Clear All on that page.

Now that the queries cache is cleared we might still see updates because Joomla did not, in fact, clear the updates cache (the #__updates table) when we told it to because of the database query cache. That's another thing that makes no sense. Therefore we need to click on Clear Cache in the Update page again to have Joomla actually clear the update cache.

At this point we can now tell Joomla to Find Updates which will trigger the entire process we detailed above, finding the updates and using the correct extra_query. At this point applying updates works fine since we've worked around all the weird things Joomla does when trying to find and apply updates. Lucklily, this is only required once.

Why did that happen

The problem is that the extra_query column in the #__updates table went missing. There are two reasons why this could happen:

1. You clicked on the Refresh Update Sites button in the Updates page. This feature is VERY BUGGY. It silently removes the extra_query column from ALL RECORDS of the #__update_sites table. You get no warning whatsoever.

2. A third party extension triggered the Refresh Update Sites feature or stupidly deleted the extra_query column of the #__updates_sites table.

Neither of these two root causes are under our control, nor can we do anything to stop that from happening.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

adiehm

I did try that a very similar sequence of events as i've seen that issue with the Download ID before.

SO I tried your steps and it makes no difference.

 

Go to the backend of your site
Go to our software's main page e.g. click on Components and then on Akeeba Backup or Admin Tools or Akeeba Ticket System Professional depending on which of our software you have.

Note : I started with Admin Tools

Click on Options in the top right corner. This opens Joomla's com_config component showing the settings for our software.
Clear your Download ID.
Click on Save & Close.

Insert extra steps here:
Save and close kicks you out of the Joomla back end
Use browser back button to return to the options page in the joomla back end
Click on Cancel to get back to Akeeba Admin Tools

Click on Options in the top right corner again.
Enter your Download ID.
Click on Save & Close.

Insert extra steps here:
Save and close kicks you out of the Joomla back end
Use browser back button to return to the options page in the joomla back end
Click on Cancel to get back to Akeeba Admin Tools

If you have more than one Professional version of our software installed repeat the process from the beginning until here for each one of: Akeeba Backup Professional, Admin Tools Professional, Akeeba Ticket System Professional.

Note: I Repeated process for Akeeba Back up

Note: Saving Akeeba Back up options does not evict you from the joomla back end like Akeeba Admin does

Go to Extensions, Manage, Update.
Click on Clear Cache.
Go to System, Clear Cache
From the dropdown select Administrator
Click on Clear All

Note: its Delete All not Clear All

even if the list is empty. Joomla has a hidden query cache which screws up updates if you don't clear it.
Go to Extensions, Manage, Update.
Click on Clear Cache again.
Click on Find Updates.
You should now be able to perform the updates to our software

Result : Makes absolutely no difference - Akeeba Admin Tools still fails

Warning
Error connecting to the server: 403
Before updating ensure that the update is compatible with your Joomla! installation.
You are strongly advised to make a backup of your installation before you start updating.

Error
Failed to download package. Download it and install manually from https://www.akeeba.com/download/admintools/6-0-4/pkg_admintools-6-0-4-pro-zip.zip.

adiehm

Ok i have figured what is going wrong and it sort of aligns with your description

The Extra_Query field is empty for Akeeba Admin Tools but not Akeeba Back up. It should not be empty as its been in there and working on 100's of sites for years. Something very recent has emptied that field. Possibly the previous update to Akeeba Admin Tools 6.0.3 or Joomla 3.9.25 as a suggestion.

No amount of adding and removing of the download key via the Options screen makes any difference. I think I've tried every combination of adding, removing, cache cleaning I can think of. Version 6.0.3 simply will not save the Download Key into the Update_Sites table.

If i manually add the Download ID into the Extra_Query field via PHP My Admin then the update works perfectly :)

How or Why the Extra_Query field is empty for Akeeba Admin Tools for 100+ sites i have no idea.

Why trying to save the Download Key using version 6.0.3 does not work I have no idea.

Interestingly after updating to 6.0.4 no amount of adding, removing, cache cleaning will remove the Download key from the Extra_Query field

 

 

nicholas
Akeeba Staff
Manager

The Download ID seems to be removed from the extra_query when you update an extension. I've seen that happening with JCE Pro as well so it doesn't look it's an issue we caused in any way. More frustratingly, I have not seen it happening in all sites. My sites (development, testing and live) seem to have survived. My wife's sites were affected. We are both on the same host. I have no idea why this happens on some sites but not all of them. It seems that the sites which failed the installation of the first Akeeba Backup 8 and Admin Tools 6 versions were the ones also affected by the Download ID disappearing from the extra_query in #__update_sites. Correlation doesn't mean causation because on my test site where I inflicted the same problem with updates I can't seem to be able to reproduce the extra_query being cleared.

As for why the Download ID cannot be applied, I can tell you why since I've debugged this issue this morning on an affected site that belongs to my wife (it really helps that we are working from home on adjacent desks). Joomla 3.9.25 introduced yet another bug, this time with the XML forms' hidden field. This is used in the Admin Tools config.xml file which is rendered by Joomla's com_config when you click the Options page. The bug is fatal on PHP 8 but merely a warning on PHP 7. However, on PHP 7 it causes clicking on Save & Close in the Options page (which is rendered by Joomla itself) to redirect you to the public homepage of your site instead of taking you back to Admin Tools. This makes managing your Download ID in Admin Tools very hard. Since the Options are rendered by com_config we have no way of adding a post-save hook. What we do is ask Joomla to pretty please redirect back to Admin Tools' Control Panel page which detects the change in the saved Download ID and applies it to the #__update_sites table. Since this redirection is broken the code doesn't run which is why you have this problem. This is a Joomla 3.9.25 problem, not an Admin Tools 6.0.4 problem.

You will need to follow a workaround.

Go to Admin Tools. Click on Options, Update. Remove your Download ID. Click on Save. Go to Components, Admin Tools. This clears the extra_query and tells Admin Tools a Download ID is needed. In fact, it should prompt you for a Download ID at the top of its Control Panel page.

You can either use the prompt to enter your Download ID or follow the manual approach: click on Options, Update; enter your Download ID; click on Save; go to Components, Admin Tools. In both cases the #__extensions table is updated with the new Download ID, you are back to Admin Tools' Control Panel page and the extra_query is updated in the #__update_sites table.

However, this is not enough if an update (e.g. 6.0.3 to 6.0.4) has already been found. Instead, you need to follow the next steps in precisely the following order to work around another bug in Joomla.

Go to Extensions, Manage, Update. Click on Clear Cache.

Now go to System, Clear Cache which is a completely different page that has a confusingly similar name. Click the drop-down menu on that page that currently reads Site and choose Administrator. The page reloads. After the page reloads click on Delete All in the toolbar.

Now go to Extensions, Manage, Update. Click on Clear Cache.

After the page reloads click on Clear Cache again. Seriously, this is required. I know it makes no sense, but you really do need it because the first Clear Cache may not update the last update check time in #__update_sites according to my debugging.

Now you can click on Find Updates. After the update to Admin Tools is found you can install it just fine.

In case you lost track, this is three (3) different Joomla core bugs you need to work around. I have been reporting two of them privately since 2014. No action taken from the project. I made a public report on March 5th and was met with open hostility and threats to our company. So, no, I won't bother reporting the third one. They just don't care or at least they treat me like garbage ever since my first ever bug report to the Joomla project all the way back in 2009.

I would very much prefer if we didn't have to use the completely broken Joomla extensions update feature which causes all these issues. However, the Joomla project FORCES us to use it since 2017 on pain of permanent removal from the Joomla Extensions Directory.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

adiehm

Hi Nicholas.

Yes that seems to work - what a pain in the backside to fix - I have 80 sites to apply this fix to :(

Maybe its time to abort mission on using the #__Extra_Query field to store update keys.

I've noticed that JCE Pro, JCH OPtimise Pro, OSMap Pro all use alternative methods to store the download key these days.

Interestingly Regular Labs still uses #__Extra_Query field to store update keys.

 

nicholas
Akeeba Staff
Manager

The extra_query is the canonical way in Joomla 4. So making a change to that now, right before Joomla 4 is out, makes no sense. I saw that today someone else sent a PR which will address the issues in Joomla. Hopefully it gets merged into Joomla 3.10.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

System Task
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!