Support

Site Restoration

#30721 restoration from Google Drive

Posted in ‘Site restoration’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

PHP version
n/a
CMS Type
Other
CMS Version
n/a
Backup Tool Version
n/a
Kickstart version
n/a

Latest post by on Saturday, 26 January 2019 17:17 CST

interworld
Does restore from Google Drive works?
This was the only reason for us to upgrade to PRO as we need regular backups to a remote location (Google Drive), but looks like we were not informed that the restore would not work. So what is the point of storing remotely to GD if we cannot restore? Even the direct download from GD may be corrupted as we are using browser download function, correct?

nicholas
Akeeba Staff
Manager
This is incorrect. You can of course restore a backup located in Google Drive or any other remote storage.

If you still have access to your site's backend you can go to Manage Backups, find the backup entry you want and the click on the Manage Remote Backup button. This lets you fetch the backup archive back to your server. Once the archive is back on the server you can restore it. This is the documented workflow for typical restorations following a rather small, typicalyl self-inflicted, snafu which renders parts of the site inaccessible.

If you do not have access to your site's backend (or if your server, hosting company or data centre is no longer in business or has been wiped out in a natural or man-made disaster) you can download the archive from Google Drive to your computer and upload it to your new server. Then you can use Kickstart to restore it. This is the disaster recovery scenario and it also works if you want to make copies of your site.

Alternatively, if you are really tight on space, you can extract the backup archive on your computer and upload all the extracted files to your server. Trying to access your site will launch the restoration script, located in /installation on your server. This is NOT the regular restoration method. It is merely the second-to-last resort option when space is at a premium and you have no way to have both the backup archive and the site's files fit on the server at the same time. It's unlikely you will need it but I thought I should mention it anyway since it's not very clear why you wanted to restore directly from off-site storage.

Please do remember that PHP can not open a file for binary read (what you need to extract a backup archive or any other archive such as a ZIP file using any PHP-based software, not just our own) unless the archive is stored in a location supported by the PHP fopen stream wrappers and allows rapid random access of arbitrary lengths (as opposed to sequential download of the entire file). Due to the way Google Drive and other remote storage providers work you cannot fulfill that requirement. If you were to try to restore directly from them you'd run out of PHP memory or hit the PHP or web server timeout. Basically, restoration would fail. We can't offer a restoration option which would fail on all practical scenarios for reasons that have nothing to do with us. That's why you ultimately need the archive on your web server to restore it, we do offer the tools to do it and have documented them as well.

Moreover, I believe that you are missing the real reason for having your backups remotely stored: it ensures that if your site, server, hosting company or data center goes away your backups are not lost with it. Sites do get hacked with the attackers trashing / wiping them. Disk drives and hardware, in general, does fail. Hosting companies do get hacked or go out of business, all data (and their backups) gone with zero warning. Data centers do get affected by natural or man-made disasters. Putting your backups outside your server ensures that they will survive even in the event of a disaster like that. More realistically, half the times someone had to restore a site from a backup is because they deleted or overwrote more than they thought they would. Backups on the server are very likely to be affected in this case. So soring them off-site is always a "must have" and documented as such. Finally, remotely stored backups don't take up space on your site's server but that's just a secondary reason. The primary reason is always being able to have a reliable backup at hand when it all goes pear-shaped.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

interworld
Hi Nicholas, thank you for the reply.

My initial comment was after searching Akeeba support topics for the related process when I found this:
https://www.akeebabackup.com/support/site-restoration/Ticket/29795-kickstart-pro-import-from-url-fails-to-get-archive-from-google-drive.html

Where you said:
...You need to do some manipulation of the URL to get a direct download URL for Google Drive. Please note that this trick may no longer work. I've not tried it myself. I no longer use Google Drive ...

So this raised 1st concern.

The second concern was when clicked on Manage remotely stored files button, we get a popup with this info (see attached):
-----Remotely stored files management
[red delete button]
Download to your desktop
Sorry, the currently selected engine does not support downloading remotely stored files to your PC or you have already deleted the files from the remote server.----

Then we tried Transfer Site option, but we keep getting errors after Validating:
cURL Error 67 connecting to remote FTP server: Access denied: 530
or: Could not connect to the remote FTP server.

When using the same FTP parameters in Filezilla, we can connect to the server.

So, where do we go from here?
Akeeba PRO features are not working for us

nicholas
Akeeba Staff
Manager
I am under the impression that you have already formed the opinion that our software doesn't work. The evidence you have collected and presented to me so far points to the opposite direction. I have to produce a much longer response than I'd like to as I first need to explain why the Professional features do, in fact, work before we move onto troubleshooting the issue you seem to have with the Site Transfer Wizard. I would like to kindly request keeping a more open mind and be willing to work with me so I can help you. I'm not here to debate; I am here to help you. Thank you in advance.

Kickstart can import files from URLs using the Import from URL feature. As documented, the URL has to be the actual URL which downloads the file. In case it's not clear, this means that accessing the URL should result in the remote server sending a byte stream accurately representing your file contents. Google Drive does not ever you an actual download URL. The sharing URL they generate for you is an application page. This means that it is an HTML page which uses JavaScript to create a temporary download URL when you click the Download button on it. That temporary download URL cannot be captured -so that would be able to use it with the Import from URL feature of Kickstart- in any way that's sensible for non-developers, it is only valid for the browser session that generated it -so even if you could capture it it is useless since Kickstart downloads the file through your server, not your browser- and expires near instantly. Therefore you cannot get the actual download URL to the file from Google Drive and feed it to Kickstart's Import from URL feature. Therefore your problem is not with Kickstart being unable to download a file, it's with Google Drive being unable to generate a download URL which you could use with anything (be it Kickstart or whetever else) to download the file directly. Please understand that we are not Google and we do not control how Google Drive works. We can only give you practical alternatives which I already did.

Regarding what you see in the Manage remotely stored files popup, it contradicts your assertions both in your original ticket post and your response. What you describe is what you actually see when the file is already on your server, i.e. you can restore it. Since the file is already on the server we do not render the Fetch back to server button. I believe the reasoning behind this choice is self-explanatory. I want you to take a look at the backup record. Immediately to the left of the Manage remotely stored files button there is a green check under the Status column which means that the file is already on the server. The meaning of the Status is documented. You can also hover over it to see some more information about it. Alternatively, you can click the info icon to the bottom and right of the Manage remotely stored files button to be told in plain English whether your file is available on your server or not. This behavior of the Manage Backups page is also documented.

If the file exists only on Google Drive but not on your server you do see a button labeled Fetch back to server in the Manager remotely stored files popup. In fact, I just took a backup to Google Drive just so that I could take a screenshot to show you (ironically, since Google Drive would NOT let me create a publicly shared URL for the screenshot, let alone a direct download URL to use in the image tag, I had to upload it to Dropbox):

Clicking on the Fetch back to server button does fetch the file back to the server. The status of the backup record does turn green with a checkmark. In this case the Manage remotely stored files does no longer show the Fetch back to server button, as it should. In this case, as I explained, you can restore the backup by selecting the checkbox to its left and clicking on the Restore button at the top of the Manage Backups page.

Also note that the warning text you are referring to is on a section below a header whose title is "Download to your desktop". This is a completely different feature than the one we discussed you need. This feature lets you download a backup archive from the remote storage to your local computer (your desktop, i.e. the physical device where the browser you use to view the page runs on). However, Google Drive does not let you create this kind of URL, as I already explained, which is why you do not see any download buttons. Since this feature is not possible with Google Drive we render a warning telling you that. The message is very explicit about not being able to download to your PC. Nowhere does it say that you cannot download from Google Drive to your web server which is what we need to accomplish your desired task.

It should be perfectly clear now that both sending the backups to Google Drive and retrieving them from Google Drive to your web server do work. These are the Professional features involved in your original ticket. Therefore the argument that the Professional features do not work is not true. Moreover, they are unrelated to the actual issue you have, as you began to describe it towards the end of your reply.

The Transfer Site is a different feature used to transfer a backup archive to a remote server. I assume that this is what you actually needed help with, something which was objectively impossible to know by reading the first post in your ticket. In fact, the issue you described is completely unrelated to Google Drive.

The cURL error 67 referenced in the error message (per their documentation) means "The remote server denied curl to login". It also tells me that you used the "FTP over cURL" upload method. If you are unsure what this means, cURL is a programming library used to connect to remote servers. It's offered by PHP itself through the PHP cURL extension. This is one of the two ways PHP offers to connect to remote FTP servers, the other one being the native FTP features of PHP. We offer support for both. You chose the cURL one.

Now, the error states that the remote FTP server denied the login. And yes, the error is not very friendly but that's only because we relay error messages from cURL since it's proven impractical to intercept them in a meaningful way. That's why we offer support, to clarify these obscure messages and help you work past them. The error you got can mean different things: user error (you gave it the wrong credentials), server error (your web server mangles the username or password), a firewall on either server (web or FTP) or a configuration issue with the remote FTP server disallowing the FTP user from logging in. Since you can connect with FileZilla and can see the files I can readily discount the last possibility.

Clarification: I operate under the assumption that you are trying to restore your backup to a different server or site. Also, I assume that you have read the documentation and the information on your screen and dully understand that the target site must be empty, i.e. completely devoid of any Joomla, WordPress or other PHP application files. I also assume that you are aware that a database and a corresponding database user must have already been created. If something I just mentioned seems unclear to you please do ask me to clarify it and I will happily do so. I really do want to help you.

Back to our troubleshooting. First, double check the FTP server name, username and password. If they are correct, double check the FTP Directory. If that's correct to please contact your host and ask them if they block access to the FTP server from the IP address of your web server. Remember that the FTP connection happens at a much lower level than our software. Ultimately, if firewalls on either or both servers prevent them from talking to each other PHP is unable to transfer data from your web server to your FTP server. In this case our software which is written in PHP and can only use PHP's FTP features to connect to FTP servers won't be able to transfer data over FTP. If you want, we can arrange that we get access to your site and troubleshoot this issue ourselves so at the very least we can tell you what exactly is going wrong and which steps you can take to fix it.

I feel obliged to clarify something in the interest of clarity and preventing any further miscommunication. We do NOT promise that we can wave a magic wand and fully resolve the issue all by ourselves, even if we are given access to your servers. That's because your servers' configuration is not under our control and your issue seems to originate from the server setup (I will be more confident on the root cause once I have more concrete information). What we do promise is that we can perform troubleshooting on the server itself and tell you what we are seeing, i.e. why you are having this issue, as well as which steps you can reasonably follow to fix it. Some of these steps may require asking your host to do something that we tell you to do. This is not deflection, it's the objective inability to modify the server configuration of a server which is not under our direct control and we have not set up ourselves. This is exactly what we state in our Terms of Service and what is realistically possible when anyone is troubleshooting software running on servers not under their control.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

System Task
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!