Support

Admin Tools

#31031 error BIGINT UNSIGNED value is out of range in '(`dbusername`.`#__menu`.`level` - 1)'

Posted in ‘Admin Tools for Joomla! 4 & 5’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
n/a
PHP version
n/a
Admin Tools version
n/a

Latest post by on Saturday, 06 April 2019 17:17 CDT

webdevtim
Database Version—5.5.61-cll-lve
Database Collation—utf8mb4_unicode_ci
Database Connection Collation—utf8mb4_unicode_ci

i am getting error BIGINT UNSIGNED value is out of range in '(`dbname`.`#__menu`.`level` - 1)' when attempting to install a dev version of K2 2.9.1 over the existing K2 installation. I had no problem re-installing Joomla 3.9.3 over the existing installation – no error. I did this because one category of pages is not displaying, and Fotis Evangelou suggested that this might solve that problem. However getting the BIGINT UNSIGNED value is out of range error, pointed to more fundamental problems.

I attempted to change the database collation to utf8mb4_unicode_ci using the Admin Tools "Change Database Collation" but as you can see it remains utf8mbr_general_ci. There is a mix of utf8_general_ci, utf8_unicode_ci, utf8mb4_unicode_ci and utf8mb4_unicode_ci tables and there is a mix of INNODB and MyiSAM database engines for the various tables.

I also ran Optimize Database again, also with no results, still can't insatall K2 2.9.2-dev.

Fotis Evangelou suggested that I create a new Joomla/K2 installation and migrate everything over to the new installation, a month's worth of work, because the existing Joomla installation is really screwed up. Before I do that I wanted to get your appraisal of what my options are at this point.

Below are the arguments that prompted me to want to change to utfsmb4_unicode_ci.

Note: Since MySQL 5.5.3 you should use utf8mb4 rather than utf8. They both refer to the UTF-8 encoding, but the older utf8 had a MySQL-specific limitation preventing use of characters numbered above 0xFFFD.



Accuracy



utf8mb4_unicode_ci is based on the Unicode standard for sorting and comparison, which sorts accurately in a very wide range of languages.



utf8mb4_unicode_ci fails to implement all of the Unicode sorting rules, which will result in undesirable sorting in some situations, such as when using particular languages or characters.



Performance



utf8mb4_unicode_ci is faster at comparisons and sorting, because it takes a bunch of performance-related shortcuts.



On modern servers, this performance boost will be all but negligible. It was devised in a time when servers had a tiny fraction of the CPU performance of today's computers.



utf8mb4_unicode_ci, which uses the Unicode rules for sorting and comparison, employs a fairly complex algorithm for correct sorting in a wide range of languages and when using a wide range of special characters. These rules need to take into account language-specific conventions; not everybody sorts their characters in what we would call 'alphabetical order'.

nicholas
Akeeba Staff
Manager
First of all, it's normal to have a mix of collations when you try to convert a site. Some older software may have not updated its database structure for UTF8MB4 requirements. For instance, MySQL only allows database keys with entries up to 767 bytes long. If you have a VARCHAR(192) column it takes up 3 x 192 = 576 bytes under UTF8 but 4 x 192 = 768 bytes under UTF8MB4. This makes it an invalid key size. If there's an INDEX / KEY on that column the conversion to UTF8MB4 will fail, therefore it needs to remain plain old UTF8.

That said, this has absolutely nothing to do with your problem! If you look at the error message, it tells you that trying to write to the Joomla menu table failed because something tried to store the value -1 on a column which is supposed to only accept unsigned long integers. "Unsigned" means "without a sign", i.e. only positive values. Clearly, -1 is negative, not positive, hence it's out of range.

The level column where this happens is what is used internally by Joomla! to figure out the hierarchy of menu items. Therefore the error message tells us that your Joomla! menu structure is, not to put too fine a point on it, cocked up. The first thing you should try is going to the backend of your site, Menus, Manage and click the Rebuild button in the toolbar. It tries to fix exactly this kind of issues.

If that didn't make any difference then you should probably rebuild your site, as painful as it may be, because that'd mean you have a far more complicated issue.

PS: You may wonder why this happened when you tried to install an update. It's probably because Joomla is trying to regenerate the backend Components menu during the update of any component. Since the menus table, shared by the front- and backend, is screwed up this triggered an invalid data write which raised the database error.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

webdevtim
Thank you very much for this detailed explanation and heads up. I suppose that situation will not be going away anytime soon, because a lot of extensions developers are loathe to re-write their code to take advantage of new database standards. You have helped me understand some things about collations.

Question, why does MySQL only allow database keys up to 767 bytes, when modern Unicode characters require 4 bytes for the entire character set to be represented. Also since searches under utf8mb4_unicode_ci are much more accurate than under utf8_general_ci or even utf8mb4_unicode_ci, why hasn't MySQL adopted utf8mb4_unicode_ci as it's standard?

About the main issue here

I tried rebuilding the menus, but it made no difference, I tried rebuilding each menu individually, but it still made no difference

Then I thought what have I got to loose, and I re-installed the production version of K2 2.9.0, and it installed without throwing an exception.

I then tried to re-install the dev version of K2 2.9.1 from GitHub and it also installed without throwing that exception this time.

So I guess the issue arose out of installing various builds of K2 2.9.1, which Fotis frantically threw together because Joomla 3.9.3 broke some of his code, and in that process, the K2 related menus got screwed up.

Everything works now without any exception being thrown, much to my relief.

nicholas
Akeeba Staff
Manager
Developers want to rewrite their code to take advantage of improvements in PHP, MySQL and Joomla! itself. The problem is that users and hosts don't want to upgrade their PHP and MySQL versions. Case in point: I am still supporting PHP 5.4, a version of PHP which stopped receiving even bug fixes on September 3rd, 2015. Its usage only dropped below the point where I can safely discontinue supporting it two months ago. If you ask me as a developer I only want to support PHP 7.2 and 7.3, MySQL 5.7 and Joomla! 3.9 at the time of this writing.

Regarding indices, it's mostly the fault of developers and bad conventions. Most of us used a VARCHAR(255) for title fields. That seemingly random number comes from the MySQL index size and the fact that MySQL would only support 3-byte UTF8 until quite recently. When we wanted to create an index we would create an index on the field without any qualifiers. This was stupid because that slows down MySQL when searching in most cases. What we should do is put a qualifier on the maximum length of the index, e.g. tell it to create an index on some_field(30) which tells MySQL "index only the first 30 characters". On most sites this only puts a dozen or two of rows into each index position. This means a much more compact index (MySQL can read it faster) for a tradeoff of processing a dozen or so rows instead of just one. When you do the math, it's faster. I did NOT know any of that because I am developer, not a database administrator. I watched a presentation a few years ago by a database admin on the subject. But can I apply it on my software? It's complicated and error-prone. I am still trying to find a good solution to a past mistake...

I am glad you managed to fix your site :) I am always of the mind that fixing should first be attempted before applying the nuclear option. I've seen too many "developers" offer the nuclear option for trivial issues which ultimately drove clients away from the CMS, making them think that if anything slightly breaks their site is hosed. I hate that mentality. I hate these "developers". Joomla is a great CMS with a lot of options to help you fix most cock-ups imagineable. All a developer has to do to help their clients is actually know how the CMS works and, most importantly, how to use the CMS.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

System Task
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!