Plugin index

latin1 to utf8

WARNING: carefully read this documentation and BACKUP YOUR DATABASE before activating/using this plugin.

Textpattern always uses ‘utf8’ to store data in the database, but depending on which MySQL version you used when installing Textpattern, the character set of the tables and columns differs:

  1. MySQL versions older than 4.1: the character set of the tables created by Textpattern is typically ‘latin1’.
  2. MySQL versions 4.1 and higher: the tables are set to use the ‘utf8’ character set, which matches the actual character set of the data stored in those tables.

Having the tables/columns themselves use the same character set as the data stored in them has some advantages when performing searches or sorting in Textpattern, especially if you use characters outside US-ASCII.

If you upgrade your MySQL version from pre-4.1 to 4.1 or higher, the table/column character sets are not automatically converted to ‘utf8’, so they stay ‘latin1’ while the data stored in those tables is still ‘utf8’.

Changing this is tricky, because if you simply change the character set, MySQL assumes that the stored data must also be converted from ‘latin1’ to ‘utf8’, which leads to garbage, because the data is already ‘utf8’.

This plugin avoids the unwanted character set conversion of the stored data by taking the following steps for each Textpattern table:

  1. drop indexes
  2. convert all columns that contain text to use a ‘binary’ charset (instead of ‘latin1’)
  3. convert those columns yet again, this time to ‘utf8’ charset (instead of ‘binary’)
  4. add indexes
  5. optimize table

Afterwards an attempt is made to update the ‘dbcharset’ setting in the textpattern/config.php file to ‘utf8’. Should this fail due to lack of permissions, you will be asked to make this change yourself. Do not skip this step, otherwise you will experience irreparable data corruption so severe that you will cry like a baby (you made backups, right?), much to the amusement of onlookers.

How to use this plugin:

  1. MAKE BACKUPS FIRST! (not just the TXP tables, but the entire database)
  2. Make sure you have MySQL version 4.1 or higher installed.
  3. Activate the plugin.
  4. Click here and follow the instructions on the screen.