After the failed upgrade attempt a fortnight ago I spent a lot of time looking in the wrong direction. The main problem was that users were unable to log in. So I thought it must have been session related. I tried many experiments with cookies, cache and sessions locally but couldn't workout what was going wrong. Everything seemed to be working correctly apart from the failed logins.
After many attempts at upgrading locally, dropping and recreating the database then running the upgrade script I narrowed the problem down to character set in the database, mainly the user table. The upgrade converted the data in the user table as well as many others to utf-8 from latin1. Due to some strange, well strange to me, characters used in username and signatures the conversion of the users table failed. luckily each time it failed it pointed to a row so I just had to find that row and look around that for the record with the strange character and fix it then rerun the database upgrade. The users table has about 35000 records and this was done locally while I recorded the record or user id to fix on the live site later.
The downside to this method is that some users now have a different username or signature. So I tried looking for other methods of converting between latin1 and utf-8 and although I found some suggested solutions nothing was looking like it was stable enough or successful enough to use.
So I kept running the upgrade query for users, fixing the errors and runing it again until all the errors were gone. It took a while to get to this stage and by then I needed a new database dump to make sure I wouldn't miss any new data. So I fixed my list of errors on the site, took a database dump and started again locally. Once all the errors were sorted out and I was sure people would still be able to logon, I was ready to attempt the upgrade.
The process I use to upgrade is firstly create a new directory at the same level as the public_html directory with all the new files and correct permissions. That way I don't need to be uploading while the site is offline and I can convert back easily if something goes wrong. It requires a fair amount of disk space. Once the files are in place I put the site offline, create a new copy of the database then point the settings to the new database and swap the names old public_html directory and the new one. Then run the upgrade script and configure as required.
Luckily for me everything seemed to go well this time. There was a few minor problems with file attachments that I forgot to copy over and smilies but everything else seems to be going fine.
More issues may arise over the next few days, we will have to wait and see.
Thanks for your patience and sorry for any inconvenience.