To all of my regular readers, apologies if this post seems a bit out of the ordinary, but if you read on, you’ll see why I’m documenting this here, as opposed to in my ticketing system.
On Friday afternoon, for no apparent reason, while I was the only user on my RT 3.8.0 system, after clicking a link, I was presented with the logon prompt.
Puzzled, I entered my username and password and logged back in. Everything seemed normal, and the ticket I was trying to edit was there. I continued my edits, and clicked to update the ticket.
Login prompt again.
I tried using a different browser (trusty Firefox instead of Opera, which I’ve been experimenting with lately). Same thing. Login prompt after every click.
Some research in the RT Mailing List finally led me to uncomment the following line in my RT_SiteConfig.pm file:
Set($WebSessionClass, ‘Apache::Session::File’);
Doing this, and restarting Apache fixed the problem. Or so it seemed…
Last night, while reviewing the server’s backups, I realized that the nightly MySQL dump from the 22nd was much smaller than previous nights’ backups. Curious, I re-ran the backup script. It immediately failed, saying that rt3.sessions was corrupt.
“Session? That sounds familiar…” More Googling. I found several references to the same sort of problems. One fix I found was to run myisamchk on the data file in question which, in this case, was /var/lib/mysql/rt3/sessions.MYI. I did this, and it identified errors. I proceeded to run myisamchck -r -q sessions in the data directory. It threw a lot of data on the screen, and appeared to be repairing lots of stuff.
So this explains the sudden need to add this line to my RT_SiteConfig.pm where it was never needed previously – database corruption. One mystery solved.
I figured if the sessions table was corrupt, maybe running a quick repair on the rest of the tables would be a good idea. I wrote up a quick bash script to loop through all of the table files in the directory and myisamchck -r -q all of them.
Here’s the puzzling part where my memory gets a little fuzzy. I am fairly certain that RT was working fine after this, and I went to bed. This morning, I attempted to log in, did so, but then saw no tickets! Clicking my saved bookmarks to take me directly to my preferred view of a queue showed 0 tickets. Similarly, requesting all “new” and “open” tickets in my “support” queue from the command line returned 0 results.
However, if I fire up mysql, I can select data from the tables just fine.
I pulled a copy of last night’s mysqldump, and attempted to open it in Notepad++, which I was using to take notes (since I couldn’t very well take notes into RT, could I?). Bad idea. Notepad++ crashed, taking all of my notes (essentially, everything you’ve read up to this point) down with it.
So it seems I have a couple of options: attempt to roll back to the latest known good MySQL dump, which may be 3-4 days old at this point, or figure out what happened to the files (indexes?) and repair them.
Any suggestions greatly appreciated.