Month: October 2015

MySQL won’t reboot

After the 100% full hard disk problem I couldn’t get MySQL to restart. I looked at the error logs and found these lines.


151010 16:51:18 [ERROR] /usr/sbin/mysqld: Table './SS_connellsville/alerts' is marked as crashed and should be repaired
151010 16:51:18 [Warning] Checking table:   './SS_connellsville/alerts'
151010 16:51:56 [ERROR] /usr/sbin/mysqld: Table './SS_greensburg/values_RSSI_3' is marked as crashed and should be repaired
151010 16:51:56 [Warning] Checking table:   './SS_greensburg/values_RSSI_3'
151010 16:51:58 [ERROR] /usr/sbin/mysqld: Table './master/node_groups' is marked as crashed and should be repaired
151010 16:51:58 [Warning] Checking table:   './master/node_groups'

A stackoverflow post recommended using mysqlcheck. It must be used when the mysqld server is running. I couldn’t see how to set it to just repair the databases that were causing problems, but I figured that it wouldn’t hurt to run it on all of them. I was a bit concerned that it would run into problems when the cron jobs ran, but I set it running anyway. It changed the record count on all of the databases and ran for many hours. At least six, but I stopped monitoring and let it run overnight. I stopped MySQL and did a restart and it started without errors.


mysqlcheck --repair --use-frm --all-databases

Disk space

I was asked to take a look at a server that was no longer responding. The first thing I did after going to the co-lo and booting the console in single user repair mode was look at disk usage.


$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             70557052  70557052         0 100% /
none                    508792       224    508568   1% /dev
none                    513040         0    513040   0% /dev/shm
none                    513040      1076    511964   1% /var/run
none                    513040         0    513040   0% /var/lock
none                    513040         0    513040   0% /lib/init/rw
/dev/sdb1             70557052  46954944  20018012  71% /media/Backup

As you can see the server usage went to 100%. Normally it is near the size of the hot backup drive. So some file suddenly got large. When this happens, none of the services will respond. So you can’t ssh in and the websites stop responding.

Then I started looking for files that could be causing the problem. I was told that the last time this happened the log files weren’t being erased. I check on this server from time to time and this week it was at 82%, so that didn’t seem likely but I ran this command anyway.


sudo find /var/log/ -size +20M | xargs du | sort -n -r | less

Nothing really jumped out at me but there were lots of old logs that had been zipped, and some that were labelled ‘old’, so I ran this to get rid of them

rm *.gz
rm *.old

It made a little improvement, but not enough to get the df command to go below 100%. It did go to 68852672 which was enough to get ssh into the server after a reboot. I kept looking for the source of the large files but had no luck. We have a call in to the guy who fixed it last time hoping he remembers what he did.

Critical thinking – a response to a misguided forum post.

Remember the part I said about ‘woo magnetism’ and how it is dangerous? The reason it is important to challenge silly beliefs like dowsing, global warming denial, chemtrails, etc. is that they lead to fuzzy thinking. No one is harmed by dowsers—unless you count the money spent on them. But that lack of critical thinking does lead to beliefs that are harmful.

One of the very first principals of critical thinking is that anecdotes are not evidence. Seeing someone find water using dowsing does not mean it works. If you understood the science behind it you would know that it cannot work. That’s one of the red flags for any ‘amazing breakthrough’ touted in junk email and websites flogging woo. Scientists (and people who use critical thinking) know that people are easily fooled. Confirmation bias, wishful thinking, confounding factors, and luck all influence how you interpret the world. Scientists try to remove the extraneous factors. Woomeisters gloss over them—if they acknowledge them at all.

The lack of critical thinking can have disastrous consequences when it spills over into real life issues. Your wife’s cancer is a case in point. Herbal remedies will not help her cancer at all. That is a fact. What confuses people like you—who don’t understand statistics and science—is that sometimes surgery is enough to stop the cancer. Sometimes cancers go into remission themselves. So an anecdote on a website that is selling some kind of miracle cure ‘that doctor’s don’t want you to know about’ is worthless. I hope your wife overcomes her cancer, but her chances would be much greater if she went to real doctors.

And the science of global warning is not a left-right issue. For reasons that escape me, the people in Foxlimbeckistan have made it into a political issue. But the science is not in any way political.

I’m fairly careful to moderate what I say and I don’t intend to say anything mean about you personally—although it’s possible that I did. But I don’t have much patience with willful ignorance, so you may have interpreted my stridently attacking your ideas and lack of critical thinking as attacking your character or integrity.

http://blogs.scientificamerican.com/rosetta-stones/2015/01/12/wallaces-woeful-wager-how-a-founder-of-modern-biology-got-suckered-by-flat-earthers/

Grep Tricks

Often I need to mess with the content of arrays. They often look like this.

“99”, ”

And I’ll want to delete the contents of one section.


^"([0-9]*)"(,"")(,"[a-zA-Z0-9 ,-/'&#ñéá]*")
"\1"\3\2

This says, start at the beginning of each line ^, look for double quotes, then a number—repeated as many times as you want, then another double quote,

Look for any letter, number, and a bunch of special characters that don’t include double quotes “.

([a-zA-Z0-9 ,-/’&#ñéá]*)