Quoture

Return contents of XML tag using Shell Script

I recently had the requirement to obtain a list of Order ID’s for a given barcode using the command line.

I’m sure there’s a better way to do this, but this gets the job done.

The text I’m collecting sits nested in tags like this ‘xxx‘. I also had an issue with the XML files all being on one line, so some tidy up was required before looking for the tags. This is where XMLLint is brilliant. You can use it if you have slightly malformed XML using the –recover argument or if it’s all on one line, it will indent it neatly for you using the –format argument.

So here’s the command:

grep -H 3000000028223 * -R | cut -d: -f1 | xargs xmllint --recover --format | sed -n 's|\(.*\)|\1|p'

When used, change the barcode and the tags used in the sed command to match your requirements.

Low CPU script to mysqldump and zip databases.

A script which neatly dumps all databases with low CPU priority so your production services won’t be affected. I’ve used

ionice

and

gzip -3c

to lower the priority of the compression process.

Just set your variables at the top and set it in cron to run at a maximum of once per hour. If you set it to more, it’ll overwrite the same backup name.

It will keep 1 weeks worth of backups, but because the our (0-23) is used in the filename, if you change the time of the backup, you’ll have to manually go and delete the old backup that corresponded to the hour you used to run it.

day=`date +%a`
hr=`date +%k`

#Set backup dir (no trailing slash needed)
backup_dir='/backup'
username='root'
password='yourpass'

#Make backup directory.
mkdir -p $backup_dir/$day

for db in `echo "show databases" | mysql -u root -p237harrogate |grep -v Database |grep -v information_schema`;
do
        # dump database
        ionice -c3 nice -n7 mysqldump --add-drop-table --allow-keywords --compress  --quick --single-transaction -u $username -p$password $db | gzip -3c > $backup_dir/$day/$db\_$hr.sql.gz
done

Simple Shell Script to Clear Cache on Multiple Servers

I work with Magento, and it does a good job of caching but after an update, I need to be able to quickly and easily clear cache on all three of our web servers.

Here’s a simple script which takes care of that. All you need to do is configure your servers and the cache dir.

Be careful when using this, it is destructive, so double check all of your paths first.

#!/bin/sh

#Declare servers
servers=( "xxx.xxx.xxx.xxx" "xxx.xxx.xxx.xxx" "xxx.xxx.xxx.xxx" )

#Set cache directory
cache_dir="/var/www/var/cache/"

#Loop through each item in the array
for ip in ${servers[@]}
do
        ssh root@$ip "rm -rf $cache_dir/*"
done

Installing RLIB on Ubuntu

It took me a while to figure out which packages I had to install before I could configure rlib to work with PHP, MySQL and to generate HTML reports.

To use the PHP queries, which in turn access MySQL, you need to install the following packages on Ubuntu:

php5-dev

and

libmysql++-dev

If you also need to be able to produce HTML reports, rather than PDF’s, you’ll need to install gd. The dev package which you need is:

libgd2-noxmp-dev

When you run configure, you should see

MYSQL: yes

PHP: yes

and

HTML: yes

Then it’s just a case of running make, then make install.

Find & Replace accross multiple files & subdirectories

This is a nice and easy way to do a find and replace accross multiple files in Linux.

First change to the directory where the files are located, then run this command to list all of the files this command will affect.

find . -name "*.xml"

Obviously, you should change the string between the quotes to match the files you want to work with. Press enter and make sure only the files you want to change are listed.

Next, we actually do the work with this command.

find . -name "*.xml" | xargs sed -i 's/4.95<\/ShippingCost>/5.00<\/ShippingCost>/g;'

This is actually fairly simple. We send the list of files generated by the find command to xargs. This runs sed -i ‘s/search_pattern/replace_pattern/g;’ for each file.

Escaping
Note: If you’re working with tags, the closing tags will have a forward slash. This needs to be escaped, as I’ve done above with a backslash before it.

WordPress Auto Update asking for Connection Information

This is a pretty easy one. If you’re getting this annoying page instead of a nicely updated WordPress, it’s because your webserver (apache) can’t write to the filesystem.

I spent about half an hour trying to figure this out. I changed everything inside the directory to the same user my webserver runs as. In my case, it’s www-data – Still no dice!

The key was to cd .. to the parent directory and change the ownership of the actual blog directory as well.

Hope this saves you some time.

Google apps running on a domain and subdomain simultaneously

I recently had to add a second google apps installation to a subdomain of a domain which was already running Google Apps. I use Godaddy, and spent hours trying to “add a subdomain” using instructions from their help center. I find Godaddy’s help is constantly out of date because they upgrade their systems so much.

Either way, the records that you need to add would be the same regardless of which DNS provider you use.

Our pre-existing installation looked like this:

Type
Host
Points To


A
@
10.11.12.13


CNAME  
mail  
ghs.google.com


MX
@
ASPMX.L.GOOGLE.COM


MX
@
ALT1.ASPMX.L.GOOGLE.COM


MX
@
ALT2.ASPMX.L.GOOGLE.COM


MX
@
ASPMX2.GOOGLEMAIL.COM


MX
@
ASPMX3.GOOGLEMAIL.COM

Adding the subdomain was just a case of adding the records listed in bold:

Type
Host
Points To


A
@
10.11.12.13


A
mysubdomain
10.11.12.13


CNAME  
mail  
ghs.google.com


CNAME  
mail.mysubdomain  
ghs.google.com


MX
@
ASPMX.L.GOOGLE.COM


MX
@
ALT1.ASPMX.L.GOOGLE.COM


MX
@
ALT2.ASPMX.L.GOOGLE.COM


MX
@
ASPMX2.GOOGLEMAIL.COM


MX
@
ASPMX3.GOOGLEMAIL.COM


MX
mysubdomain
ASPMX.L.GOOGLE.COM


MX
mysubdomain
ALT1.ASPMX.L.GOOGLE.COM


MX
mysubdomain
ALT2.ASPMX.L.GOOGLE.COM


MX
mysubdomain
ASPMX2.GOOGLEMAIL.COM


MX
mysubdomain
ASPMX3.GOOGLEMAIL.COM

Easy OSX SSH Connection to your Linux box using Public Key and SSH Config

It took me a while to get my head around PPK authentication, then when I did, even though I didn’t have to type the password anymore, I got tired of typing this into my mac console every time I want to connect:

ssh [email protected]

Far too long!!

This isn’t the most secure way to implement this, but that’s not the purpose of this tutorial. You can harden it as you wish, but here’s a quick and easy way to make it so you can just type this to connect to your server:

ssh svr1

In this tutorial, I’ll refer to the computer you’re connecting from as the ‘client’ and the computer you’re connecting to as the ‘server’. It also assumed you have the SSH package installed and running on the server.

1. On the client machine, cd to your home directory and make sure .ssh exists. If it doesn’t, type: 

   mkdir ~/.ssh

2. Generate a public and private key by typing:

   cd ~/.ssh
   ssh-keygen

   Just hit enter until it’s finished. This will generate two files in the .ssh folder. id_rsa, which is your private key, and id_rsa.pub, which is your public key.

3. On the server, log into the server and again, make sure the .ssh directory exists for your user. If it doesn’t exist, do the following:

   mkdir /home/beren/.ssh

   * obviously, substitude ‘beren’ with your user home directory on the server.

4. Create or open (or both):

    /home/beren/.ssh/authorized_keys

   on the server and paste your PUBLIC key (id_rsa.pub) that you created earlier, into here.

5. Test connectivity from your client to the server. You should now be able to connect without typing a password:

   Beren's-Mac-Pro:~ berengamble$ ssh [email protected] -p 2020
   Last login: Fri Mar 11 10:42:03 2011 from svr1.adsl.isp.com

   See? No prompt for password! We’re getting there!

6. To make it truly quick and easy, you can add the above details into ~/.ssh/config on the client to create a shortcut, if you will. My config file looks like this:

   Host svr1
   Hostname svr1.mydomain.com
   Port 2020 // - Ignore this if you are using the standard port.
   User beren

   Now you can connect simply by typing:

   ssh svr1

Find the first and last day of past month

Say it’s February 2011 and you want to perform calculations on the entire month of February 2010. You can use this query to do do it.

SELECT SUM(total)
FROM invoice
WHERE date_time BETWEEN DATE_FORMAT( NOW() - INTERVAL 1 YEAR, "%Y-%m-01" )
AND LAST_DAY( NOW() - INTERVAL 1 YEAR )

Adjusting PHP.INI values for each Virtual Host

If you’re like me and you have your dev and live environment running on the same server under a single instance of apache, you’ve probably got..

display_errors = No

..set in your php.ini, as you don’t ever want users to see ugly PHP errors, should they occur.

If you’re running PHP as FCGI, you can specify entirely seperate php.ini files for each Virtual Host by using this line in your config or .htaccess by simply by adding this directive:

SetInv PHPRC /directory/containing/your_php.ini

But if you’re running Mod_PHP, like me, the php.ini only gets read once, when you start apache. So a different tac is required. This involves setting php.ini options directly in your Virtual Host config file using this syntax:

php_value display_errors 1

Note, that you have to use integer values as “yes” or “no” won’t work like it does in php.ini

Now, here’s the part I wasted hours on; wondering why display_errors had no effect: Make sure you increase the verbosity of your logging. This means, if you have disabled some of the error_reporting options in php.ini like this:

error_reporting = E_ALL & ~E_DEPRECATED & ~E_NOTICE

Deprecated and Notice will remain suppressed unless you enable them in your Virtual Host config file.

Showing all errors is easy. In your Virtual Host config file, add this line under :

php_value = error_reporting -1

If you want to customise what is reported, you need to refer to the table on this page.

What you need to do is add all the values in the left column of the errors you want to report on.
eg:

1 (E_ERROR) +
2 (E_WARNING) +
4 (E_PARSE) +
8 (E_NOTICE)
= 15
Your line in your Virtual Host config file would look like this.

php_value error_reporting 15