RHEV and RHEL Clustering - Fencing without RHEVM - The script

Following up on the blog post from a few weeks ago (here) I finally got around to creating a fence script that allows for fencing a VM without an available RHEV manager.


I've placed the script in my GitHub HERE, you need to copy it to /usr/sbin and give it execution permission.


How does it work?

I replicated the fence_virtsh script and changed the code to add the necessary commands.
The list of hosts where the script checks for the VM is passed in the "ipaddr" field.

Since this is a custom fencing script you cannot configure it directly in Luci, you need to edit the cluster.conf file manually.

The fence device can be added to a node like this:

<clusternode name="server1" nodeid="1">
    <fence>
        <method name="RHEV-NOMGT">
            <device name="rhev-nomgt" port="Linux-Serv1"/>
        </method>
    </fence>
</clusternode>

The fence device itself is define like this:

<fencedevice agent="fence_rhev_nomgt" ipaddr="192.168.1.1,192.168.1.2" login="root" name="rhev-nomgt" passwd="password"/>

In the clusternode fence method definition, the "port" is the name of VM in the RHEV system.

The "ipaddr" parameter, in the fence device, is a list of the hostnames (or ip addresses) of the hypervisors where the VM can run separated by a comma. The login and password refer to the login and password of the root user on the hypervisors. I know, this is not very safe but the hypervisors don't allow the creation of other users and for my scenario this won't be an issue.

Example cluster.conf for a two-node cluster:


<?xml version="1.0"?>
<cluster config_version="1" name="RHEVCLUS">
        <clusternodes>
                <clusternode name="server1" nodeid="1">
                        <fence>
                                <method name="RHEV">
                                        <device name="rhev-nomgt" port="Linux-Serv1"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="server2" nodeid="2">
                        <fence>
                                <method name="RHEV">
                                        <device name="rhev-nomgt" port="Linux-Serv2"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <cman expected_votes="1" two_node="1"/>
        <fencedevices>
                <fencedevice agent="fence_rhev_nomgt" ipaddr="172.18.56.251,172.18.56.252" login="root" name="rhev-nomgt" passwd="password"/>
              
        </fencedevices>
</cluster>


PLEASE TAKE NOTE:
This fencing method is intended as a fail-safe option when there is no other possible fencing option available and to ensure the cluster doesn't halt in specific situations where there RHEV Manager isn't available. This should no be used as a primary fencing method! And please note that it can cause issues in databases such as loss of data and file corruption. To use this method you should be fully aware of the risks.

RHEV and RHEL Clustering - Fencing without RHEVM

UPDATE: I've added the script see the post: here.


For an upcoming project I'll be using a Red Hat Cluster inside a RHEV environment.
At first glance I didn't see any problems since RHEL's High Availability add-on already includes a fencing script for the RHEV-M.
But what happens when the RHEV-M is down or unresponsive and the cluster need to fence one of the nodes?

This could mean trouble since the cluster would stop every service it manages resulting in a potential downtime for our applications.

After some research I've come up with a possible solution that allows for the fencing of a VM without a RHEV-M.

The process is quite simple but needs a few steps:

1 - Get a list of all the hypervisors inside your RHEV system where the VM can run

2 - For each of these hosts do the following operations until we find the VM:

• Connect to the host as root
• Check if there is a QEMU process for our VM in our current host. If there is proceed with the following commands, if not then try the next hypervisor.
• Create a new set of credentials to interact with libvirt: 
saslpasswd2 -p -a libvirt fenceagent

(fenceagent is a username and this command will ask for a password)

• Restart the VM with the following command:
virsh  qemu-monitor-command --hmp VM_NAME system_reset

(Change VM_NAME for the name of the VM as it appears on RHEV)

• Remove the user you created.
• Log off from the hypervisor

In a few days I'll transform this into a Python script so I can add it to the Cluster. 

I've already validated this process manually so I think there will be no major issues with it.

But there is a potential issue, since this requires an iteration over all the hypervisors (or at least until you find the VM) it can that a lot of time if there are lots of hypervisors, but at least your cluster won't go berserk :D

This will also need some extra configuration like a list of Hypervisors where the VM can be run and the VM name also needs to be passed as an argument to the fence script

For future reference, I based this "algorithm" in the following information:

http://comments.gmane.org/gmane.comp.emulators.ovirt.user/6551

http://blog.vmsplice.net/2011/03/how-to-access-qemu-monitor-through.html

More updates on this to follow.

Remembering Red Hat Linux 5.2 (from 1998) - Part I (Installation)

My first contact with Linux was when I was finishing High-School in 1999. A friend from class gave a copy of Red Hat Linux 5.2 and a small 20 pages manual.
I remember getting home firing up my "powerful" AMD K6 machine and getting immediately frustrated: I didn't have enough disk space for a dual boot...

No problem, since I had recently installed my Windows 98 I decided to backup any important files to floppy disks and format the hard drive.

Before starting the installation process I opened the PC case and took note of most of my Hardware chips since my friend warned me this would be required in the setup process.

And so I happily proceeded with the installation. Everything was going great until I had to choose my video card... a Riva TNT2 64 which, at the time, wasn't supported. This ruined the experience for me since I was stuck with a 640x480 8 bit display. Since it was my first time running Linux I didn't know enough command-line to get started so I had to accept defeat and re-installed Windows 98 :(

Although it was a bad first experience, at the time I was very intrigued and excited about the concept of a free operating system and that "regular" people we're joining up to create something completely open-source and that anyone could join in and contribute.
I had no idea that this "community" project would be such a big part of my life :-)

Anyway, for the sake of nostalgia and to experience the O.S. I never had the change to try, here is a guide to running Red Hat Linux 5.2 on QEMU.


1 - Download a Red Hat Linux 5.2 iso image from the Internet Archive:
https://archive.org/details/redhat-5.2_release

2 - Download QEMU for your linux distro (use your package management tool) or Windows from here: http://qemu.weilnetz.de
I used version (2.1.0-rc2)

3 - Extract the contents of the installation package of QEMU or run the installer.

4 - Create a disk image to use with your Red Hat 5.2 Virtual Machine, I'll be creating a disk image with 1GB since this was more than enough at the time (my first disk had about 850 MB)

qemu-img create -f qcow2 redhat52.img 1G

5 - Run qemu with a special vga card (cirrus) that is compatible with these old linux distributions, also use 256M of RAM and attach the cdrom image to the VM:

qemu-system-i386 -hda "<path to disk image>\redhat52.img" -cdrom "<path to iso>\redhat-5.2-i386.iso"  -m 256 -vga cirrus

The -vga cirrus parameter is essential to get a desktop later on.

6 - Begin the installation! I made a small video (~6min) with the installation process. I tried not to complicate things too much and chose a Workstation default configuration. Video below:

On the next post I'll be going over the desktop and software that came with this version.

Moodbar and playlist creation - Part I

I like music! I listen to music everyday since it helps relax,  get inspired and even lifts me up when I have the blues.

Unfortunately I can't access online music services during the day (damn proxies...) so I have to rely on my own music collection for all my needs.

You can imagine that listening to the same tracks day in and day out can get a bit tiresome...

I always tried to use players that had "smart" playlist generation in order to create some nice coherent playlists. However these "smart" playlist creating apps only work well to find similar artist/bands to the one you're listening to or to suggest popular tracks from other similar artists/bands.

That's great when you have a huge music collection with lots of artists/bands and albums or for web services that have artists and bands you never heard of...

Besides, different tracks from the same artist/bands don't always have the same "mood".
I'll illustrate this with an example from my favourite band: Pearl Jam

Listen to the following songs:

Do the Evolution: http://www.youtube.com/watch?v=aDaOgu2CQtI
and
Sirens: http://www.youtube.com/watch?v=qQXP6TDtW0w

They are in completely different contexts and can easily have been from two distinct bands.
This is something that usually happens within a band, songs that translate into completely different states of mind and that can really mess up a playlist and your mood :-)

So how can you make this less of a problem?

A few years ago I was introduced to a great music player for the KDE desktop: Amarok (amarok.kde.org). Amarok had a nice feature that almost went unnoticed but had great potential, the moodbar.

What is a moodbar you might ask?
Read this: http://blog.randomprocesses.net/2007/02/moodbar.html
and this: http://cratoo.de/amarok/ismir-crc.pdf
and maybe this as well: http://userbase.kde.org/Amarok/Manual/Various/Moodbar

Making a quick summary, a moodbar is a graphical representation of the different frequencies in a song as they occur over time.

It looks something like this for the songs I mentioned earlier:

Moodbar for "Sirens"

Moodbar for "Do the Evolution"

You can easily identified when something happens in the song since the vertical bars are ordered with the song's duration. You can also see that the two moodbars are very different in overall aspect/hue, just like the songs.

So this got me wondering if we could use this to select similar songs....


Turns out you actually can... :D

I used a simple Python script to read moodbar files generated by my current music player, Clementime (http://www.clementine-player.org/) which is based on Amarok, and select songs that have a similar "mood" to a given song.

I started by creating a playlist of Bands/Artists I wanted to listen to and then run the script to process the files in that list and created a new list of songs that had a similar mood to my selected song.


This is what I came up with:

Playlist based on "Sirens" (first 20 songs):

Pearl Jam - Sirens

Rage Against The Machine - Year Of Tha Boomerang

The Black Keys - I Got Mine

Pearl Jam - Supersonic

Nirvana - Spank Thru

Pearl Jam - Swallowed Whole

The Black Keys - The Breaks

The White Stripes - Screwdriver

Rage Against The Machine - Down Rodeo

Ramones - Rock 'N' Roll High School

The White Stripes - Black Math

The White Stripes - Instinct Blues

Nirvana - Come As You Are

Pearl Jam - Mind Your Manners

Pearl Jam - Even Flow

Ramones - Sheena Is A Punk Rocker

Pearl Jam - Why Go

Nirvana - Beeswax

Rage Against The Machine - Down Rodeo

The Black Keys - Nova Baby

And the combined moodbar look like this:

Playlist based on "Do the Evolution" (first 20 songs):

Pearl Jam - Do The Evolution

The White Stripes - Now Mary

The White Stripes - Jack The Ripper

Nirvana - Rape Me

The White Stripes - Hotel Yorba

The Black Keys - Stack Shot Billy

The Black Keys - Keep Me

Pearl Jam - State of Love and Trust

Led Zeppelin - Dazed And Confused

Pearl Jam - Corduroy

Temple Of The Dog - Pushin Forward Back

Nirvana - Moist Vagina

Pearl Jam - Arc

Pearl Jam - Help Help

Rage Against The Machine - Microphone Fiend

Rage Against The Machine - Sleep Now In The Fire

The White Stripes - I Think I Smell A Rat

Rage Against The Machine - Microphone Fiend

Nirvana - Radio Friendly Unit Shifter

The Black Keys - Just Got To Be

Moodbar for the playlist:

Not bad, the process can be refined a bit but you can immediately see that the overall hue of the playlists is different and so are the songs but they are similar in overall sound.



I'll post the Python screen in another post since I still have to clean it up and optimize some stuff.

First adventures in Python

One of my goals for the next couple of months is to learn Python. I guess this is long overdue and I should probably have picked it up a few years ago... Oh well, never too late to start...

Since I have an issue with following basic tutorials and writing code that has no practical use, I decided to create a small app to monitor databases.

Nothing to fancy, just a listing  a couple of running SQL queries, table locks and user sessions.
The nice part is that it can  do this for three different database systems: Oracle, MySQL and PostgreSQL (this one is not yet implement).

I placed all of the code on github (HERE) so you can check it out by yourself.


How does it work?

Well I created a simple interface using ncurses the is divided into 4 areas: Header, Running Queries, Current DB Locks, Current Sessions.

The interface is the same for all database systems and the information to display is the same (within each DB's possibilities).

Here are some screenshots:

MySQL Database:

 Oracle database:

It currently is in a very alpha stage, and there are some things I need to polish. For instance the terminal it runs on needs to have at least 51 lines (which is easily changed in the code), there are few exception handling, etc...
But there also some nice things: long running queries (over 5 minutes) should appear in RED,  the query text is divided into two lines so you can more easily identify bad queries.


The information is constantly refreshing (currently once per second) and it should have a low impact on your database since the queries it runs are on internal database tables/views.

I plan on adding some more features like system/db load, traffic, query cache hit, key efficiency, etc....

So stay tuned for more updates...

Linux script to monitor an Apache Tomcat WebApp

Sometimes the world is against us and stuff goes wrong.
This can happen a lot with Tomcat and bad behavioured WebApps.

To mitigate crashes and unresponsive WebApps sometimes it is useful to check if they are working or that they haven't crashed Tomcat... (not my fault! :-))

Since I was running a Tomcat server on a Cluster system I needed a script that could check the status of a particular webapp and if tomcat was also alive.

I came up with THIS, a simple script that starts/stops a WebApp, checks if it is running and even checks if Tomcat hasn't died...


So how does it work?

First we need to setup some variables:

The first one tells the script where the webapp is running, this can be change to include a specific IP Address (Virtual address used in clusters for instance)
TOMCAT_SERVER=`hostname`

Next is the script that controls tomcat (if you aren't using one you can change the code to use the built in startup and shutdown commands)
TOMCAT_SCRIPT=/etc/init.d/tomcat

This is the port where Tomcat is running, usually its 8080
TOMCAT_PORT=8080

This port is used to check if Tomcat is running.
Why use the Shutdown port instead of the normal port? Because I had an issue where the normal port was open but there was no shutdown port and Tomcat was not running properly...
TOMCAT_SHUTDOWN_PORT=8005

We also need the credentials for a user with permission to start/stop webapps.
TOMCAT_USER=user
TOMCAT_PASS=pass

Next we need to define the WebApp name to be controlled. There are two ways to do this, either by the script, assuming we use something like: tomcat_MyWebApp or tomcat-MyWebApp.
Or just set it manually.

#The Line below is used to extract the Webapp name from the script
#the webapps name is assumed to the last sequence of alphanumeric
#chars in the scriptname (for instance: tomcat_MyWebApp -> WA name=MyWebApp)

#TOMCAT_WA_NAME=`echo $0|egrep -o "[a-zA-Z0-9]*$"`

#Or you can manually set the WS name here 
TOMCAT_WA_NAME=MyWebApp


Finally we need to define the installation dir for Tomcat, this should be the same as the CATALINA_DIR in Tomcat's configuration. This is used to check the log for Memory errors.
#Tomcat dir aka CATALINA_DIR
TOMCAT_DIR=/usr/local/jakarta-tomcat

The script then uses curl to invoke Tomcat start/stop WebApps commands.


The initial version of the status routine just opened the WebApp default page to see if it got some text.
Later, I added another verification to see if Tomcat hadn't died with a Permgen error (memory issues, bad app..).

You can extend the status routine to check for internal variables in your WebApp and determine  it's status or any other condition that signals a fault and requires a restart.


To make this restart webapps and Tomcat in scenarios where you don't have a cluster you can use the watchdog template script on my blog (This one!) to make a periodic checks and mimic cluster behaviour.

Getting an ILI9341 SPI screen working on the RaspberryPI

I recently bought a small SPI screen from ebay which uses the ILI9341 controller.
The screen is identical to the one made by ElecFreaks (This one!).
Although I got mine really cheap from eBay....

I decided to use notro drivers since they are flexible and already provide a driver for the ILI9341. You can find the source code and install instructions here.

I actually compiled a custom kernel where I added his drivers.

After compilation and module installation I just had to figure out the parameters I needed to use.

But first we still need to hook it up to the RaspPI :-)

I used the following connections, also from notro's Wiki

I used the wiring schema for an Adafruit LCD, check the original table here.

Display	Driver gpio name	Raspberry Pi	Remark
BL	led	GPIO18
SCK		SCLK
MISO			Used by SD-card
MOSI		MOSI
CS		CE0	Chip Select
SDCS			SD-card Chip Select
RST	reset	GPIO25
D/C	dc	GPIO24
VIN		3.3V
GND		GND

Aside from a normal SPI connection, please notice the BL(led), RST(reset) and D/C(dc) pins since these will be used as parameters for the module.


Once you have everything hooked up, compiled and installed you can test out the screen by loading the module:


sudo modprobe fbtft_device custom name=fb_ili9341  gpios=reset:25,dc:24,led:18 speed=16000000 rotate=90 bgr=1

If you change one of the pins above, please change the gpios parameter accordingly.
Also, you have the rotate option to put the screen in landscape mode, by default it will be in portrait mode.
The bgr=1 option is to exchange the blue and red colors, by default the colors don't appear correctly and this option is necessary.
Once you load the module nothing will happen on your screen (this made think it wasn't working...) you can test it out by either sending the console to the framebuffer with:

con2fbmap 1 1

(it means, send console 1 to framebuffer 1)

or using X

FRAMEBUFFER=/dev/fb1 startx


If you want to make the module load automatically, add the following line to /etc/modules:
fbtft_device custom name=fb_ili9341  gpios=reset:25,dc:24,led:18 speed=16000000 rotate=90 bgr=1

EDIT: For the built-in version, you can add this to the kernel command line:
fbtft_device.name=fb_ili9341 fbtft_device.gpios=reset:25,dc:24,led:18 fbtft_device.speed=16000000 fbtft_device.rotate=90 fbtft_device.bgr=1 fbtft_device.custom=1


Hope you find this useful.