Friday, 9 November 2012

LVM: lvextend failing

02.48pm. Today I ran into the following problem whilst trying to extend a LVM logicial volume.

[root@HOSTNAME mnt]# lvextend -L +20GB /dev/vg_HOSTNAME/lv_var
Extending logical volume lv_var to 60.00 GB
device-mapper: reload ioctl failed: Invalid argument
Failed to suspend lv_var

Version information:

[root@HOSTNAME var]# lvm version 
LVM version:     2.02.56(1)-RHEL5 (2010-04-22)
Library version: 1.02.39-RHEL5 (2010-04-22)  Driver version:  4.11.

Distribution Information:

[ChrisDaish@HOSTNAME ~]$ cat /etc/redhat-release
CentOS release 5.5 (Final)

After a quick google I discovered that these chaps had the same problem: http://www.linuxgeek.net/2011/04/06/three-virtual-machines/

Essentially I had created a new Physical Volume (PV) on /dev/sdb:

[root@HOSTNAME mnt]# pvcreate /dev/sdb2

I then extended the current Volume Group (VG) to include the new PV:

[root@HOSTNAME mnt]# vgextend vg_HOSTNAME /dev/sdb2
  Volume group "vg_HOSTNAME" successfully extended

You can confirm the new PV is part of the VG:

[root@HOSTNAME mnt]# vgdisplay
  --- Volume group ---
  VG Name               vg_HOSTNAME
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  15
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               5
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               116.08 GB
  PE Size               4.00 MB
  Total PE              29716
  Alloc PE / Size       20224 / 79.00 GB
  Free  PE / Size       9492 / 37.08 GB
  VG UUID               OEuajs-XUDQ-tgJB-NIVZ-cCIH-N3fh-1v0rHX

The penultimate task is to extend the Logical Volume, however this is when I ran into the error:

[root@HOSTNAME mnt]# lvextend -L +20GB /dev/vg_HOSTNAME/lv_var
Extending logical volume lv_var to 60.00 GB
device-mapper: reload ioctl failed: Invalid argument
Failed to suspend lv_var

After a good couple of hours experimenting, a fix was discovered (by @JamesYale) that allowed me to extend the LV.

[root@HOSTNAME ~]# lvextend -L +20G /dev/vg_HOSTNAME/lv_var /dev/sdb2
Extending logical volume lv_var to 61.37 GB
Logical volume lv_var successfully resized

By specifying the second PV in the lvextend command, I was able to increase the storage as required.. I can only presume this was because the first PV was getting close to full (even though it reported 6GB free).

Finally I can use resize2fs to extend the file system accordingly.

[root@HOSTNAME ~]# resize2fs /dev/vg_HOSTNAME/lv_var
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/vg_HOSTNAME/lv_var is mounted on /var; on-line resizing required
Performing an on-line resize of /dev/vg_HOSTNAME/lv_var to 16087040 (4k) blocks.
The filesystem on /dev/vg_HOSTNAME/lv_var is now 16087040 blocks long.

Hopefully this will help others who run into this problem, leave a comment below if so.

Tuesday, 6 November 2012

Postfix Mail Queue Tools

04:26pm. A quick post on Postfix tools:

Flush postfix mail queue with:
postqueue -f
View mail queue with either:
mailq or postqueue -p
More to come....

Crontab + PAM + User with no password

15.53pm. Today I discovered an issue whereby a local users crontab was not running. The user was originally created to only run a script and was not required to own credentials as we could always su to the user when required. However as time passes the requirement to run the script became more and more frequent so I decided to create a crontab entry for the user. This is the error I then received:
tail -f /var/log/cron
Oct 30 11:21:01 SERVERNAME crond[22739]: Authentication service cannot retrieve authentication info
Oct 30 11:21:01 SERVERNAME crond[22739]: CRON (USERNAME) ERROR: failed to open PAM security session: Success
Oct 30 11:21:01 SERVERNAME crond[22739]: CRON (USERNAME) ERROR: cannot set security context
This server uses the Pluggable Authentication Module (PAM), so I thought I would take a look at the following log file too:
tail -f /var/log/secure
Oct 30 11:19:01 SERVERNAME unix_chkpwd[22423]: could not get username from shadow (USERNAME))
Oct 30 11:19:01 SERVERNAME crond
[22422]: pam_unix(crond:account): unix_update returned error 9
So this log points me to the to the /etc/shadow file, as the local user does not own a set of credentials there is no entry.

I then tried to give the user a password (and populate the shadow file) but I then hit this problem:
[root@SERVERNAME home]# passwd USERNAME
Changing password for user USERNAME.
passwd: Authentication token manipulation error
Thankfully  I was able to get around this stumbling block by using the 'vipw' tool. However I had to make a fake change to /usr/passwd file for the tool to move on to /etc/shadow, I then re-ran the tool to revert the change. Once complete the crontab worked as expected.

Hurrah!

Apache Alias's to the rescue!

15.19pm.Are you are looking to consolidate multiple web-roots into one, but want to ensure that the old URLs are still active and function as required? If so Apache Alias's are your friend. With a single command in a vhost or httpd.conf you can transparently redirect a user to a new location.
Alias /MyFirstWebsite/images/profilePhotos /var/www/mySecondWebsite.com/images/betterProfilePhotos 
I put this above any <Location> or <Directory> statements, but below the DocumentRoot,ServerName, and ServerAlias directives.

It is important to always test these changes with an apache/httpd reload rather than a restart to minimise downtime if a config error has occurred.

NB: All required files should also be in the new web-root and have the same name, if not you will receive a 404.

Friday, 2 November 2012

Securely Mounting Samba / CIFS file systems in CentOS 5.6

14.27pm. Earlier I wrote a blog post on how to mount a cifs share in CentOS however I was made aware by a reader that any user on the machine would be able to view the /etc/fstab file and view the credentials inside. This is obviously not ideal, so after a little reading I discovered that I can instead link to a file containing the credentials from /etc/fstab. This credentials file could then be locked down so only root can view it.

Firstly I created a new file in /etc/
touch /etc/COMPUTERHOSTNAME-cifs.credentials
Populate the file with the username and password as required:
username=DOMAIN\USERNAME
password=PASSWORD
chmod the file to 700 so only root can read, write, and execute the file.
chmod 700 /etc/COMPUTERHOSTNAME-cifs.credentials
Finally update your /etc/fstab to read the credentials file:
//SERVERNAME/SHARE /mnt/MOUNTPOINT/ cifs credentials=/etc/COMPUTERHOSTNAME-cifs.credentials 0 0
Once again check the fstab works as expected, umount the cifs mount, and re-running:
mount -a
We can also test viewing the file as a normal user, to ensure the credentials are hidden:
[user@HOSTNAME NormalUser]$ cat /etc/COMPUTERHOSTNAME-cifs.credentials
cat: /etc/COMPUTERHOSTNAME-cifs.credentials: Permission denied
Bingo!

Update: 6-11-2012
Last week I wrote this article to mount a cifs share securely however today I was required to mount a cifs share (which happened to be their ActiveDirectory home drive) to the users home folder on Linux. Using the instructions above I was able to mount the home drive, however I was then unable to access the mount as a normal user. I resolved this by using the following line in the /etc/fstab file:

//SERVER/SHARE /home/USER/Home-Drive/ cifs credentials=/home/user/domain.credentials,file_mode=0770,dir_mode=0770,uid=X,gid=X 0 0

I was able to ascertain the users uid and gid by grabbing the output of "id" as the user. I also created the mount point folder in preperation
sudo su - AnotherUser
id
mkdir ~/Home-Drive
As root I could then run 'mount -a'. This solution does have its flaws as the users password will eventually expire, this in turn will require the credentials file to be updated. The user will also have to either reboot the machine or ask the root user to perform any remount when required.

Update: 25-02-2014
Another scenario would be to use a service account rather then a specific user account to mount the share. These service accounts are usually locked down to ensure they cannot be used outside their desired role.

fstab entry:
//serverName/Share$     /mnt/ShareName     cifs    ro,credentials=/root/.cifs.credentials 0 0

credentials file (chmod 700):
username=domain\lb-slp-en-grp02-cifs
password=password

On the Windows Server I then need to give this user RO access. This can be performed by locating the share under the shares management tool (Computer Management -> Shared Folders -> Shares). Locate the 'Security' tab found under the properties for the share and add the cifs user found above.

Global / System wide environmental variables on Mac OSX

2.40pm. Today I was asked how to set environmental variables on Mac OSX, after a little research it would appear most solutions required the user to create a ~./MacOSX/environment.plist in their home directory. However this meant that every user on the box had to jump through this hoop.

To apply the environmental variable system wide I performed the following:
sudo vi /etc/launchd.conf
I then populated the /etc/launchd.conf file with my environmental variables, something like:
setenv    JAVA_HOME    /opt/java/bin/java
setenv    VIDEO_IN       /Videos/In
Reboot the Mac and you are all set.

Mounting Samba / CIFS file systems in CentOS 5.6

10.50am. Mounting remote smb/cifs filesystems can be a little tricky due to the slashes required to complete the task.

From the command line:
mkdir /mnt/MOUNTPOINT (if you have not already done so)

mount -t cifs "//SERVERNAME/SHARE" /mnt/MOUNTPOINT/ -o username=DOMAINNAME\\USERNAME,password=PASSWORD
NB: the text in capitals will need to be substituted for your own scenario. Also note that there are two backward slashes between DOMAINNAME and USERNAME. This is by design as the cli will cancel out just one.

Once successful you may want to have this file system automatically mounted on boot. This can be achieved by modifying the /etc/fstab.

Insert into /etc/fstab:
//SERVERNAME/SHARE /mnt/MOUNTPOINT/ cifs username=DOMAIN\USERNAME,password=PASSWORD
NB: Again the text in capitals will need to be substituted for your own requirements. This time we only require one backslash between DOMAINNAME and USERNAME.

Once the fstab has been modified it is good practice to test that is performs as required, we can do that by performing the following:
umount /mnt/MOUNTPOINT (may still be mounted from testing earlier).
mount -a
Any errors will display on the command line, the mount can also be tested by getting a directory listing.

Thursday, 25 October 2012

Checking if a package or rpm is installed on CentOS, Fedora, or Redhat system

4.55pm. Another quick post, I wanted to find out if a package was already installed on a Fedora system, at first I though I could use:
yum search PackageName
However this only told me that it was available in the yum repositories, not if it was installed. Instead I wanted:
rpm -qa|grep PackageName 
The "rpm -qa" command queries the local package manager database and outputs every package you have installed (including yum installs). I then pipe it to grep along with a search term.

The end result tells me what packages and versions are installed on that system, and if I need to run the following command or not:
yum install PackageName

Friday, 8 April 2011

Git is good!

5.30pm. So I started off a new top secret project at work that I can only only describe as 'project black bear', a great name I am sure you'll aggree, which actually serves two purposes; firstly giving the project a name, until we come up with a better one, and secondly keeping the whole project hush hush within the college.

So my team all cracked open their editor of choice and started some fun PHP/MySQL hacking. We all then hit a point when we needed to merge our code, thankfully I had already had a play with git a couple of months ago and setup a new project. A couple of pulls, commits, and pushes later we were all running the same code and pushing code to the central project whenever we made a major change.

We were then able to monitor the progress via the commit messages and a great cgi script; gitweb. Sure we did receive the odd commit error however this was easily fixed with a quick diff merge.

After a pretty constructive day I'm pretty sure its time to crack open a cold beer and enjoy the rarety of British sun (not the paper).

Thursday, 31 March 2011

nrpe check_smtp & check_mysql notes

4.12pm. Today I have been playing with nagios plugins that allow me to monitor every service on my development box (Sherpa).

NB: These are local NRPE based checks (the daemon only runs on localhost).

* 'check_smtp' is setup by installing 'nagios-plugins-smtp.x86_64' and then monitored by nrpe by adding the following command to the '/etc/nagios/nrpe.cfg' file.
command[check_smtp]=/usr/lib64/nagios/plugins/check_smtp -H localhost
I can then check this from my NetworkMonitor box via:
# /usr/lib64/nagios/plugins/check_nrpe -H sherpa -c check_smtp
SMTP OK - 0.013 sec. response time|time=0.013064s;;;0.000000
* 'check_mysql' is a little more difficult as it requires a new MySQL user to be created, but easily done. Again we install 'nagios-plugins-mysql.x86_64' but have to be perform the following before editing the '/etc/nagios/nrpe.cfg' file.
# mysql -u xxxxx -p
mysql> CREATE USER 'nrpe'@'localhost' IDENTIFIED BY '*****';
mysql> exit
Now we can test the command locally via:
# /usr/lib64/nagios/plugins/check_mysql -H localhost -u nrpe -p *****
or just add the following to the nrpe.cfg file.
command[check_mysql]=/usr/lib64/nagios/plugins/check_mysql -H localhost -u nrpe -p *****
Dont forget to restart the local nrpe service(!).

Now onto adding the service templates on our Nagios box:
define service{
use generic-service
host_name Sherpa.xxxxxxxxxxxx
service_description postfix SMTP Service Check
check_command check_nrpe!check_smtp
}
define service{
use generic-service
host_name Sherpa.xxxxxxxxxxxx
service_description MySQLd Service Check
check_command check_nrpe!check_mysql
}
reload nagios, job done.

Wednesday, 30 March 2011

Nagios NRPE: Unable to read output

3.25pm. Today I started looking into rolling out nagios monitoring over our *nix servers after the success of recently adding Windows Servers (via NSClient++) and our Core Switches (via SNMP).

Most of my servers are based on CentOS 5.5 so a quick search for nrpe (the linux alternative of nsclient++) on yum provided the required results:
nagios-plugins-nrpe.x86_64 : Provides nrpe plugin for Nagios
nrpe.x86_64 : Host/service/network monitoring agent for Nagios
Installing both packages went pretty well, adding the required nrpe user and service (just needed to 'chkconfig nrpe on' to ensure it started on boot). I then added the NetworkMonitor host IP to the nrpe.cfg file.
allowed_hosts=127.0.0.1,192.168.200.46
Annoyingly this is where I hit a road block, I was able to test communication from the Nagios/NetworkMonitor Box to the NRPE client:
# /usr/lib64/nagios/plugins/check_nrpe -H Sherpa
NRPE v2.12
However when it came to retrieving any useful data such as check_users or check_load I received the following error:
# /usr/lib64/nagios/plugins/check_nrpe -H sherpa -c check_users
NRPE: Unable to read output
# /usr/lib64/nagios/plugins/check_nrpe -H sherpa -c check_load
NRPE: Unable to read output
A quick in the NRPE plugins folder (/usr/lib64/nagios/plugins) provided the answer; it only contained the check_nrpe plugin (that I had installed earlier) and not a lot else. Sure enough I had only installed the nagios nrpe plugin (and not all the nrpe plugins that I first thought). Back to yum:
# yum search nagios-plugins-load
nagios-plugins-load.x86_64 : Nagios Plugin - check_load
So it looks like there is a plugin for each check, that's worth knowing! I can apply this logic to every item I now want to check on this host; or if I am feeling lazy install:
nagios-plugins-all.x86_64 : Nagios Plugins - All plugins
Which does the lot! Now to retest....

# /usr/lib64/nagios/plugins/check_nrpe -H sherpa -c check_users
USERS OK - 1 users currently logged in |users=1;5;10;0

# /usr/lib64/nagios/plugins/check_nrpe -H sherpa -c check_load
OK - load average: 0.00, 0.01, 0.00|load1=0.000;15.000;30.000;0; load5=0.010;10.000;25.000;0; load15=0.000;5.000;20.000;0;

Success!

Thursday, 24 March 2011

Monitoring a HP Procurve 5412zl switch interface with Nagios (SNMP)

8.50am. Yesterday I had a couple of minutes spare to build on my Nagios setup. Since setting up Nagios I have been looking into monitoring the fibre interfaces that connect my core and satellite switches. Unfortunately I was not able to find any good examples of this done before so I set out to try and set it up myself.

To complete this task I used the check_snmp plugin as a base and built up my own command to check the interface status. After running though an snmpwalk on the switch (snmpwalk -v1 IPHERE -c public >~/Procurve_snmpwalk) I discovered the following MIBS that I can use:

ifSpeed.X (speed of link)
ifOperStatus.X (link up or down)
ifOutOctets.X & ifInOctets.X
ifOutUcastPkts.X & ifInUcastPkts.X
ifOutNUcastPkts.X & ifInNUcastPkts.X

The X indicates the Port ID, however to use this integer we need to check what ID links to what interface. This can be achieved by performing a query on the ID using the ifDescr.X MIB. The results to my snmpwalk already gave me the results I required however they can be checked manually on the command line via:
/usr/lib64/nagios/plugins/check_snmp -H %HostnameHere% -C public -o ifDescr.X
By changing the value of X, I am presented with the interface name. For example in my case ID 2 = Port A2 on the switch.

Now I know what the interface ID is for the port is I can start to build up commands based on the MIBS i discovered above. To start with I wanted a simple interface check (up/down) so I wrote the following to the /etc/nagios/objects/commands.cfg file:

define command{
     command_name check_interface_up
     command_line $USER1$/check_snmp -H $HOSTADDRESS$ -C $ARG1$ -o $ARG2$ -w 0:1 -c 0:1 -l 'Link Status'
}

I have set the warning & critical ranges to 0:1. I am unsure if this is the best way to do it, but it does work. The switch returns 1 for interface up and 2 for interface down.

I then added the following to my switch template:

define service{
     use generic-service
     host_name Block-B-5412zl
     service_description Block-A-3500yl uplink 2 status
     check_command check_interface_up!public!ifOperStatus.2
}

Using the ! symbol as a separator I am able to pass different variables to the command; $ARG1$ and $ARG2$ .

$ARG1$ = the community string checked
$ARG2$ = This is where I define the SNMP MIB ifOperStatus.2, however I can replace this with different MIBs that I mentioned earlier.

Thats pretty much it, reload nagios and I am now able to monitor switches interfaces. Next task is monitoring the traffic flowing through the interfaces by monitoring ifOutUcastPkts.X & ifInUcastPkts.X in cacti.

Friday, 18 March 2011

4.50pm. I was recently told about the idea of publishing nagios alerts to a private twitter account

As it was a nice quiet day so I had a look at implementing the scripts today. I had a few problems getting the tweets through our http proxy, but managed to solve this problem by adding:

$ENV{http_proxy}='http://proxy:8080';

to the top of the nitter(v2) script. I then moved the script to /etc/nagios/ and I was away.


Fridays goal: complete.
10.50am. So I have had to look this command up a couple of times now as I keep forgetting. The command for installing CPAN modules into Perl is as follows:

perl -MCPAN -e shell
cpan> install Type::ModuleName

Today's reason for its use? Installing a Perl script to process twitter OAUTH authentication.
Why? Sending alerts from Nagios to twitter.

Wednesday, 16 March 2011

3.50pm. This week I have mostly been installing and configuring up: Nagios.

Cutting straight to the verdict: awesome.

So it takes a little while to set-up, but once you have learnt the slightly odd templates/services/commands/hosts system its just a case of replication.

So far I have configured it to monitor:
* the majority of network printers,
* our Windows servers,
* our Linux servers,
* ....and a handful of procurve switches.

I did have to disable the alerts on the Network Printers as they became incredibly annoying when office staff were saving the planet by turning them off at night.

Next step is to gather more statistics from the linux servers, and monitor key fibre interfaces on the switches.

Wednesday, 9 March 2011

6.45pm. Its done. The data mining/extraction script I have been working on in Perl is finally complete. Last week I set myself a goal of automating the search process that we as website users perform daily. I went to work and set myself up a dummy search site and had a play. A couple of days later (along with a few beers along the way) I have accomplished the following:

* The ability to login into access controlled sites that use session tracking.
* Download a copy of every searchable record, starting with an id 1 and ending with 10000.
* Extract the data from the local files, and build a CSV file.
* Clean up the CSV file, removing all null entries, and blank lines.

The end result?

A 700 line CSV containing information I require from an external searchable system.

Now where's that beer....

Friday, 4 March 2011

10.00pm. Okay so I having quite a productive day in the office, I decided to continue my brain workout by picking up some more Perl code.

I wrote a couple of scripts that are able to download data from search pages, including a version that is able to pass login information, this made the whole challenge slightly more difficult, in the end I was able to utilise the --keep-session-cookies & --save-cookies/--load-cookies arguments of wget to achieve my goals.

The next step was to read in an array of search variables to pass to wget, easily done. I then started looking at automating the whole process by looking at search forms that used unique id's in their POST data; with some friendly $int++ I was able to generate a whole bunch of searches with success.

Next step: Trawling through the pages to extract the data I require.

8.000pm. So over the last 2 days at work I have been setting up and configuring nagios. It's been on my to-do list for a while now (years) and I have finally found a couple of days to crack on with it. Its pretty easy to set up, and once you have worked out the difference between templates, hostgroups, and servicegroups you're all set.

I guess the only frustrating part is the constant personal email bombardment (every hour) if you leave something misconfigured in the office, can't really blame the product for that though.

Thursday, 3 March 2011

11.40am. Well that was easy. I upgraded our virtualisation infrastructure to Xenserver 5.6 a couple of weeks ago but at the time I forgot to upgrade the XenTools packages. With 5 minutes spare today I was able to complete this task by performing the following:

* Attach the XenTools ISO by selecting VM->Install XenServer Tools.
* Mount the DVD:
mount /dev/xvdd /mnt
* Run the script on the mounted DVD.
./mnt/Linux/install.sh
* Unmount the DVD
umount /mnt
* Restart the VM.
* Check under Virtualisation State (found under the general tab for that VM) it says: Optimized

Fin.

Tuesday, 1 March 2011

7.50pm. So I watched 'the social network' over the weekend, and after watching the 'hacking' scene at the beginning of the film, I can confirm that downloading the jpeg pictures via an apache index is indeed kids stuff.

In case you were you were wondering I accomplished it via:
wget -r -A .jpeg APACHE_INDEX_PAGE_URL_HERE