LVM: lvextend failing

02.48pm. Today I ran into the following problem whilst trying to extend a LVM logicial volume.

[root@HOSTNAME mnt]# lvextend -L +20GB /dev/vg_HOSTNAME/lv_var
Extending logical volume lv_var to 60.00 GB
device-mapper: reload ioctl failed: Invalid argument
Failed to suspend lv_var

Version information:

[root@HOSTNAME var]# lvm version 
LVM version:     2.02.56(1)-RHEL5 (2010-04-22)
Library version: 1.02.39-RHEL5 (2010-04-22)  Driver version:  4.11.

Distribution Information:

[ChrisDaish@HOSTNAME ~]$ cat /etc/redhat-release
CentOS release 5.5 (Final)

After a quick google I discovered that these chaps had the same problem: http://www.linuxgeek.net/2011/04/06/three-virtual-machines/

Essentially I had created a new Physical Volume (PV) on /dev/sdb:

[root@HOSTNAME mnt]# pvcreate /dev/sdb2

I then extended the current Volume Group (VG) to include the new PV:

[root@HOSTNAME mnt]# vgextend vg_HOSTNAME /dev/sdb2

  Volume group "vg_HOSTNAME" successfully extended

You can confirm the new PV is part of the VG:

[root@HOSTNAME mnt]# vgdisplay

  --- Volume group ---

  VG Name               vg_HOSTNAME

  System ID

  Format                lvm2

  Metadata Areas        2

  Metadata Sequence No  15

  VG Access             read/write

  VG Status             resizable

  MAX LV                0

  Cur LV                6

  Open LV               5

  Max PV                0

  Cur PV                2

  Act PV                2

  VG Size               116.08 GB

  PE Size               4.00 MB

  Total PE              29716

  Alloc PE / Size       20224 / 79.00 GB

  Free  PE / Size       9492 / 37.08 GB

  VG UUID               OEuajs-XUDQ-tgJB-NIVZ-cCIH-N3fh-1v0rHX

The penultimate task is to extend the Logical Volume, however this is when I ran into the error:

[root@HOSTNAME mnt]# lvextend -L +20GB /dev/vg_HOSTNAME/lv_var
Extending logical volume lv_var to 60.00 GB
device-mapper: reload ioctl failed: Invalid argument
Failed to suspend lv_var

After a good couple of hours experimenting, a fix was discovered (by @JamesYale) that allowed me to extend the LV.

[root@HOSTNAME ~]# lvextend -L +20G /dev/vg_HOSTNAME/lv_var /dev/sdb2
Extending logical volume lv_var to 61.37 GB
Logical volume lv_var successfully resized

By specifying the second PV in the lvextend command, I was able to increase the storage as required.. I can only presume this was because the first PV was getting close to full (even though it reported 6GB free).

Finally I can use resize2fs to extend the file system accordingly.

[root@HOSTNAME ~]# resize2fs /dev/vg_HOSTNAME/lv_var
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/vg_HOSTNAME/lv_var is mounted on /var; on-line resizing required
Performing an on-line resize of /dev/vg_HOSTNAME/lv_var to 16087040 (4k) blocks.
The filesystem on /dev/vg_HOSTNAME/lv_var is now 16087040 blocks long.

Hopefully this will help others who run into this problem, leave a comment below if so.

Postfix Mail Queue Tools

04:26pm. A quick post on Postfix tools:

Flush postfix mail queue with:

postqueue -f

View mail queue with either:

mailq or postqueue -p

More to come....

Crontab + PAM + User with no password

15.53pm. Today I discovered an issue whereby a local users crontab was not running. The user was originally created to only run a script and was not required to own credentials as we could always su to the user when required. However as time passes the requirement to run the script became more and more frequent so I decided to create a crontab entry for the user. This is the error I then received:

tail -f /var/log/cron
Oct 30 11:21:01 SERVERNAME crond[22739]: Authentication service cannot retrieve authentication info
Oct 30 11:21:01 SERVERNAME crond[22739]: CRON (USERNAME) ERROR: failed to open PAM security session: Success
Oct 30 11:21:01 SERVERNAME crond[22739]: CRON (USERNAME) ERROR: cannot set security context

This server uses the Pluggable Authentication Module (PAM), so I thought I would take a look at the following log file too:

tail -f /var/log/secure
Oct 30 11:19:01 SERVERNAME unix_chkpwd[22423]: could not get username from shadow (USERNAME))
Oct 30 11:19:01 SERVERNAME crond[22422]: pam_unix(crond:account): unix_update returned error 9

So this log points me to the to the /etc/shadow file, as the local user does not own a set of credentials there is no entry.

I then tried to give the user a password (and populate the shadow file) but I then hit this problem:

[root@SERVERNAME home]# passwd USERNAME
Changing password for user USERNAME.
passwd: Authentication token manipulation error

Thankfully  I was able to get around this stumbling block by using the 'vipw' tool. However I had to make a fake change to /usr/passwd file for the tool to move on to /etc/shadow, I then re-ran the tool to revert the change. Once complete the crontab worked as expected.

Hurrah!

Apache Alias's to the rescue!

15.19pm.Are you are looking to consolidate multiple web-roots into one, but want to ensure that the old URLs are still active and function as required? If so Apache Alias's are your friend. With a single command in a vhost or httpd.conf you can transparently redirect a user to a new location.

Alias /MyFirstWebsite/images/profilePhotos /var/www/mySecondWebsite.com/images/betterProfilePhotos 

I put this above any <Location> or <Directory> statements, but below the DocumentRoot,ServerName, and ServerAlias directives.

It is important to always test these changes with an apache/httpd reload rather than a restart to minimise downtime if a config error has occurred.

NB: All required files should also be in the new web-root and have the same name, if not you will receive a 404.

Securely Mounting Samba / CIFS file systems in CentOS 5.6

14.27pm. Earlier I wrote a blog post on how to mount a cifs share in CentOS however I was made aware by a reader that any user on the machine would be able to view the /etc/fstab file and view the credentials inside. This is obviously not ideal, so after a little reading I discovered that I can instead link to a file containing the credentials from /etc/fstab. This credentials file could then be locked down so only root can view it.

Firstly I created a new file in /etc/

touch /etc/COMPUTERHOSTNAME-cifs.credentials

Populate the file with the username and password as required:

username=DOMAIN\USERNAME
password=PASSWORD

chmod the file to 700 so only root can read, write, and execute the file.

chmod 700 /etc/COMPUTERHOSTNAME-cifs.credentials

Finally update your /etc/fstab to read the credentials file:

//SERVERNAME/SHARE /mnt/MOUNTPOINT/ cifs credentials=/etc/COMPUTERHOSTNAME-cifs.credentials 0 0

Once again check the fstab works as expected, umount the cifs mount, and re-running:

mount -a

We can also test viewing the file as a normal user, to ensure the credentials are hidden:

[user@HOSTNAME NormalUser]$ cat /etc/COMPUTERHOSTNAME-cifs.credentials
cat: /etc/COMPUTERHOSTNAME-cifs.credentials: Permission denied

Bingo!

Update: 6-11-2012
Last week I wrote this article to mount a cifs share securely however today I was required to mount a cifs share (which happened to be their ActiveDirectory home drive) to the users home folder on Linux. Using the instructions above I was able to mount the home drive, however I was then unable to access the mount as a normal user. I resolved this by using the following line in the /etc/fstab file:

//SERVER/SHARE /home/USER/Home-Drive/ cifs credentials=/home/user/domain.credentials,file_mode=0770,dir_mode=0770,uid=X,gid=X 0 0

I was able to ascertain the users uid and gid by grabbing the output of "id" as the user. I also created the mount point folder in preperation

sudo su - AnotherUser
id
mkdir ~/Home-Drive

As root I could then run 'mount -a'. This solution does have its flaws as the users password will eventually expire, this in turn will require the credentials file to be updated. The user will also have to either reboot the machine or ask the root user to perform any remount when required.

Update: 25-02-2014
Another scenario would be to use a service account rather then a specific user account to mount the share. These service accounts are usually locked down to ensure they cannot be used outside their desired role.

fstab entry:
//serverName/Share$     /mnt/ShareName     cifs    ro,credentials=/root/.cifs.credentials 0 0

credentials file (chmod 700):
username=domain\lb-slp-en-grp02-cifs
password=password

On the Windows Server I then need to give this user RO access. This can be performed by locating the share under the shares management tool (Computer Management -> Shared Folders -> Shares). Locate the 'Security' tab found under the properties for the share and add the cifs user found above.

Global / System wide environmental variables on Mac OSX

2.40pm. Today I was asked how to set environmental variables on Mac OSX, after a little research it would appear most solutions required the user to create a ~./MacOSX/environment.plist in their home directory. However this meant that every user on the box had to jump through this hoop.

To apply the environmental variable system wide I performed the following:

sudo vi /etc/launchd.conf

I then populated the /etc/launchd.conf file with my environmental variables, something like:

setenv    JAVA_HOME    /opt/java/bin/java
setenv    VIDEO_IN       /Videos/In

Reboot the Mac and you are all set.

Mounting Samba / CIFS file systems in CentOS 5.6

10.50am. Mounting remote smb/cifs filesystems can be a little tricky due to the slashes required to complete the task.

From the command line:

mkdir /mnt/MOUNTPOINT (if you have not already done so)

mount -t cifs "//SERVERNAME/SHARE" /mnt/MOUNTPOINT/ -o username=DOMAINNAME\\USERNAME,password=PASSWORD

NB: the text in capitals will need to be substituted for your own scenario. Also note that there are two backward slashes between DOMAINNAME and USERNAME. This is by design as the cli will cancel out just one.

Once successful you may want to have this file system automatically mounted on boot. This can be achieved by modifying the /etc/fstab.

Insert into /etc/fstab:

//SERVERNAME/SHARE /mnt/MOUNTPOINT/ cifs username=DOMAIN\USERNAME,password=PASSWORD

NB: Again the text in capitals will need to be substituted for your own requirements. This time we only require one backslash between DOMAINNAME and USERNAME.

Once the fstab has been modified it is good practice to test that is performs as required, we can do that by performing the following:

umount /mnt/MOUNTPOINT (may still be mounted from testing earlier).
mount -a

Any errors will display on the command line, the mount can also be tested by getting a directory listing.