Archive for June, 2009

Last night I decided to browse to my vSphere server and download the latest Virtual Infrastructure client, I did this and everything went well – Too easy.

After installing this and having a play with vcenter 4, I decided to try web access so I browsed back to the default page as shown below.


I clicked on the “Log into Web Access” button and got the following error ” 503 service unavailable”


I then logged into my vSphere host and run the following command

chkconfig –list |grep “vmware-web” (double dash before list, not sure why but the font doesnt show this clearly)

Heres what I found


It now looks like Web Access is no longer automatically enabled and you have to perform the following command to make to start at boot.

chkconfig –level 345 vmware-webAccess on

and if you just want to start it the once you can run

/etc/init.d/vmware-webAccess start or service vmware-webAccess start

Hope this helps anyone out there searching.


For a while now Ive been using Workstation 6 on my laptop for a portable Virtual Infrastructure which can be used for testing and customer demos.

I mounted the ISO image and preceeded to upgrade virtual center only to receive the following meanacing error


Im taking a guess here but Id say there are tones of people like myself who have built lab systems and to minimise the number of systems needed, just installed Virtual Center on a domain controller.

There is no work around for this and as much as it annoys me having to go and build another machine, Im a firm beleiver that a domain controller should be just that. Youd be amazed at how many customer sites Ive seen over the years where the domain controller is also a file server, anti virus server, backup server, citrix server ( you get the idea ).

Ive been doing a fair bit of virtualization security lately and I thought id share a few tid bits on what Ive done and why. If y0u find this useful then check back every couple of days as ill be adding additional steps and verifying if these apply to VI3, Vsphere or both.

Most of you who are familiar with ESX will know  the default “Out Of The Box” behaviour restricts the user root from logging in directly via SSH which generally means either root user must authenticate as a standard user and then SU to root or log in directly from the console.

So now that weve established how ROOT user can and can not be used to login into the service console, lets look at a couple more options to tighten things even further.

Deny SSH access to all but specific ip addresses

Ive configured /etc/hosts.allow with 3 specific ip addresses that can SSH to the esx hosts in our cluster and configured /etc/hosts.deny with sshd:all which means all other IP addresses not configured in the hosts.allow files get the hard word and fail with an error “session terminated unexpectedly”

Only users in the WHEEL group can SU to Root

As mentioned above already if the sshd_config file is configured with Permit Root Login = no option, then ROOT is not able to login to the service console remotely e.g. via Putty. We can also take additional steps and limit which standard users can SU to root by changing the /etc/pam.d/su file and then adding only specific users to the wheel group. Here’s what you’ll need to do;

vi /etc/pam.d/su

Then un comment the following line

#auth       required     /lib/security/$ISA/ use_uid

Once youve done that, we can now go and look at which users are part of the WHEEL group.

cat /etc/group |grep wheel

which should return the following “wheel:x:10:root

Now if your standard user is called testuser, just add a comma and add testuser so it looks like this.


Now you can add users to the service console, but only user “testuser” will be able to SU to root

Password Protect GRUB

After reading this in one of the hardening guides, I decided that I wanted to protect the ESX boot loader (GRUB).

Just quickly for those who dont know, A quick extract from the GRUB website explaining what it is.

GRUB is the first software program that runs when a computer starts. It is responsible for loading and transferring control to the operating system kernel software. The kernel, in turn, initializes the rest of the operating system

Hardening the Linux boot loader is not new to me, its something I had done before but I would like to stick my hand up here and admit previously I had used a password in plain txt, this is very very LAZY. One of the golden rules of security is never leave passwords hanging around in plain text, even though the password was specific to grub alone (not used for ROOT or standard user) and I had restricted access to the grub configuration file using chmod 600 /boot/grub/grub.conf I still don’t consider this to be good enough, shame on me.

So years later after learning a few tricks, when I secure ESX systems I use md5crypt to generate a md5 hashed password which I then use in the configuration file. below Ill show you what I did to achieve this.

First make a backup of grub.conf file (Just in case)

[root@esxvsphere1 ~]# cp /boot/grub/grub.conf /home

Next lets generate that md5 hash using my chosen password of “testing”

[root@esxvsphere1 ~]# grub

GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

 [ Minimal BASH-like line editing is supported.  For the first word, TAB
   lists possible command completions.  Anywhere else TAB lists the possible
   completions of a device/filename.]

grub> md5crypt

Password: *******
Encrypted: $1$BPuA5/$zrkdzGA88uApbEL126tlE.


Exit, and now its time to take the hash and place it in our grub.conf file

[root@esxvsphere1 ~]# vi /boot/grub/grub.conf

Then add the following syntax password –md5 <md5hash>  right after the last # in the grub description section.

default 0
###################### grub.conf #####################
# this file was generated by
# Any entries in this file marked with the comment
#   #vmware:autogenerated esx
# Should not be edited by hand.  They are likely to
# be clobbered on or before the next reboot.
password –md5 $1$BPuA5/$zrkdzGA88uApbEL126tlE.

That’s it, all done. So now just one last step. As mentioned above I like to restrict access to the grub.conf file using chmod 600 /boot/grub/grub.conf

Now Im going to post some screen shots showing how this has changed the behaviour of GRUB, but in order to do that, ill need to start with how it looks by default

Origional Boot Loader : Note the naughty “E” for edit which gives anyone at the console the ability to edit the parameters which are passed to the kernel.



Password protected Boot Loader : Now notice “E” has been replaced with the option “P” to enter password which then allows access to the additional features



Now lets hit the “P” and make sure it requests for password


That’s it, all done.

Recently I needed to setup AD authentication with ESX 3.5 as part of a security hardening exercise which stated users other then root needed to authenticate against AD rather then using local passwords.

Off I went on my quest to enable this which brought me to this VMware document here. After browsing through it and thinking “That looks fairly simple” I went and run the following command on the ESX 3.5 service console

esxcfg-auth –enablead – –

useradd testuser  (This creates a user account on the ESX server, don’t set password for this account)

I then launched a putty session and tried to login but I  kept getting the error “access denied”,  so I  went and tailed the messages log using tail -f /var/log/messages and noticed the error “Time Skew to great” which told me this was a time issue.

I looked at the time on the ESX service console and it was within 30 seconds so I was a bit puzzled because in the past Ive read and experienced problems only if the time was skewed more then 10 minutes . After a heap of playing around I thought well ill set the ESX server to use the AD controller for time so I went and configured NTP and gave it another crack.

Success …. so from this  another reminder to myself about just how important time is within an ESX cluster.

So the next think I wanted to test was if these steps where the same for Vsphere. After running the exact commands as shown above I can confirm the exact same steps also configure AD authentication with Vsphere.

Also just as a note, if you read one of my  posts last week about the Vsphere installer (here) … you might have noticed that NTP can be configured during the install process which is really good to see because Ive lost count of the times Ive seen people forget to set this up post install. (Yes myself included)

I was working on a customers site not long ago and had to set a static MAC address for a Virtual Machine so when it moved between hosts via a manual or DRS initiated migration it would not alter the MAC and in turn break the application that was hardcoded to a particular MAC.

I remember thinking at the time if there was a way you could tell a VM to ignore or mask additional CPU’s to aviod licensing contraints…. as this particular application was bound to MAC and also number of CPU’s.

 A month or so later im browsing around and I found Duncan from yellow-bricks posted this neat little trick.

  1. Power off the VM
  2. Right click on the VM and select “Edit Settings…”
  3. Select the “Options” tab
  4. Click on “General” (in the “Advanced” options section)
  5. Click “Configuration Parameters…” (in the pane on the right)
  6. Click “Add Row”
  7. Enter “cpuid.coresPerSocket” in the “Name” column
  8. Enter a value (try 2, 4, or in the “Value” column
  9. Click “OK”
  10. Power on the VM

The VM will now appear to the OS as having multi-core CPUs with the number of cores per CPU given by the value that you selected. For example, if you create an 8 VCPU VM and set “cpuid.coresPerSocket = 2″ it will be recognized as 4 dual-core CPU’s by the OS while it’s actually utilizing 8 physical cores.