SPIE.org : SPIE Newsroom : Ultrashort-pulse laser ablation: insights from molecular-dynamics simulation

SPIE.org : SPIE Newsroom : Ultrashort-pulse laser ablation: insights from molecular-dynamics simulation

When PXE saves the day...

It was so shocking last night when one of our filers crashed. It would have not so terrible if the mainboard is , say, normal.  This system is my self built NAS-station, having 6 SATA ports. Yes, it is not normal, since all the ports are used by the harddisk, and I used "internal" USB slot to boot. I had no choice. The weird thing is that no IDE connector exist, but the old-useless-ugly-space-wasting-floppy-disk-connector is there! I never understand brain damaged companies that keep retaining this connector. To my opinion IDE is much more useful like for plugging a flash-disk.

The headache is, no direct access to the server room, no available replacement of the USB stick, and no port to plug a harddisk. It was time to think of a strategy. The only hope is network boot, pxe, as the boot server is alway running in the head. Thank to IPMI, I could just do everything remote although the speed is somehow irritating. Well the strategy is, boot to network, use ramdisk (tmpfs). This should be easy, I thought.

I worked on automating node installation using pxe, tftp, http, etc, a couple of years ago. So I dig again a bunch of scripts I wrote. It functioned, and it should still function this time. This is actually a brilliant idea of mine, if I'm allowed to say so, that to an unregistered node or any alien computer is given an installation boot scheme. One need only to create pxe boot configuration named before the mac address of the ip given by the dhcp server in hexadecimal form. Deleting the file would send a node to the default configuration, which is the installation. This default scheme is created by modifying initrd to allow loading installation script from the http server on the head. It's quite efficient and practical, since a user may change the script according to his need, not only for installation purpose. I called this a respond script, because it is used as a respond to the request sent by a booting node. The node send information of it's configuration (minimalist) such as mac addresses, information about hardisks in the request which is named by the first mac address.

The respond script will be executed by the node. To install a node it may have a straight forward step.

1. Format hardisk
2. Create disk image, maybe using debootstrap, etc.
3. Load harddisk content from the server and put it in the harddisk. I used wget to take a targz-ed disk image. Quite efficent.
4. Make a correct PXE boot configuration and let the node reboot.

This time I failed to use the scheme for installation. The reason is, that mdadm and lvm2 can not be installed in the debootstraped directory. So I used it just to diagnose the broken NAS.  I only need to put 'sh' in the respond script to put me in shell command line.  So I installed first a lazy computer (it did not do any work) and make sure that everything works fine and able to load mdraid and lvm volumes, and take the targz-ed image from it.

Then It came to my mind, it should be easier just to put the disk image inside the initrd. Yes, the size explodes but who cares. One needs only about 250MB unzipped space for NAS. I modified initrd so it mount a tmpfs as root, and untargzs the attached disk image there.  Now, actually I did not need the scheme I told you before, but I don't feel guilty telling you that because this idea came after meshing around with it. So I need only two files, the kernel and the fat initrd.

In init script inside initrd file, there are two options to load root file system, local and nfs. I could have added new options, say ramdisk, but I decided to do it in a rude way: force the system to use ramdisk.
A simple inclusion in scripts directory, called 'ramdisk' is sufficient:

# ramdisk

mountroot ()
{
  wait_for_udev 10
  mount -t tmpfs none ${rootmnt} -o size=2048M
  tar xzvf /disk.tgz -C ${rootmnt}
  rm /disk.tgz
}

and set $BOOT variable in init script to ramdisk. Actually I did it hard way, call directly scripts/ramdisk before 'maybe_break mountroot'.

Boots... and smiled.... I then did ssh to the node and all the raids and lvms are there. Mount them, export to nfs, done. At the end I copied the kernel and initrd also to a USB disk. Now it's better because the disk is only used for loading the image in boot time.

Notes:

• debian squeeze is used.
• unpacking initrd: gzip -dc path/to/initrd-file|cpio -id
• repacking initrd: find .|cpio -H newc -o|gzip > ../initrd-ramdisk , from the unpacked directory.
• Services on server:
• dhcp/bootp
• Required packages for NAS:
• mdadm
• lvm2
• rsync
• ssh
• modification needed in the disk image:
• /etc/fstab
• /etc//udev/rules.d/70-persistent-net.rules
• /etc/exports
• cleaning of unnecessary file under /usr and /var

Using ipmitool @SuperMicro server

It' about time using IPMI instead of KVM-switch. In fact this capability is all the time there in our servers. Don't ask why I did not do this from the beginning.
Anyway, this is how to redirect console output to serial over lan (SOL), and controlling power via IPMI.
In fact a SuperMicro server has already embedded IPMI in the mainboard, which is accessible using http connection (Luckily its true for me). It has as well a KVM over ip facility, using java's iKVM. But, this java dependency is not so "flexible" for my need, regarding the fact that java plugin does not (currently) always work (particularly in x64). I think console application is still the perfect tool. So, I put my eyes to impiconsole and ipmitool.

Before using remote management, first you have to do some BIOS settings. This figures explain those settings, quite clearly:

Remote access settings 

IPMI IP settings

Notice that COM3 must be used to make SOL works properly.

After all the settings are correct, you may direct a browser to the specified ip address, and the server can be fully controlled using browser. Console redirection will work if java plugin is configured correctly (under "Remote Control" menu).

Now install ipmitool and freeipmi-tools on the managing server (client, or whatsoever):

apt-get install ipmitool freeipmi-tools

Ipmitool is used for controlling/managing the server. It has a complete set of instructions. In fact, only "power on" and "power off" command are the most useful ones. Example:

ipmitool -H 192.168.0.100 -U ADMIN -P ADMIN power on

to power on the unit. ADMIN and ADMIN is the default user name and password.

To enable access using SOL, the following configuration must be done:

1. @/boot/grub/grub.cfg add this:

serial --unit=2 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console
and add "serial console=ttyS2,115200n8" to the "linux" line.

Don't use splash!
(Yes, grub2 configuration file looks a bit weird...)

2. @/etc/inittab add this:

s0:2345:respawn:/sbin/agetty ttyS2 115200

3. Make sure that ttyS2 is listed in /etc/securetty

Thats it. Now call:

ipmiconsole -h 192.168.0.100 -u ADMIN -p ADMIN

and boot the server. This is more or less the console output from the server:

For installation purpose, ipmiconsole is good if the installation media is able to redirect its output to the serial port. If not, the iKVM konsole (http access) is better, since it does not need any COM port at all.
This ipmitool command has the same function as ipmiconsole:

ipmitool -I lanplus -H 192.168.0.100 -U ADMIN -P ADMIN sol activate

See manual page of ipmitool and ipmiconsole for more complete reference.

One good thing from this server that IPMI may use a network wire together with the first interface (eth0), so I can use one less cable. Indeed, a separate RJ45 connector is also provided. In the client side I used multiple address in one of interfaces. Embarrassingly, I found it out recently that this is so easy to configure in Linux.

Say, in the admin computer (actually the main server) you have eth0 as your main interface, and is connected to the real ip address in the network. Since the network cable in the server (node) is used together by IPMI and the main interface, both are plugged into the same hub connection. Normally we need two interfaces plugged to the same hub at admin side.  Uneconomical indeed.
The better way is using multiple address for a single interface. After bringing eth0 up, bring another interface, eth0:0, as example:

ifconfig eth0:0 192.168.0.1 netmask 255.255.0.0

In debian system, one can put in /etc/network/interfaces:

auto eth0:0

iface eth0:0 inet static
address 192.168.0.1
netmask 255.255.0.0

Now the first interface can be used both to access the internet with its original address (eth0) and also for connecting IPMI devices in 192.168.0.0 network.
For mac user, multiple address in a single interface can be easily configured in "Network Preferences".