# vmstat
procs -
memory----
-swap -
io---- system -
cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 7710432 15784 136772 0 0 32 195 504 21 1 2 97 0 0
# dmesg
Linux version 2.6.18-92.1.1.el5 (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-41)) #1 SMP Thu May 22 09:01:47 EDT 2008
Command line: ro root=/dev/local/root rhgb quiet
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000efef0000 (usable)
BIOS-e820: 00000000efef0000 - 00000000efeff000 (ACPI data)
BIOS-e820: 00000000efeff000 - 00000000eff00000 (ACPI NVS)
BIOS-e820: 00000000eff00000 - 00000000f0000000 (usable)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000000200000000 (usable)
DMI present.
ACPI: RSDP (v000 PTLTD ) @ 0x00000000000f6c60
ACPI: RSDT (v001 PTLTD RSDT 0x06040000 LTP 0x00000000) @ 0x00000000efefab5a
ACPI: FADT (v001 INTEL 440BX 0x06040000 PTL 0x000f4240) @ 0x00000000efefef06
ACPI: MADT (v001 PTLTD APIC 0x06040000 LTP 0x00000000) @ 0x00000000efefef7a
ACPI: BOOT (v001 PTLTD $SBFTBL$ 0x06040000 LTP 0x00000001) @ 0x00000000efefefd8
ACPI: DSDT (v001 PTLTD Custom 0x06040000 MSFT 0x0100000d) @ 0x0000000000000000
No NUMA configuration found
Faking a node at 0000000000000000-0000000200000000
Bootmem setup node 0 0000000000000000-0000000200000000
Memory for crash kernel (0x0 to 0x0) notwithin permissible range
disabling kdump
On node 0 totalpages: 2001576
DMA zone: 2688 pages, LIFO batch:0
DMA32 zone: 964648 pages, LIFO batch:31
Normal zone: 1034240 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x1008
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 7:7 APIC version 17
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1 7:7 APIC version 17
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Setting APIC routing to physical flat
Using ACPI (MADT) for SMP configuration information
Nosave address range: 000000000009f000 - 00000000000a0000
Nosave address range: 00000000000a0000 - 00000000000ca000
Nosave address range: 00000000000ca000 - 00000000000cc000
Nosave address range: 00000000000cc000 - 00000000000dc000
Nosave address range: 00000000000dc000 - 0000000000100000
Nosave address range: 00000000efef0000 - 00000000efeff000
Nosave address range: 00000000efeff000 - 00000000eff00000
Nosave address range: 00000000f0000000 - 00000000fec00000
Nosave address range: 00000000fec00000 - 00000000fec10000
Nosave address range: 00000000fec10000 - 00000000fee00000
Nosave address range: 00000000fee00000 - 00000000fee01000
Nosave address range: 00000000fee01000 - 00000000fffe0000
Nosave address range: 00000000fffe0000 - 0000000100000000
Allocating PCI resources starting at f1000000 (gap: f0000000:ec00000)
SMP: Allowing 2 CPUs, 0 hotplug CPUs
Built 1 zonelists. Total pages: 2001576
Kernel command line: ro root=/dev/local/root rhgb quiet
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Checking aperture...
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing software IO TLB between 0x1c2b000 - 0x5c2b000
Memory: 7924360k/8388608k available (2459k kernel code, 201652k reserved, 1247k data, 196k init)
Calibrating delay using timer specific routine.. 5677.46 BogoMIPS (lpj=2838730)
Security Framework v1.0.0 initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 256
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
SMP alternatives: switching to UP code
ACPI: Core revision 20060707
Using local APIC timer interrupts.
result 4124540
Detected 4.124 MHz APIC timer.
SMP alternatives: switching to SMP code
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 5667.53 BogoMIPS (lpj=2833767)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 6144K
Intel(R) Xeon(R) CPU E5440 @ 2.83GHz stepping 08
Brought up 2 CPUs
testing NMI watchdog ... <4>WARNING: CPU#0: NMI appears to be stuck (0->0)!
time.c: Using 3.579545 MHz WALL PM GTOD PIT/TSC timer.
time.c: Detected 2833.008 MHz processor.
sizeof(vma)=176 bytes
sizeof(page)=56 bytes
sizeof(inode)=560 bytes
sizeof(dentry)=216 bytes
sizeof(ext3inode)=760 bytes
sizeof(buffer_head)=96 bytes
sizeof(skbuff)=240 bytes
migration_cost=221
checking if image is initramfs... it is
Freeing initrd memory: 3205k freed
NET: Registered protocol family 16
No dock devices found.
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge (0000:00)
PCI: Probing PCI hardware (bus 00)
PCI quirk: region 1000-103f claimed by PIIX4 ACPI
PCI quirk: region 1040-104f claimed by PIIX4 SMB
Boot video device is 0000:00:0f.0
ACPI: PCI Interrupt Routing Table http://\_SB_.PCI0._PRT
ACPI: PCI Interrupt Link (IRQs 3 4 5 6 7 9 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link (IRQs 3 4 5 6 7 *9 10 11 14 15)
ACPI: PCI Interrupt Link (IRQs 3 4 5 6 7 9 10 *11 14 15)
ACPI: PCI Interrupt Link (IRQs 3 4 5 6 7 9 10 11 14 15) *0, disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 12 devices
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
PCI-GART: No AMD northbridge found.
PCI: Bridge: 0000:00:01.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Setting latency timer of device 0000:00:01.0 to 64
NET: Registered protocol family 2
IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
Simple Boot Flag at 0x36 set to 0x80
audit: initializing netlink socket (disabled)
audit(1214048697.540:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
SELinux: Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key 8D87DC113F30B7
- User ID: Red Hat, Inc. (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Limiting direct PCI/PCI transfers.
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI: Processor (supports 8 throttling states)
ACPI: Processor (supports 8 throttling states)
ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present
ACPI: Getting cpuindex for acpiid 0x2
ACPI Exception (acpi_processor-0681): AE_NOT_FOUND, Processor Device is not present
ACPI: Getting cpuindex for acpiid 0x3
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
agpgart: Detected an Intel 440BX Chipset.
agpgart: AGP aperture is 256M @ 0x0
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
PIIX4: IDE controller at PCI slot 0000:00:07.1
PIIX4: chipset revision 1
PIIX4: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0x1050-0x1057, BIOS settings: hda:DMA, hdb:pio
Probing IDE interface ide0...
hda: VMware Virtual IDE CDROM Drive, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: PS/2 Controller at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
ACPI: (supports S0 S1 S4 S5)
Freeing unused kernel memory: 196k freed
Write protecting the kernel read-only data: 476k
input: AT Translated Set 2 keyboard as /class/input/input0
ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
USB Universal Host Controller Interface driver v3.0
SCSI subsystem initialized
Fusion MPT base driver 3.04.05
Copyright (c) 1999-2007 LSI Corporation
Fusion MPT SPI Host driver 3.04.05
GSI 16 sharing vector 0xA9 and IRQ 16
ACPI: PCI Interrupt 0000:00:10.0[A] -> GSI 17 (level, low) -> IRQ 169
mptbase: ioc0: Initiating bringup
ioc0: LSI53C1030 B0: Capabilities=
scsi0 : ioc0: LSI53C1030 B0, FwRev=00000000h, Ports=1, MaxQ=128, IRQ=169
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
target0:0:0: Beginning Domain Validation
target0:0:0: Domain Validation skipping write tests
target0:0:0: Ending Domain Validation
target0:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sda: 15872000 512-byte hdwr sectors (8126 MB)
sda: test WP failed, assume Write Enabled
sda: cache data unavailable
sda: assuming drive cache: write through
SCSI device sda: 15872000 512-byte hdwr sectors (8126 MB)
sda: test WP failed, assume Write Enabled
sda: cache data unavailable
sda: assuming drive cache: write through
sda: sda1 sda2
sd 0:0:0:0: Attached scsi disk sda
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
target0:0:1: Beginning Domain Validation
target0:0:1: Domain Validation skipping write tests
target0:0:1: Ending Domain Validation
target0:0:1: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sdb: 2097152 512-byte hdwr sectors (1074 MB)
sdb: test WP failed, assume Write Enabled
sdb: cache data unavailable
sdb: assuming drive cache: write through
SCSI device sdb: 2097152 512-byte hdwr sectors (1074 MB)
sdb: test WP failed, assume Write Enabled
sdb: cache data unavailable
sdb: assuming drive cache: write through
sdb: sdb1
sd 0:0:1:0: Attached scsi disk sdb
Vendor: HP Model: MSA2012i Rev: J210
Type: Direct-Access ANSI SCSI revision: 05
target0:0:2: Beginning Domain Validation
target0:0:2: Domain Validation skipping write tests
target0:0:2: Ending Domain Validation
target0:0:2: FAST-80 SCSI 80.0 MB/s DT (12.5 ns, offset 127)
SCSI device sdc: 97658880 512-byte hdwr sectors (50001 MB)
sdc: Write Protect is off
sdc: Mode Sense: 93 00 00 08
SCSI device sdc: drive cache: write back
SCSI device sdc: 97658880 512-byte hdwr sectors (50001 MB)
sdc: Write Protect is off
sdc: Mode Sense: 93 00 00 08
SCSI device sdc: drive cache: write back
sdc: sdc1
sd 0:0:2:0: Attached scsi disk sdc
Vendor: VMware Model: Virtual disk Rev: 1.0
Type: Direct-Access ANSI SCSI revision: 02
target0:0:3: Beginning Domain Validation
target0:0:3: Domain Validation skipping write tests
target0:0:3: Ending Domain Validation
target0:0:3: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sdd: 96468992 512-byte hdwr sectors (49392 MB)
sdd: test WP failed, assume Write Enabled
sdd: cache data unavailable
sdd: assuming drive cache: write through
SCSI device sdd: 96468992 512-byte hdwr sectors (49392 MB)
sdd: test WP failed, assume Write Enabled
sdd: cache data unavailable
sdd: assuming drive cache: write through
sdd: unknown partition table
sd 0:0:3:0: Attached scsi disk sdd
libata version 3.00 loaded.
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
input: ImPS/2 Generic Wheel Mouse as /class/input/input1
input: AT Translated Set 2 keyboard as /class/input/input2
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: Disabled at runtime.
SELinux: Unregistering netfilter hooks
audit(1214048703.838:2): selinux=0 auid=4294967295 ses=4294967295
hda: ATAPI 1X CD-ROM drive, 32kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
piix4_smbus 0000:00:07.3: Found 0000:00:07.3 device
piix4_smbus 0000:00:07.3: Host SMBus controller not enabled!
pcnet32.c:v1.32 18.Mar.2006 tsbogend@alpha.franken.de
GSI 17 sharing vector 0xB1 and IRQ 17
ACPI: PCI Interrupt 0000:00:11.0[A] -> GSI 18 (level, low) -> IRQ 177
pcnet32: PCnet/PCI II 79C970A at 0x1400, 00 50 56 b4 4c 34 assigned IRQ 177.
eth0: registered as PCnet/PCI II 79C970A
pcnet32: 1 cards_found.
input: PC Speaker as /class/input/input3
parport: PnPBIOS parport detected.
parport0: PC-style at 0x378, irq 7
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
sd 0:0:0:0: Attached scsi generic sg0 type 0
sd 0:0:1:0: Attached scsi generic sg1 type 0
sd 0:0:2:0: Attached scsi generic sg2 type 0
sd 0:0:3:0: Attached scsi generic sg3 type 0
lp0: using parport0 (interrupt-driven).
lp0: console ready
ACPI: AC Adapter (on-line)
ACPI: Power Button (FF)
ibm_acpi: ec object not found
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
device-mapper: multipath: version 1.0.5 loaded
EXT3 FS on dm-0, internal journal
kjournald starting. Commit interval 5 seconds
EXT3 FS on sdb1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on dm-1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 1638392k swap on /dev/local/swap. Priority:-1 extents:1 across:1638392k
IA-32 Microcode Update Driver: v1.14a
eth0: link up
Bluetooth: Core ver 2.10
NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP ver 2.8
Bluetooth: L2CAP socket layer initialized
Bluetooth: RFCOMM socket layer initialized
Bluetooth: RFCOMM TTY layer initialized
Bluetooth: RFCOMM ver 1.8
Bluetooth: HIDP (Human Interface Emulation) ver 1.1
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
eth0: no IPv6 routers present
SCSI device sdd: 96468992 512-byte hdwr sectors (49392 MB)
sdd: test WP failed, assume Write Enabled
sdd: cache data unavailable
sdd: assuming drive cache: write through
sdd: sdd1
SCSI device sdd: 96468992 512-byte hdwr sectors (49392 MB)
sdd: test WP failed, assume Write Enabled
sdd: cache data unavailable
sdd: assuming drive cache: write through
sdd: sdd1
kjournald starting. Commit interval 5 seconds
EXT3 FS on sdd1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on sdd1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
I also constantly see logs in /var/log/vmkernel like this:
Jun 21 11:48:04 esx1 vmkernel: 0:21:55:02.428 cpu0:1077)iSCSI: session 0x352180a0 recv_cmd 0x3f64bc00, cdb 0x0, status 0x2, response 0x0, senselen 42, key 06, ASC/ASCQ 29/03, itt 2764 task 0x352019a0 to (0 0 1 5), san01.mydomain.com
Jun 21 11:48:04 esx1 vmkernel: iSCSI: Sense 70000600 0000000a 0000000
same for /var/log/vmkwarning:
Jun 21 11:31:45 esx1 vmkernel: 0:21:38:43.666 cpu1:1040)WARNING: ScsiPath: 3187: Plugin 'legacyMP' had an error (Not support
ed) while claiming path 'vmhba32:C0:T2:L0'.Skipping the path.
Jun 21 11:31:45 esx1 vmkernel: 0:21:38:43.666 cpu1:1040)WARNING: SCSI: 279: SCSI device type 0xd is not supported. Cannot cr
eate target vmhba32:3:0
Jun 21 11:31:45 esx1 vmkernel: 0:21:38:43.666 cpu1:1040)WARNING: SCSI: 1249: LegacyMP Plugin could not claim path: vmhba32:
3:0. Not supported
Jun 21 11:31:45 esx1 vmkernel: 0:21:38:43.666 cpu1:1040)WARNING: ScsiPath: 3187: Plugin 'legacyMP' had an error (Not support
ed) while claiming path 'vmhba32:C0:T3:L0'.Skipping the path.
Now, as far as multipathing is concerned, the SAN is an active/active array but only configured on 1 NIC at the moment because it's a preliminary setup before we move all servers to the datacenter. Could this be the problem? I filled in all 4 IP's for the SAN in the dynamic discovery tab of the software iSCSI initator and it should be able to find them all, as the SAN is fully patched to the switch. (this will of course be 2 switches in the final configuration).
Koen