Different disk size on different controllers

Garter · #1 (**permalink**) 07-16-2011, 01:40 PM

Hi folks,
I just recently build myself a NAS with Oracle Solaris 11 Express.

MOBO: Asus M4N68T-M LE V2 nForce 630a AM3
PCI-e Controller: 2x Exsys EX-3503 with Sil3132 Chipset (latest FW)
HDD: 7x 2000GB Samsung EcoGreen F4 HD204UI 32MB 3.5" (latest FW)
OS: SunOS zaruba 5.11 snv_151a i86pc i386 i86pc Solaris

Since I'm totally new to Solaris I grabed some guides and started playing with creating some zpools. At first there weren't any problems but after some zpool destroy tank the nex zpool create tank ... produced following error message:

Code:

# format
Searching for disks...
Failed to inquiry this logical diskdone


AVAILABLE DISK SELECTIONS:
       0. c7d0 < cyl 19454 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@8/ide@0/cmdk@0,0
       1. c8d0 <SAMSUNG-S2H7J90B50963-0001-1.82TB>
          /pci@0,0/pci-ide@8/ide@1/cmdk@0,0
       2. c9d0 <SAMSUNG-S2H7J90B51077-0001-1.82TB>
          /pci@0,0/pci-ide@8,1/ide@0/cmdk@0,0
       3. c10d0 <SAMSUNG-S2H7J90B50962-0001-1.82TB>
          /pci@0,0/pci-ide@8,1/ide@1/cmdk@0,0
       4. c13t0d0 <ATA-SAMSUNG HD204UI-0001-1.82TB>
          /pci@0,0/pci10de,3e8@9/pci1095,3132@0/disk@0,0
       5. c13t1d0 <ATA-SAMSUNG HD204UI-0001-1.82TB>
          /pci@0,0/pci10de,3e8@9/pci1095,3132@0/disk@1,0
       6. c14t0d0 <ATA-SAMSUNG HD204UI-0001-1.82TB>
          /pci@0,0/pci10de,3e9@b/pci1095,3132@0/disk@0,0
       7. c14t1d0 <ATA-SAMSUNG HD204UI-0001-1.82TB>
          /pci@0,0/pci10de,3e9@b/pci1095,3132@0/disk@1,0
Specify disk (enter its number): ^C
# zpool create tank mirror c8d0 c9d0 mirror c10d0 c13t0d0 mirror c13t1d0 c14t0d0 spare c14t1d0
invalid vdev specification
use '-f' to override the following errors:
mirror contains devices of different sizes

That totally blew my mind because it wasn't the case before. A quick iostat -E shows that's indeed true.

Code:

~# iostat -E
cmdk0     Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Model: SAMSUNG HD161GJ Revision:  Serial No: S14DJ90B511278  Size: 160.04GB <160039305216 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0
cmdk1     Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Model: SAMSUNG HD204UI Revision:  Serial No: S2H7J90B509630  Size: 2000.37GB <2000371580928 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0
cmdk2     Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Model: SAMSUNG HD204UI Revision:  Serial No: S2H7J90B510778  Size: 2000.37GB <2000371580928 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0
cmdk3     Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Model: SAMSUNG HD204UI Revision:  Serial No: S2H7J90B509629  Size: 2000.37GB <2000371580928 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0
sd1       Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: SAMSUNG HD204UI  Revision: 0001 Serial No:
Size: 2000.40GB <2000398934016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 14 Predictive Failure Analysis: 0
sd2       Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: SAMSUNG HD204UI  Revision: 0001 Serial No:
Size: 2000.40GB <2000398934016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 14 Predictive Failure Analysis: 0
sd3       Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: SAMSUNG HD204UI  Revision: 0001 Serial No:
Size: 2000.40GB <2000398934016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 13 Predictive Failure Analysis: 0
sd4       Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: SAMSUNG HD204UI  Revision: 0001 Serial No:
Size: 2000.40GB <2000398934016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 14 Predictive Failure Analysis: 0

The strange thing though is when I plug the apparently smaller disks from the MOBO controllers to the add-in controllers iostat -E shows 2000398934016 bytes instead of 2000371580928 bytes.
I don't know which is the correct value but like I said before this behavior didn't occur in the beginning. Also what's with the illegal requests

I also tried to change/restore the EFI label but to no avail.

Code:

~# format -e
Searching for disks...
Failed to inquiry this logical diskdone


AVAILABLE DISK SELECTIONS:
       0. c7d0 < cyl 19454 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@8/ide@0/cmdk@0,0
       1. c8d0 <SAMSUNG-S2H7J90B50963-0001-1.82TB>
          /pci@0,0/pci-ide@8/ide@1/cmdk@0,0
       2. c9d0 <SAMSUNG-S2H7J90B51077-0001-1.82TB>
          /pci@0,0/pci-ide@8,1/ide@0/cmdk@0,0
       3. c10d0 <SAMSUNG-S2H7J90B50962-0001-1.82TB>
          /pci@0,0/pci-ide@8,1/ide@1/cmdk@0,0
       4. c13t0d0 <ATA-SAMSUNG HD204UI-0001-1.82TB>
          /pci@0,0/pci10de,3e8@9/pci1095,3132@0/disk@0,0
       5. c13t1d0 <ATA-SAMSUNG HD204UI-0001-1.82TB>
          /pci@0,0/pci10de,3e8@9/pci1095,3132@0/disk@1,0
       6. c14t0d0 <ATA-SAMSUNG HD204UI-0001-1.82TB>
          /pci@0,0/pci10de,3e9@b/pci1095,3132@0/disk@0,0
       7. c14t1d0 <ATA-SAMSUNG HD204UI-0001-1.82TB>
          /pci@0,0/pci10de,3e9@b/pci1095,3132@0/disk@1,0
Specify disk (enter its number): 4
selecting c13t0d0
[disk formatted]


FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        inquiry    - show vendor, product and revision
        scsi       - independent SCSI mode selects
        cache      - enable, disable or query SCSI disk cache
        volname    - set 8-character volume name
        !<cmd>     - execute <cmd>, then return
        quit
format> verify

Volume name = <        >
ascii name  = <ATA-SAMSUNG HD204UI-0001-1.82TB>
bytes/sector    =  512
sectors = 3907029167
accessible sectors = 3907029134
Part      Tag    Flag     First Sector          Size          Last Sector
  0 unassigned    wm                 0            0                0
  1 unassigned    wm                 0            0                0
  2 unassigned    wm                 0            0                0
  3 unassigned    wm                 0            0                0
  4 unassigned    wm                 0            0                0
  5 unassigned    wm                 0            0                0
  6 unassigned    wm                 0            0                0
  7 unassigned    wm                 0            0                0
  8 unassigned    wm                 0            0                0

format> label
[0] SMI Label
[1] EFI Label
Specify Label type[1]:
Ready to label disk, continue? y

no reserved partition found

format>

Same for all four disks on the add-in controllers

But that reminds me. I read when I zpool destroy data is written to the disks so the OS knows the zpool is no longer active. I thought that might be where the missing space was gone, fired up gparted and actually found and deleted two partitions , where one partition was like 8MB, on each disk. In retrospective that might have not been the smartest move ...

Since I still have the option and I don't know how hard it is to fix this mess. Would a reinstall fix my problem?

So long story short. My identical devices (they had the exact same Byte count before) are not identical anymore and I'd like to know why that is and how I can fix that. True it's just ~27MB missing so the whole pool would lose some 100MB but I still do wanna know what happened here.

Looking forward to your replies.
Best regards
Garter