ZFS/ZPOOL problem after disk exchange

Hi all,

one of the disks in an X4500 was about to die and we replaced it after
taking it zfs-offline/unconfigure/disconnect. Exchange went fine connect
and confiugre works fine, however zpool online pool c1t6d0 does not return.

zpool status shows only the header (including that the system is running
in "degraded") mode but also does not return after that in reasonable
time (i.e. 30 minutes). Also zfs list does also not return.

Any idea what's wrong?

Thanks

Carsten

Addendum:

The box stalled and after a reboot it's more responsive:

# cfgadm |grep c1t6
sata1/6::dsk/c1t6d0 disk connected configured ok

# zpool online atlashome c1t6d0
Bringing device c1t6d0 online

but:

zpool status
pool: atlashome
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas
exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver completed with 0 errors on Mon Aug 25 19:49:13 2008
config:


[...]

c1t6d0 UNAVAIL 0 0 0 cannot open

Still no idea here.

Carsten

Carsten Aulbert wrote:
> Addendum:
>
> The box stalled and after a reboot it's more responsive:
>
> # cfgadm |grep c1t6
> sata1/6::dsk/c1t6d0 disk connected configured ok
>
> # zpool online atlashome c1t6d0
> Bringing device c1t6d0 online
>
> but:
>
> zpool status
> pool: atlashome
> state: DEGRADED
> status: One or more devices could not be opened. Sufficient replicas
> exist for
> the pool to continue functioning in a degraded state.
> action: Attach the missing device and online it using 'zpool online'.
> see: http://www.sun.com/msg/ZFS-8000-D3

what does a complete zpool status -x show?

> c1t6d0 UNAVAIL 0 0 0 cannot open

what does ' echo |format' show?

Cydrome Leader wrote:
> what does a complete zpool status -x show?
>

# zpool status -x
pool: atlashome
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas
exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: resilver completed with 0 errors on Mon Aug 25 19:49:13 2008
config:

NAME STATE READ WRITE CKSUM
atlashome DEGRADED 0 0 0
raidz2 ONLINE 0 0 0
c0t0d0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
c6t0d0 ONLINE 0 0 0
c7t0d0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
c5t1d0 ONLINE 0 0 0
c6t1d0 ONLINE 0 0 0
c7t1d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c4t2d0 ONLINE 0 0 0
c5t2d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c6t2d0 ONLINE 0 0 0
c7t2d0 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
c6t3d0 ONLINE 0 0 0
c7t3d0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
c4t4d0 ONLINE 0 0 0
c6t4d0 ONLINE 0 0 0
c7t4d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
raidz2 DEGRADED 0 0 0
c4t5d0 ONLINE 0 0 0
c5t5d0 ONLINE 0 0 0
c6t5d0 ONLINE 0 0 0
c7t5d0 ONLINE 0 0 0
c0t6d0 ONLINE 0 0 0
c1t6d0 UNAVAIL 0 0 0 cannot open
c4t6d0 ONLINE 0 0 0
c5t6d0 ONLINE 0 0 0
c6t6d0 ONLINE 0 0 0
c7t6d0 ONLINE 0 0 0
c0t7d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
c4t7d0 ONLINE 0 0 0
c5t7d0 ONLINE 0 0 0
c6t7d0 ONLINE 0 0 0
c7t7d0 ONLINE 0 0 0

errors: No known data errors

>> c1t6d0 UNAVAIL 0 0 0 cannot open
>
> what does ' echo |format' show?
>

13. c1t5d0
/pci-at-0,0/pci1022,7458-at-2/pci11ab,11ab-at-1/disk-at-5,0
14. c1t6d0
/pci-at-0,0/pci1022,7458-at-2/pci11ab,11ab-at-1/disk-at-6,0
15. c1t7d0
/pci-at-0,0/pci1022,7458-at-2/pci11ab,11ab-at-1/disk-at-7,0
Sh...

thanks, I guess this needs an EFI label, right?

Carsten

Hi,

Franco wrote:
>
> Where is your mirror ?

Which mirror? The boot disks are c5t0d0 and c5t4d0 those are mirrored
but not under ZFS (not yet supported under 10u5 IIRC).

>
> Looks like you have a faulty disk..
>
>> c1t6d0 UNAVAIL 0 0 0 cannot open
>
> What does iostat -En | grep c1t6d0 show ?

iostat -En | grep c1t6d0
c1t6d0 Soft Errors: 696 Hard Errors: 0 Transport Errors: 0

Cool, that's a brand new disk from Sun. But most of the other drives
have about the same number of soft errors:

# iostat -En|grep Soft
c2t0d0 Soft Errors: 1 Hard Errors: 2 Transport Errors: 0
c3t0d0 Soft Errors: 4 Hard Errors: 0 Transport Errors: 0
c6t0d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c5t0d0 Soft Errors: 82 Hard Errors: 0 Transport Errors: 0
c4t0d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c7t0d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c0t0d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c1t0d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c7t1d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c4t1d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c0t1d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c1t1d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c6t1d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c7t2d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c4t2d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c5t1d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c0t2d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c1t2d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c6t2d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c7t3d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c4t3d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c5t2d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c0t3d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c7t4d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c1t3d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c6t3d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c4t4d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c0t4d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c5t3d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c7t5d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c1t4d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c6t4d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c5t4d0 Soft Errors: 82 Hard Errors: 0 Transport Errors: 0
c4t5d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c0t5d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c7t6d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c1t5d0 Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
c6t5d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c4t6d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c0t6d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c7t7d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c6t6d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c1t6d0 Soft Errors: 696 Hard Errors: 0 Transport Errors: 0
c5t5d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c4t7d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c0t7d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c6t7d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c1t7d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c5t6d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
c5t7d0 Soft Errors: 487 Hard Errors: 0 Transport Errors: 0

So how to proceed? The latest resilvering was "successful", however the
disk is still not in the zpool (same as before).

Cheers

Carsten

Hi (it seemed my last post vanished)

Franco wrote:
>
> Where is your mirror ?

What mirror? Our setup has only a mirror for booting, i.e. c5t0d0 and
c5t4d0.

>
> Looks like you have a faulty disk..
>
>> c1t6d0 UNAVAIL 0 0 0 cannot open
>

Great, brand new disk from Sun.

> What does iostat -En | grep c1t6d0 show ?

# iostat -En | grep c1t6d0
c1t6d0 Soft Errors: 698 Hard Errors: 0 Transport Errors: 0

Does not look good for a new drive, although many other drives show soft
errors:


s08:~# iostat -En | grep Soft
c2t0d0 Soft Errors: 1 Hard Errors: 2 Transport Errors: 0
c3t0d0 Soft Errors: 4 Hard Errors: 0 Transport Errors: 0
c6t0d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c5t0d0 Soft Errors: 84 Hard Errors: 0 Transport Errors: 0
c4t0d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c7t0d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c0t0d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c1t0d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c7t1d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c4t1d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c0t1d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c1t1d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c6t1d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c7t2d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c4t2d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c5t1d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c0t2d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c1t2d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c6t2d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c7t3d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c4t3d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c5t2d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c0t3d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c7t4d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c1t3d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c6t3d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c4t4d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c0t4d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c5t3d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c7t5d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c1t4d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c6t4d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c5t4d0 Soft Errors: 84 Hard Errors: 0 Transport Errors: 0
c4t5d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c0t5d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c7t6d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c1t5d0 Soft Errors: 62 Hard Errors: 0 Transport Errors: 0
c6t5d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c4t6d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c0t6d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c7t7d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c6t6d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c1t6d0 Soft Errors: 698 Hard Errors: 0 Transport Errors: 0
c5t5d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c4t7d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c0t7d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c6t7d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c1t7d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c5t6d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0
c5t7d0 Soft Errors: 489 Hard Errors: 0 Transport Errors: 0

Well, anyone knows how to proceed?

Cheers

Carsten

On Aug 26, 7:01*am, Carsten Aulbert wrote:
> Hi,
>
> Franco wrote:
>
> > Where is your mirror ?
>
> Which mirror? The boot disks are c5t0d0 and c5t4d0 those are mirrored
> but not under ZFS (not yet supported under 10u5 IIRC).
>
>
>
> > Looks like you have a faulty disk..
>
> >> * * * * * * c1t6d0 *UNAVAIL * * *0 * * 0 * * 0 *cannot open
>
> > What does iostat -En | grep c1t6d0 show ?
>
> iostat -En | grep c1t6d0
> c1t6d0 * * * * * Soft Errors: 696 Hard Errors: 0 Transport Errors: 0
>
> Cool, that's a brand new disk from Sun. But most of the other drives
> have about the same number of soft errors:
>
> # iostat -En|grep Soft
> c2t0d0 * * * * * Soft Errors: 1 Hard Errors: 2 Transport Errors: 0
> c3t0d0 * * * * * Soft Errors: 4 Hard Errors: 0 Transport Errors: 0
> c6t0d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c5t0d0 * * * * * Soft Errors: 82 Hard Errors: 0 Transport Errors: 0
> c4t0d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c7t0d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c0t0d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c1t0d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c7t1d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c4t1d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c0t1d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c1t1d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c6t1d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c7t2d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c4t2d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c5t1d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c0t2d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c1t2d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c6t2d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c7t3d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c4t3d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c5t2d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c0t3d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c7t4d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c1t3d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c6t3d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c4t4d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c0t4d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c5t3d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c7t5d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c1t4d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c6t4d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c5t4d0 * * * * * Soft Errors: 82 Hard Errors: 0 Transport Errors: 0
> c4t5d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c0t5d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c7t6d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c1t5d0 * * * * * Soft Errors: 60 Hard Errors: 0 Transport Errors: 0
> c6t5d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c4t6d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c0t6d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c7t7d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c6t6d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c1t6d0 * * * * * Soft Errors: 696 Hard Errors: 0 Transport Errors: 0
> c5t5d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c4t7d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c0t7d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c6t7d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c1t7d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c5t6d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
> c5t7d0 * * * * * Soft Errors: 487 Hard Errors: 0 Transport Errors: 0
>
> So how to proceed? The latest resilvering was "successful", however the
> disk is still not in the zpool (same as before).
>
> Cheers
>
> Carsten

Why did you not create a mirror on your zpool ?

The setup you have at the moment is throwing out loads of errors on
the disks.

If possible, can you reboot the server ? This will clear the iostat
output.

I'd advise replacing c1t6d0, although you are going to lose data
because you have no viable mirrors in your zpool.

Make sure you do a full backup before replacing the drive.

Regards,

Franco wrote:

> Why did you not create a mirror on your zpool ?
>

Why should I? I want the boxes to be good bandwidth and max capacity.
Entering mirrored set-ups I would waste a lot of disk space.

> The setup you have at the moment is throwing out loads of errors on
> the disks.
>
> If possible, can you reboot the server ? This will clear the iostat
> output.
>
The box was rebooted about 18hrs ago.

> I'd advise replacing c1t6d0, although you are going to lose data
> because you have no viable mirrors in your zpool.
>

Why's that? This part of the disk is part of a raidz2, thus two disks
redundancy. A mirror is not the only way to make data "safe".

> Make sure you do a full backup before replacing the drive.

Yeah already two backups taken.

Thanks

Carsten

His zpool is a raidz2.

Carsten Aulbert wrote:
> Cydrome Leader wrote:
>> what does a complete zpool status -x show?
>>
>
> # zpool status -x
> pool: atlashome
> state: DEGRADED
> status: One or more devices could not be opened. Sufficient replicas
> exist for
> the pool to continue functioning in a degraded state.
> action: Attach the missing device and online it using 'zpool online'.
> see: http://www.sun.com/msg/ZFS-8000-D3
> scrub: resilver completed with 0 errors on Mon Aug 25 19:49:13 2008
> config:
>
> NAME STATE READ WRITE CKSUM
> atlashome DEGRADED 0 0 0
> raidz2 ONLINE 0 0 0
> c0t0d0 ONLINE 0 0 0
> c1t0d0 ONLINE 0 0 0
> c4t0d0 ONLINE 0 0 0
> c6t0d0 ONLINE 0 0 0
> c7t0d0 ONLINE 0 0 0
> c0t1d0 ONLINE 0 0 0
> c1t1d0 ONLINE 0 0 0
> c4t1d0 ONLINE 0 0 0
> c5t1d0 ONLINE 0 0 0
> c6t1d0 ONLINE 0 0 0
> c7t1d0 ONLINE 0 0 0
> c0t2d0 ONLINE 0 0 0
> c1t2d0 ONLINE 0 0 0
> c4t2d0 ONLINE 0 0 0
> c5t2d0 ONLINE 0 0 0
> raidz2 ONLINE 0 0 0
> c6t2d0 ONLINE 0 0 0
> c7t2d0 ONLINE 0 0 0
> c0t3d0 ONLINE 0 0 0
> c1t3d0 ONLINE 0 0 0
> c4t3d0 ONLINE 0 0 0
> c5t3d0 ONLINE 0 0 0
> c6t3d0 ONLINE 0 0 0
> c7t3d0 ONLINE 0 0 0
> c0t4d0 ONLINE 0 0 0
> c1t4d0 ONLINE 0 0 0
> c4t4d0 ONLINE 0 0 0
> c6t4d0 ONLINE 0 0 0
> c7t4d0 ONLINE 0 0 0
> c0t5d0 ONLINE 0 0 0
> c1t5d0 ONLINE 0 0 0
> raidz2 DEGRADED 0 0 0
> c4t5d0 ONLINE 0 0 0
> c5t5d0 ONLINE 0 0 0
> c6t5d0 ONLINE 0 0 0
> c7t5d0 ONLINE 0 0 0
> c0t6d0 ONLINE 0 0 0
> c1t6d0 UNAVAIL 0 0 0 cannot open
> c4t6d0 ONLINE 0 0 0
> c5t6d0 ONLINE 0 0 0
> c6t6d0 ONLINE 0 0 0
> c7t6d0 ONLINE 0 0 0
> c0t7d0 ONLINE 0 0 0
> c1t7d0 ONLINE 0 0 0
> c4t7d0 ONLINE 0 0 0
> c5t7d0 ONLINE 0 0 0
> c6t7d0 ONLINE 0 0 0
> c7t7d0 ONLINE 0 0 0
>
> errors: No known data errors
>
>>> c1t6d0 UNAVAIL 0 0 0 cannot open
>>
>> what does ' echo |format' show?
>>
>
> 13. c1t5d0
> /pci-at-0,0/pci1022,7458-at-2/pci11ab,11ab-at-1/disk-at-5,0
> 14. c1t6d0
> /pci-at-0,0/pci1022,7458-at-2/pci11ab,11ab-at-1/disk-at-6,0
> 15. c1t7d0
> /pci-at-0,0/pci1022,7458-at-2/pci11ab,11ab-at-1/disk-at-7,0
> Sh...
>
> thanks, I guess this needs an EFI label, right?
>
> Carsten

it looks like something is wrong with that disk, but it's also interesting
you're getting so many errors on your disks anyways.

doesa prtvtoc on that disk that "cannot open" show anything strange
compared to other drives?

it may have some junk labels on it that need to be removed.