ZFS vdev perf

Having this info 2 years ago would have made a big difference!!!   not true :( we used multiples vdevs

http://en.wikipedia.org/wiki/ZFS

IOPS performance of a ZFS storage pool can suffer if the ZFS raid is not appropriately configured. This applies to all types of RAID, in one way or another. If the zpool consists of only one group of disks configured as, say, raidz2 - then the IOPS performance will be that of a single disk. This means, to get high IOPS performance, the zpool should consist of several vdevs, because one vdev gives the IOPS of a single disk. However, there are ways to mitigate this IOPS performance problem, for instance add SSDs as L2ARC cache — which can boost IOPS into 100.000s http://blogs.sun.com/brendan/entry/a_quarter_million_nfs_iops . In short, a zpool should consist of several groups of vdevs, each vdev consisting of 8-12 disks. It is not recommended to create a zpool with a single large vdev, say 20 disks, because IOPS performance will be that of a single disk, which also means that resilver time will be very long (possibly weeks with future large drives).