OmniOS and getting SmartmonTools to Work

I spent a couple days off and on trying to get smartmontools to work on OmniOS.  I saw some conflicting info on whether it should work out of the box, and for me at least it did not.  Below is what I did to get it to work

Don’t bother building from scratch.  Add the below repository and install the package.

pkg set-publisher -O http://scott.mathematik.uni-ulm.de/release uulm.mawi
pkg search -pr smartmontools
pkg install smartmontools
root@OmniOS:/root# pkg info -r smartmontools
          Name: system/storage/smartmontools
       Summary: Control and monitor storage systems using SMART
         State: Installed
     Publisher: uulm.mawi
       Version: 6.3
        Branch: 0.151012
Packaging Date: Mon Sep 29 13:22:53 2014
          Size: 1.83 MB
          FMRI: pkg://uulm.mawi/system/storage/smartmontools@6.3-0.151012:20140929T132253Z

The smartmontools file in /etc/default/smartmontools might have been created automatically, but I’m not sure if this was left over from previous attempts.  Either way below is what you need.

# Defaults for smartmontools initscript (/etc/init.d/smartmontools)
# This is a POSIX shell fragment

# List of devices you want to explicitly enable S.M.A.R.T. for
# Not needed (and not recommended) if the device is monitored by smartd
#enable_smart="/dev/hda /dev/hdb"

# uncomment to start smartd on system startup
start_smartd=yes

# uncomment to pass additional options to smartd on startup
smartd_opts="--interval=1800"

Then add your disks to /etc/smartd.conf.  For me it was like below, yours will differ.

/dev/rdsk/c1t5000CCA232C02D87d0 -a -d sat,12
/dev/rdsk/c1t5000CCA232C0D31Bd0 -a -d sat,12 
/dev/rdsk/c1t5000CCA232C0EA80d0 -a -d sat,12 
/dev/rdsk/c1t5000CCA232C1543Cd0 -a -d sat,12 
/dev/rdsk/c1t5000CCA232C0AD56d0 -a -d sat,12 
/dev/rdsk/c1t5000CCA232C0BBA6d0 -a -d sat,12 
/dev/rdsk/c1t5000039FF4CF3EA6d0 -a -d sat,12 
/dev/rdsk/c1t5000039FF4E58676d0 -a -d sat,12 
/dev/rdsk/c3t5d0 -a -d sat,12

If everything works you will see below.  I do have an issue where one of my disks is repeatedly parking loudly.  I’ll need to do some research to see why.  It is good to know that all my disks are healthy.

capture

OmniOS and a Misunderstanding

I decided to go with OmniOS as it seems to be the OS everyone recommends for ZFS and NappIt and it works with the 10G networking.  Installation went smooth and without issues

Unfortunately, that was the only thing that went well.  No configuration would allow me to get a 10G write above 180-190MB/s which is just not acceptable.  I would have expected a greater than 60% increase in speed.

capture4

Then it dawned on me.  I’m being limited by the speed of the media I’m using to copy my files from (a USB 3.0 Drive).  In theory a SATA Interfaced drive is only going to hit 160-190MB/s.

So then I copied a file locally and witnessed the same speeds, and copied that file from a Local Drive on a VM to the mounted SMB share, and boom there was my speeds, topping out at 400+MB/s

NAS Hiccup

So as I was starting to do the NAS Build and I noticed an issue.  I had ordered the wrong 10G Nic for my Motherboard.   My motherboard uses 10G RJ45 and I ordered a NIC that uses SFPs. Also I had to add a larger Sata DOM as OmniOS and NappIt need the size.  I decided to add a SLOG drive even though it isn’t needed for just streaming, but if I decide to do ISCSI or NFS for VMware it would be useful.

The final build is below.   This will not inhibit me from continuing the build but will not allow me to do 10G Tests.

6X HGST 6TB He Drives

64GB DDR4 ECC

SuperMicro X10SDV-4C-TLN4F Motherboard

LSI 9211-8i HBA

Supermicro SSD-DM064-PHI 64GB SATA DOM (Disk on Module)

Fractal Node 804 Case

Intel S3710 200GB SSD (SLOG)

Intel X540-T2 10G Controller

Upcoming NAS Server

dsc_0002

I’ve finally outgrown my Synology 416J NAS.  It’s a well built device, but you are completely limited as to expanding or improving the performance.  Currently an 8TB Seagate USB3 Device has the same or better CIFS speed than my NAS.

I plan on thoroughly benchmarking my current setup in SHR, RAID10 and RAID5.  Currently I have 2X3TB and 2X6TB in the Synology NAS and in both cases I’m maxing out the Gigabit Ethernet.

I’ve been investigating using FreeNas/OmniOS/Napp-It as a storage backend for Plex, Windows Backups and general file storage.

Here is a tease of the hardware

4X HGST 6TB He Drives

64GB DDR4 ECC

SuperMicro X10SDV-4C-TLN4F Motherboard

LSI 9211-8i HBA

Supermicro SSD-DM016-PHI 16GB SATA DOM (Disk on Module)

Fractal Node 304 Case

OCZ RD400/400A M.2 SSD

Intel X520-DA2 10G Network Card for Direct Connect to Plex VM

I’m still debating about adding another 12TB of HGST 6TB Drives.   I’m also trying to figure out a workflow of how to benchmark the current NAS Device as I’d like to show myself that what I’ve built is better than what I have. Also for the ZFS, I’m unsure of my ZIL/SLOG/L2ARC configuration.  I’ll take it as a learning experience and see what would work best for performance.