How to Install and Configure Zabbix

One of the issues I have with Observium is how limited the Device Discovery is.  Not all SNMP devices are recognized, and there is no way to customize the system with new Device MIB Files.  Zabbix was recommended to me by a couple of former coworkers, so I decided to give it a shot.

First I built a standard CentOS 7 VM and ran

yum update

 

Next I installed the Zabbix Repository RPM

[root@zabbix ~]# rpm -ivh http://repo.zabbix.com/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm
Retrieving http://repo.zabbix.com/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm
warning: /var/tmp/rpm-tmp.434W6N: Header V4 DSA/SHA1 Signature, key ID 79ea5ed4: NOKEY
Preparing… ################################# [100%]
Updating / installing…
1:zabbix-release-3.0-1.el7 ################################# [100%]

The next step is to install the Zabbix Server and Web Frontend with MySQL.

yum install zabbix-server-mysql zabbix-web-mysql

Next we install the zabbix agent in order to allow the server to monitor itself

yum install zabbix-agent

Next we need to install MySQL

yum install mariadb-server

Next we enable and start the service

systemctl enable mariadb

systemctl start mariadb

Now we set the root password for the database by running

mysql_secure_installation

You will see a handful of questions, and you should evaluate them individually.  My responses are not the most secure but for a demo they are fine to use.

[root@zabbix ~]# mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we’ll need the current
password for the root user. If you’ve just installed MariaDB, and
you haven’t set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on…

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
… Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y
… Success!

Normally, root should only be allowed to connect from ‘localhost’. This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
… skipping.

By default, MariaDB comes with a database named ‘test’ that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
– Dropping test database…
… Success!
– Removing privileges on test database…
… Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
… Success!

Cleaning up…

All done! If you’ve completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

We login to the database as the root user and create the Zabbix Database

mysql -uroot -p

From the mysql prompt enter the following to create the database

create database zabbix character set utf8;

Then we grant the zabbix server rights to the database.  You can change “Changeme123” to whatever password you wish.

grant all privileges on zabbix.* to zabbix@localhost identified by ‘Changeme123‘;

Finally apply the new permissions and exit the database prompt

flush privileges;

quit;

Next we need to import the schema and data into the Database.  It will prompt you for a password, of which the zabbix password set above can be used.

zcat /usr/share/doc/zabbix-server-mysql-3.0.9/create.sql.gz | mysql -u zabbix -p zabbix

Open the following configuration file and go to line 115

vim /etc/zabbix/zabbix_server.conf

Your output should look like below after you have made the recquired changes

### Option: DBPassword
# Database password. Ignored for SQLite.
# Comment this line if no password is used.
#
# Mandatory: no
# Default:
DBPassword=Changeme123

Next we update the system for the special PHP server Settings that Zabbix needs and open the following file.

vim /etc/httpd/conf.d/zabbix.conf

Go to line 19 and change the PHP value to be your timezone.  You can determine your closet timezone here.

My file looks like below

#
# Zabbix monitoring system php web frontend
#

Alias /zabbix /usr/share/zabbix

<Directory “/usr/share/zabbix”>
Options FollowSymLinks
AllowOverride None
Require all granted

<IfModule mod_php5.c>
php_value max_execution_time 300
php_value memory_limit 128M
php_value post_max_size 16M
php_value upload_max_filesize 2M
php_value max_input_time 300
php_value always_populate_raw_post_data -1
php_value date.timezone America/New_York
</IfModule>
</Directory>

<Directory “/usr/share/zabbix/conf”>
Require all denied
</Directory>

<Directory “/usr/share/zabbix/app”>
Require all denied
</Directory>

<Directory “/usr/share/zabbix/include”>
Require all denied
</Directory>

<Directory “/usr/share/zabbix/local”>
Require all denied
</Directory>

Next we go to the host specific httpd configuration file.

 vim /etc/httpd/conf/httpd.conf

Go to line 95 and change it from

#ServerName http://www.example.com:80

to

ServerName <YOURIPADDRESS>:80

Next we restart httpd

systemctl restart httpd

Before we start the Zabbix Server you need to give the Zabbix User rights to certain folders.

chown -R zabbix:zabbix /var/log/zabbix
chown -R zabbix:zabbix /var/run/zabbix
chmod -R 775 /var/log/zabbix/
chmod -R 775 /var/run/zabbix/

Next we execute the following and let th zabbix_agentd what configuration file to use

/usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf

Next we disable SELINUX

setenforce 0

Now we need to allow HTTP traffic for FirewallD

firewall-cmd –zone=public –add-service=http –permanent

firewall-cmd –reload

Now we start the Zabbix Server

systemctl start zabbix-server

Finally check the zabbix-server status

systemctl status zabbix-server

The output should be similar to below

[root@zabbix ~]# systemctl status zabbix-server
● zabbix-server.service – Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2017-06-04 13:09:21 EDT; 39s ago
Process: 7480 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 7482 (zabbix_server)
CGroup: /system.slice/zabbix-server.service
├─7482 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
├─7487 /usr/sbin/zabbix_server: configuration syncer [waiting 60 sec for processes]
├─7488 /usr/sbin/zabbix_server: db watchdog [synced alerts config in 0.000800 sec, idle 60 sec]
├─7489 /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000005 sec, idle 5 sec]
├─7491 /usr/sbin/zabbix_server: poller #2 [got 0 values in 0.000005 sec, idle 5 sec]
├─7492 /usr/sbin/zabbix_server: poller #3 [got 0 values in 0.000005 sec, idle 5 sec]
├─7493 /usr/sbin/zabbix_server: poller #4 [got 0 values in 0.000006 sec, idle 5 sec]
├─7494 /usr/sbin/zabbix_server: poller #5 [got 0 values in 0.000007 sec, idle 5 sec]
├─7495 /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, idle 5 sec]
├─7496 /usr/sbin/zabbix_server: trapper #1 [processed data in 0.000000 sec, waiting for connection]
├─7497 /usr/sbin/zabbix_server: trapper #2 [processed data in 0.000000 sec, waiting for connection]
├─7498 /usr/sbin/zabbix_server: trapper #3 [processed data in 0.000738 sec, waiting for connection]
├─7499 /usr/sbin/zabbix_server: trapper #4 [processed data in 0.000000 sec, waiting for connection]
├─7500 /usr/sbin/zabbix_server: trapper #5 [processed data in 0.000000 sec, waiting for connection]
├─7501 /usr/sbin/zabbix_server: icmp pinger #1 [got 0 values in 0.000005 sec, idle 5 sec]
├─7502 /usr/sbin/zabbix_server: alerter [sent alerts: 0 success, 0 fail in 0.000391 sec, idle 30 sec]
├─7503 /usr/sbin/zabbix_server: housekeeper [startup idle for 30 minutes]
├─7504 /usr/sbin/zabbix_server: timer #1 [processed 0 triggers, 0 events in 0.000023 sec, 0 maintenances in 0.001165 sec, idle 30 sec…
├─7505 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000546 sec, idle 5 sec]
├─7506 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000352 sec, idle 60 sec]
├─7507 /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000001 sec, idle 1 sec]
├─7508 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000001 sec, idle 1 sec]
├─7509 /usr/sbin/zabbix_server: history syncer #3 [synced 0 items in 0.000001 sec, idle 1 sec]
├─7510 /usr/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000001 sec, idle 1 sec]
├─7511 /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000411 sec, idle 3 sec]
├─7512 /usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000002 sec, idle 5 sec]
└─7513 /usr/sbin/zabbix_server: self-monitoring [processed data in 0.000009 sec, idle 1 sec]

Jun 04 13:09:21 zabbix systemd[1]: Starting Zabbix Server…
Jun 04 13:09:21 zabbix systemd[1]: Started Zabbix Server.

Finally lets enable Zabbix to start at boot time.

systemctl enable zabbix-server

Navigate to http://<YOURIP&gt;, and we need to do some configuration

Capture

Click Next and verify that all the checks come back with a status of OK

Capture

Click Next and enter in the zabbix database password

Capture

Click Next and leave the server settings at defaults

Capture.PNG

Verify your settings and click Next

Capture.PNGClick Finish if no errors are returned

Capture

You will be brought to the zabbix login screen, where the user is admin and the password is zabbix.

Capture

 

Advertisements

How to Install SNMP on UnRAID6

One of the things I like to have in my lab environment is the ability to monitoring all OSes and keep an eye on such things as temperatures, disk space, and other sensors.  I was disheartened to find that UnRAID 6 did not have SNMP installed or configured. After some searching I was able to figure out how to get SNMP installed.

First, Log into your UnRAID Web page and Click on Plugins

Capture.PNGNext copy and paste the NerdPack Plugin into the URL Field and Click Install.  The NerdPack installs the prerequisites that you need to install SNMP.  Then you will see a plugin window pop up.

Capture.PNG

Then we go and reboot the server.  This step is not necessary, but I prefer to do this after each plugin install.

Next go to Settings, and Nerd Pack

Capture.PNG

Find the entry for Perl and click the slider

Capture.PNG

Click Apply on the bottom

Capture.PNG

The package manager will launch a window and you see the package install.

Capture.PNG

Then we go and install the UnRaid SNMP Plugin following the same steps for the previous plugin install.

Capture.PNG

Login to the host via SSH console and verify that SNMP is working by executing

snmpwalk -v2c -c public localhost

You should see output similar to below.

Capture.PNG

Now you should be able to import your host into an SNMPD Based monitoring.

Capture.PNG

No 10gbe NICs showing in VMware 6.X with X10SDV-7TP4F

When I started building my VMware ESXi server, I did not have a Switch that could handle 10Gbe SFP+.  Now that I have a Dell X1052, I figured I would cable up a 10Gbe DAC and get moving.  Much to my surprise, I received a link light on the switch, but not on the motherboard.

Capture.PNG

Digging into the VMware side, I noticed that the 10Gbe NICs are not available, only the 1Gbe NICs.

Capture

A quick Google search brought me to a great site for VMware knowledge, tinkertry.com.  It appears that the drivers for the 10Gbe are not loaded.  So following the directions here, we open the SSH Console and enter the following command.

esxcli software vib install -v https://cdn.tinkertry.com/files/net-ixgbe_4.5.1-1OEM.600.0.0.2494585.vib –no-sig-check

We then reboot our host to get the new VIB to be loaded.

Capture

Low and behold on reboot we see the two 10gbe NICs.

Capture

How To Setup a CentOS 7 Syslog Server

Today we learn how to build a syslog server.  Having a couple SuperMicro Builds, Switches and Firewalls, I want to be able to know what is going on by looking at one central source.   Easiest way to do this, and one that most Network Devices and SuperMicro Motherboards support is syslog.

First step is to build a quick CentOS 7 VM.  Next we install rsylog and other packages we will use later.  If you have installed the Minimum CentOS 7 image you will need the vim and net-tools packages.

yum install rsyslog vim net-tools

Next we open the syslog.conf file to allow the server to listen for syslog messages.

vim /etc/rsyslog.conf:

Then we un-comment the following four lines to allow syslog messages to be received on UDP and TCP Ports 514.

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

becomes

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Then we restart the syslog service.

systemctl restart rsyslog.service

Then we want to verify that the service is listening on port 514.

netstat -an | grep 514

We should see that TCP is listening and that UDP is now showing for port 514.

Capture.PNG

Finally we want to allow TCP and UDP 514 through the firewall.

firewall-cmd –permanent –zone=public –add-port=514/tcp

firewall-cmd –permanent –zone=public –add-port=514/udp

firewall-cmd –reload

Now that we have the syslog server up and listening, lets configure one of our servers to forward its message.

For my SuperMicro servers, we log into the IPMI page and click on the Configuration Tab and then Syslog.  We click “Enable Syslog” and then we enter in the IP and Port for our Syslog Server.  Finally we click Save.

Capture

For ESXi we need to login to the CLI of each ESXi host and run the following command, replacing the below IP with your syslog IP.

esxcli system syslog config set –loghost=’tcp://10.11.12.13:514′

Capture

For me Dell X1052 Switch, we log into the Administrator Web GUI and Click Log and Alerts, Expand Remote Log Servers, and Click Edit.

Capture

Then we click Add.

Capture.PNG

Then we enter the required information for our Syslog Server.

Capture.PNG

Eventually as events are generated you will see them in /var/log/messages on the your syslog server.  Once we configured our Dell Switch we saw the following message generated.

Capture

PRTG Install Instructions

I am a bit of a Monitoring junky, and while I am mostly fluent Microsoft’s System Center Operations Manager, Observium, and Nagios/Opennms I do have a passing interest in PRTG

For myself, my current Observium Install does not have the MIBs installed to monitor my Dell x1052 Switch, and I liked to see a little more into how the switch is performing.

PRTG is known for handling network gear from all manufactuers very well, so I was curious to see how it performs.

First step is to go to PRTG’s website and download a trial.  PRTG will provide you are 100% functional trial key.  From there you build you host for PRTG, which for me was a low powered Virtual Machine.  The installer is extremely simple, just put in the Trial Key and Trial Key Name and let it install.  Once installed you will see two Icons on your desktop. As an astute reader pointed out, you are limited to 100 sensors, but this is perfectly fine for a homelab.

Capture.PNG

You can navigate to your servers IP address (or host name), and select the default login.

Capture.PNGPRTG can be very busy and distracting, but lets focus on getting our Dell Switch monitored.

Capture

Let’s click on Devices and the Add Device

Capture

Then we select Network Infrastructure

Capture

Next we plug in a device name, and an IP.  There is more configuration that can be done ahead of time to auto-discover items, but we won’t go into that here.

Capture.PNG

Once your device pops up on the Main Page, click Auto Discover.

Capture

After a while, we will see the following sensors pre-populated.

Capture.PNG

In a future post, I’ll show how to add custom sensors to monitor CPU, Memory and much more your Network Devices and Switches.

 

VMWare ESXi Passthrough and Freenas

I recently had user colicstew comment that they could not get Freenas 9.10.2 U3 to see drives on his LSI2116.  Luckily I have some unused 2TB Drives and LSI 2116 Controller as part of my Xeon D Lab.

IMG_0561.jpgIMG_0560.jpg

First step was to passthrough my LSI2116 Controller and Reboot the Server.  You should see Enabled/Active for the LSI Controller which we do

Snip20170511_1.png

Next I created a VM. Snip20170511_2.png

I chose to install the OS on one of my VMware Datastores.

Snip20170511_3.png

On the Hardware Settings I set as shown.

Snip20170511_4.png

Then I clicked Add other device -> PCI Device and Add the LSI ControllerSnip20170511_5.png

Finally I added a CD-ROM Drive and attached the Freenas ISO downloaded from here.

Snip20170511_7.png

Snip20170511_8.png

Finally we boot the VM

Snip20170511_9.png

Then we select Install at the first Screen

Snip20170511_13.png

One of the oddities I ran into while creating this tutorial was the key sequence to release the mouse from the VMWare Remote Console was the escape key in the Freenas installer that bounces you back to Initial Setup Screen, so the remaining pictures are via my phone.

On the next screen you will be warned about only having 4GB of memory.  Since I’m only using this as a test system, I am not concerned.  If you were running this as an actual file server you would want at a minimum 1GB of Memory per TB of storage.

IMG_0562.jpgOn the this screen we see five drives.  One is our drive we will install FreeNAS OS to, and the other four are 2TB Hitachi Drives I attached to the Xeon D Board.

IMG_0563.jpg

Next you are warned that FreeNAS will clear the drive you are installing to.

IMG_0564.jpg

Next you are asked to set the root password.

IMG_0565.jpg

Finally you are asked to choose a BIOS Mode.  

IMG_0566.jpg

At this point, I went to install the system, but no matter what I did, it would not install the OS, failing with the below error.

Snip20170511_15.png

The problem here is that the drive that we are installing the OS to is thin provisioned.  FreeNAS does not like this.

Snip20170511_16.png

Fix is to create the disk as “Thick provisioned, eagerly zeroed”.

Snip20170511_17.png

Once that was fixed, the OS installation continued without issue.

Snip20170511_18.png

Once completed, you will see the following success message.Snip20170511_19.png

Then, I chose option four to shutdown the system.  Be sure to remove the DVD Drive, or the system will continue to boot from the Installer.  Once that is completed your system will work as expected.

Snip20170511_23.png

Unfortunately, I had no issues getting the system up and running.  colicstew feel free to email me via the contact method and we can see about troubleshooting this issue.  I would start with verifying you have the correct breakout cable.  I used one of the four breakout cables from the Motherboard.

 

How to Enable SNMPD on ESXi 6.5

I had some difficulties enabling SNMPD from the GUI on ESXi 6.5 and kept recieving the following error

Failed – An error occurred during host configuration

Quick search let me here.

Running the below steps listed at the above link allowed me to start SNMPD in the EXSi GUI without issue

esxcli system snmp set -r
esxcli system snmp set -c YOUR_STRING
esxcli system snmp set -p 161
esxcli system snmp set -L "City, State, Country"
esxcli system snmp set -C noc@example.com
esxcli system snmp set -e yes