How to Install and Configure Zabbix

One of the issues I have with Observium is how limited the Device Discovery is.  Not all SNMP devices are recognized, and there is no way to customize the system with new Device MIB Files.  Zabbix was recommended to me by a couple of former coworkers, so I decided to give it a shot.

First I built a standard CentOS 7 VM and ran

yum update

 

Next I installed the Zabbix Repository RPM

[root@zabbix ~]# rpm -ivh http://repo.zabbix.com/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm
Retrieving http://repo.zabbix.com/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm
warning: /var/tmp/rpm-tmp.434W6N: Header V4 DSA/SHA1 Signature, key ID 79ea5ed4: NOKEY
Preparing… ################################# [100%]
Updating / installing…
1:zabbix-release-3.0-1.el7 ################################# [100%]

The next step is to install the Zabbix Server and Web Frontend with MySQL.

yum install zabbix-server-mysql zabbix-web-mysql

Next we install the zabbix agent in order to allow the server to monitor itself

yum install zabbix-agent

Next we need to install MySQL

yum install mariadb-server

Next we enable and start the service

systemctl enable mariadb

systemctl start mariadb

Now we set the root password for the database by running

mysql_secure_installation

You will see a handful of questions, and you should evaluate them individually.  My responses are not the most secure but for a demo they are fine to use.

[root@zabbix ~]# mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we’ll need the current
password for the root user. If you’ve just installed MariaDB, and
you haven’t set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on…

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
… Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y
… Success!

Normally, root should only be allowed to connect from ‘localhost’. This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
… skipping.

By default, MariaDB comes with a database named ‘test’ that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
– Dropping test database…
… Success!
– Removing privileges on test database…
… Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
… Success!

Cleaning up…

All done! If you’ve completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

We login to the database as the root user and create the Zabbix Database

mysql -uroot -p

From the mysql prompt enter the following to create the database

create database zabbix character set utf8;

Then we grant the zabbix server rights to the database.  You can change “Changeme123” to whatever password you wish.

grant all privileges on zabbix.* to zabbix@localhost identified by ‘Changeme123‘;

Finally apply the new permissions and exit the database prompt

flush privileges;

quit;

Next we need to import the schema and data into the Database.  It will prompt you for a password, of which the zabbix password set above can be used.

zcat /usr/share/doc/zabbix-server-mysql-3.0.9/create.sql.gz | mysql -u zabbix -p zabbix

Open the following configuration file and go to line 115

vim /etc/zabbix/zabbix_server.conf

Your output should look like below after you have made the recquired changes

### Option: DBPassword
# Database password. Ignored for SQLite.
# Comment this line if no password is used.
#
# Mandatory: no
# Default:
DBPassword=Changeme123

Next we update the system for the special PHP server Settings that Zabbix needs and open the following file.

vim /etc/httpd/conf.d/zabbix.conf

Go to line 19 and change the PHP value to be your timezone.  You can determine your closet timezone here.

My file looks like below

#
# Zabbix monitoring system php web frontend
#

Alias /zabbix /usr/share/zabbix

<Directory “/usr/share/zabbix”>
Options FollowSymLinks
AllowOverride None
Require all granted

<IfModule mod_php5.c>
php_value max_execution_time 300
php_value memory_limit 128M
php_value post_max_size 16M
php_value upload_max_filesize 2M
php_value max_input_time 300
php_value always_populate_raw_post_data -1
php_value date.timezone America/New_York
</IfModule>
</Directory>

<Directory “/usr/share/zabbix/conf”>
Require all denied
</Directory>

<Directory “/usr/share/zabbix/app”>
Require all denied
</Directory>

<Directory “/usr/share/zabbix/include”>
Require all denied
</Directory>

<Directory “/usr/share/zabbix/local”>
Require all denied
</Directory>

Next we go to the host specific httpd configuration file.

 vim /etc/httpd/conf/httpd.conf

Go to line 95 and change it from

#ServerName http://www.example.com:80

to

ServerName <YOURIPADDRESS>:80

Next we restart httpd

systemctl restart httpd

Before we start the Zabbix Server you need to give the Zabbix User rights to certain folders.

chown -R zabbix:zabbix /var/log/zabbix
chown -R zabbix:zabbix /var/run/zabbix
chmod -R 775 /var/log/zabbix/
chmod -R 775 /var/run/zabbix/

Next we execute the following and let th zabbix_agentd what configuration file to use

/usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf

Next we disable SELINUX

setenforce 0

Now we need to allow HTTP traffic for FirewallD

firewall-cmd –zone=public –add-service=http –permanent

firewall-cmd –reload

Now we start the Zabbix Server

systemctl start zabbix-server

Finally check the zabbix-server status

systemctl status zabbix-server

The output should be similar to below

[root@zabbix ~]# systemctl status zabbix-server
● zabbix-server.service – Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2017-06-04 13:09:21 EDT; 39s ago
Process: 7480 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 7482 (zabbix_server)
CGroup: /system.slice/zabbix-server.service
├─7482 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
├─7487 /usr/sbin/zabbix_server: configuration syncer [waiting 60 sec for processes]
├─7488 /usr/sbin/zabbix_server: db watchdog [synced alerts config in 0.000800 sec, idle 60 sec]
├─7489 /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000005 sec, idle 5 sec]
├─7491 /usr/sbin/zabbix_server: poller #2 [got 0 values in 0.000005 sec, idle 5 sec]
├─7492 /usr/sbin/zabbix_server: poller #3 [got 0 values in 0.000005 sec, idle 5 sec]
├─7493 /usr/sbin/zabbix_server: poller #4 [got 0 values in 0.000006 sec, idle 5 sec]
├─7494 /usr/sbin/zabbix_server: poller #5 [got 0 values in 0.000007 sec, idle 5 sec]
├─7495 /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, idle 5 sec]
├─7496 /usr/sbin/zabbix_server: trapper #1 [processed data in 0.000000 sec, waiting for connection]
├─7497 /usr/sbin/zabbix_server: trapper #2 [processed data in 0.000000 sec, waiting for connection]
├─7498 /usr/sbin/zabbix_server: trapper #3 [processed data in 0.000738 sec, waiting for connection]
├─7499 /usr/sbin/zabbix_server: trapper #4 [processed data in 0.000000 sec, waiting for connection]
├─7500 /usr/sbin/zabbix_server: trapper #5 [processed data in 0.000000 sec, waiting for connection]
├─7501 /usr/sbin/zabbix_server: icmp pinger #1 [got 0 values in 0.000005 sec, idle 5 sec]
├─7502 /usr/sbin/zabbix_server: alerter [sent alerts: 0 success, 0 fail in 0.000391 sec, idle 30 sec]
├─7503 /usr/sbin/zabbix_server: housekeeper [startup idle for 30 minutes]
├─7504 /usr/sbin/zabbix_server: timer #1 [processed 0 triggers, 0 events in 0.000023 sec, 0 maintenances in 0.001165 sec, idle 30 sec…
├─7505 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000546 sec, idle 5 sec]
├─7506 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000352 sec, idle 60 sec]
├─7507 /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000001 sec, idle 1 sec]
├─7508 /usr/sbin/zabbix_server: history syncer #2 [synced 0 items in 0.000001 sec, idle 1 sec]
├─7509 /usr/sbin/zabbix_server: history syncer #3 [synced 0 items in 0.000001 sec, idle 1 sec]
├─7510 /usr/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000001 sec, idle 1 sec]
├─7511 /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000411 sec, idle 3 sec]
├─7512 /usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000002 sec, idle 5 sec]
└─7513 /usr/sbin/zabbix_server: self-monitoring [processed data in 0.000009 sec, idle 1 sec]

Jun 04 13:09:21 zabbix systemd[1]: Starting Zabbix Server…
Jun 04 13:09:21 zabbix systemd[1]: Started Zabbix Server.

Finally lets enable Zabbix to start at boot time.

systemctl enable zabbix-server

Navigate to http://<YOURIP&gt;, and we need to do some configuration

Capture

Click Next and verify that all the checks come back with a status of OK

Capture

Click Next and enter in the zabbix database password

Capture

Click Next and leave the server settings at defaults

Capture.PNG

Verify your settings and click Next

Capture.PNGClick Finish if no errors are returned

Capture

You will be brought to the zabbix login screen, where the user is admin and the password is zabbix.

Capture

 

Advertisements

How to Install SNMP on UnRAID6

One of the things I like to have in my lab environment is the ability to monitoring all OSes and keep an eye on such things as temperatures, disk space, and other sensors.  I was disheartened to find that UnRAID 6 did not have SNMP installed or configured. After some searching I was able to figure out how to get SNMP installed.

First, Log into your UnRAID Web page and Click on Plugins

Capture.PNGNext copy and paste the NerdPack Plugin into the URL Field and Click Install.  The NerdPack installs the prerequisites that you need to install SNMP.  Then you will see a plugin window pop up.

Capture.PNG

Then we go and reboot the server.  This step is not necessary, but I prefer to do this after each plugin install.

Next go to Settings, and Nerd Pack

Capture.PNG

Find the entry for Perl and click the slider

Capture.PNG

Click Apply on the bottom

Capture.PNG

The package manager will launch a window and you see the package install.

Capture.PNG

Then we go and install the UnRaid SNMP Plugin following the same steps for the previous plugin install.

Capture.PNG

Login to the host via SSH console and verify that SNMP is working by executing

snmpwalk -v2c -c public localhost

You should see output similar to below.

Capture.PNG

Now you should be able to import your host into an SNMPD Based monitoring.

Capture.PNG

How To Setup a CentOS 7 Syslog Server

Today we learn how to build a syslog server.  Having a couple SuperMicro Builds, Switches and Firewalls, I want to be able to know what is going on by looking at one central source.   Easiest way to do this, and one that most Network Devices and SuperMicro Motherboards support is syslog.

First step is to build a quick CentOS 7 VM.  Next we install rsylog and other packages we will use later.  If you have installed the Minimum CentOS 7 image you will need the vim and net-tools packages.

yum install rsyslog vim net-tools

Next we open the syslog.conf file to allow the server to listen for syslog messages.

vim /etc/rsyslog.conf:

Then we un-comment the following four lines to allow syslog messages to be received on UDP and TCP Ports 514.

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

becomes

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Then we restart the syslog service.

systemctl restart rsyslog.service

Then we want to verify that the service is listening on port 514.

netstat -an | grep 514

We should see that TCP is listening and that UDP is now showing for port 514.

Capture.PNG

Finally we want to allow TCP and UDP 514 through the firewall.

firewall-cmd –permanent –zone=public –add-port=514/tcp

firewall-cmd –permanent –zone=public –add-port=514/udp

firewall-cmd –reload

Now that we have the syslog server up and listening, lets configure one of our servers to forward its message.

For my SuperMicro servers, we log into the IPMI page and click on the Configuration Tab and then Syslog.  We click “Enable Syslog” and then we enter in the IP and Port for our Syslog Server.  Finally we click Save.

Capture

For ESXi we need to login to the CLI of each ESXi host and run the following command, replacing the below IP with your syslog IP.

esxcli system syslog config set –loghost=’tcp://10.11.12.13:514′

Capture

For me Dell X1052 Switch, we log into the Administrator Web GUI and Click Log and Alerts, Expand Remote Log Servers, and Click Edit.

Capture

Then we click Add.

Capture.PNG

Then we enter the required information for our Syslog Server.

Capture.PNG

Eventually as events are generated you will see them in /var/log/messages on the your syslog server.  Once we configured our Dell Switch we saw the following message generated.

Capture

PRTG Install Instructions

I am a bit of a Monitoring junky, and while I am mostly fluent Microsoft’s System Center Operations Manager, Observium, and Nagios/Opennms I do have a passing interest in PRTG

For myself, my current Observium Install does not have the MIBs installed to monitor my Dell x1052 Switch, and I liked to see a little more into how the switch is performing.

PRTG is known for handling network gear from all manufactuers very well, so I was curious to see how it performs.

First step is to go to PRTG’s website and download a trial.  PRTG will provide you are 100% functional trial key.  From there you build you host for PRTG, which for me was a low powered Virtual Machine.  The installer is extremely simple, just put in the Trial Key and Trial Key Name and let it install.  Once installed you will see two Icons on your desktop. As an astute reader pointed out, you are limited to 100 sensors, but this is perfectly fine for a homelab.

Capture.PNG

You can navigate to your servers IP address (or host name), and select the default login.

Capture.PNGPRTG can be very busy and distracting, but lets focus on getting our Dell Switch monitored.

Capture

Let’s click on Devices and the Add Device

Capture

Then we select Network Infrastructure

Capture

Next we plug in a device name, and an IP.  There is more configuration that can be done ahead of time to auto-discover items, but we won’t go into that here.

Capture.PNG

Once your device pops up on the Main Page, click Auto Discover.

Capture

After a while, we will see the following sensors pre-populated.

Capture.PNG

In a future post, I’ll show how to add custom sensors to monitor CPU, Memory and much more your Network Devices and Switches.

 

How to Enable SNMPD on ESXi 6.5

I had some difficulties enabling SNMPD from the GUI on ESXi 6.5 and kept recieving the following error

Failed – An error occurred during host configuration

Quick search let me here.

Running the below steps listed at the above link allowed me to start SNMPD in the EXSi GUI without issue

esxcli system snmp set -r
esxcli system snmp set -c YOUR_STRING
esxcli system snmp set -p 161
esxcli system snmp set -L "City, State, Country"
esxcli system snmp set -C noc@example.com
esxcli system snmp set -e yes

How to Install Observium on CentOS 7

NOTE – May 5th 2017 – Post has been updated to point to latest EPEL RPM

The directions for installing Observium Monitoring on CentOS 7 aren’t exactly accurate.  They have a handful of mistakes and missing steps.  Hopefully this guide will fill in the mistakes and missing parts and get you started on your way.  I know for me it was a bit of a headache getting it to work.  I eventually turned to Turnkey Linux for a template on how to do it.  With the release of the latest community version of Observium (0.16.10 on 26th October 2016),  I decided to give it a shot again.

We assume at the beginning of this tutorial that you have a working CentOS 7 VM that has network connectivity.

The first mistake is that Official Observium Documentation points you to install the RPMForge and EPEL Repositories.  The problem is RPMForge no longer exists, and the EPEL Link they provide is incorrect.

  1.  Install two requirements for logging in remotely to the VM and for getting the EPEL Repository setup.

yum install openssh wget

  1.  Install the EPEL Repository.

wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-9.noarch.rpm
rpm -ivh epel-release-7-9.noarch.rpm

  1.  Install the needed packages for Observium.

yum install httpd.x86_64 php.x86_64 php-mysql.x86_64 php-gd.x86_64 php-posix php-mcrypt.x86_64 php-pear.noarch cronie.x86_64 net-snmp.x86_64 net-snmp-utils.x86_64 fping.x86_64 mariadb-server.x86_64 mariadb.x86_64 MySQL-python.x86_64 rrdtool.x86_64 subversion.x86_64 jwhois.x86_64  ipmitool.x86_64 graphviz.x86_64 ImageMagick.x86_64 libvirt.x86_64 net-tools bind-utils

  1.  Make the directories needed for an Observium Install.

mkdir -p /opt/observium && cd /opt

  1.  Download and untar Observium.

wget http://www.observium.org/observium-community-latest.tar.gz
tar zxvf observium-community-latest.tar.gz

  1.  Remove the tar File.

rm observium-community-latest.tar.gz

  1.  Enable and Start the MySQL (mariadb) Service.

systemctl enable mariadb
systemctl start mariadb

  1.  Set the root password for MySQL.  We will use Changeme123.

/usr/bin/mysqladmin -u root password ‘Changeme123’

  1.  Login to a MySQL Prompt and configure the database and grant the correct privileges.

mysql -u root -p **Note MySQL will prompt you for the root password set above**
mysql> CREATE DATABASE observium DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
mysql> GRANT ALL PRIVILEGES ON observium.* TO ‘observium’@’localhost’ IDENTIFIED BY ‘Changeme123’;
mysql> exit;

  1.  Change Directory into the Observium Folder and copy the default config file to config.php in order to edit the file.

cd observium

cp config.php.default config.php

  1.  Edit the config.php in an editor of your change and Change the db_user and db_pass fields.  The remaining fields can be left to edit later.

post3

  1.  MySQL Strict mode should be enabled, but as of version 5.7 it is enabled by default.  This is a step in the official documentation, but it is one we can skip.

  2.  We run discovery.php script to setup the database schema.

./discovery.php -u

  1.  In CentOS 7 the locations of fping and ping differ from that of where Observium expects them to be.  Lets override those values in the config but adding these two lines.

$config[‘ping’] = “/usr/bin/ping”;

$config[‘fping’] = “/usr/sbin/fping”;

Post4.PNG

  1. SELinux needs to be disabled.  This is the simplest way to get Observium to work.

setenforce 0

vim /etc/selinux/config

SELINUX = permissive

Post5.PNG

  1.  Create the RRD Directory and give apache rights to it.

mkdir rrd
chown apache:apache rrd

  1.  Setup httpd.conf for use with Observium.  We assume only Observium will be run on this host.  Be sure to replace the Server in the ServerName line to be that of your fully qualified domain hostname.

vim /etc/httpd/conf/httpd.conf

<VirtualHost *:80>
DocumentRoot /opt/observium/html/
ServerName observium.domain.com
CustomLog /opt/observium/logs/access_log combined
ErrorLog /opt/observium/logs/error_log
<Directory “/opt/observium/html/”>
AllowOverride All
Options FollowSymLinks MultiViews
Require all granted
</Directory>
</VirtualHost>

Post6.PNG

  1.  Create the logs directory and give apache rights.

mkdir /opt/observium/logs
chown apache:apache /opt/observium/logs

  1.  Create your initial web login user as an admin.  We use user admin, password Changeme123 and level 10 which is admin.

cd /opt/observium
./adduser.php admin Changeme123 10

  1.  Add your first device.  Be sure that SNMP is enabled on the device and that you know the community.  For us we will use our Mikrotik Switch.

./add_device.php 10.0.1.2 public v2c

  1.  Now that Observium knows about the host, lets discovery it and poll for the data off the switch.

./discovery.php -h all
./poller.php -h all

  1.  Create a cron job to run discovery and polling on a regular interval.  A Note in the Observium documentation states.

The below example includes a username, so will only work in /etc/crontab or /etc/cron.d/observium. It will NOT work in a user crontab edited with crontab -e without removing the username.

So we create the cron file in /etc/cron.d/observium.

# Run a complete discovery of all devices once every 6 hours
33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1

# Run automated discovery of newly added devices every 5 minutes
*/5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1

# Run multi threaded poller wrapper every 5 minutes
*/5 * * * * root /opt/observium/poller-wrapper.py 8 >> /dev/null 2>&1

# Run housekeeping script daily for syslog, event log and alert log
13 5 * * * root /opt/observium/housekeeping.php -ysel

# Run housekeeping script daily for rrds, ports, orphaned entries in the database and performance data
47 4 * * * root /opt/observium/housekeeping.php -yrptb

  1.  Reload cron.

systemctl reload crond

  1.  Set httpd to start on boot.

systemctl enable httpd
systemctl start httpd

That is about it for the setup of Observium.  The official documentation states that you need to open firewall ports.  Since the firewall is not installed by default, I have skipped those steps.  If you need them they are.

#Permit HTTP through the server’s default firewall
firewall-cmd –permanent –zone=public –add-service=http
firewall-cmd –reload

If everything went correctly you will be greeted with the login hamster.

Post8.PNG

Our next post will cover how to add hosts in the Web Page, how to troubleshoot when those hosts can’t be added, and how to add SuperMicro IPMI polling.