Bischeck
-
Installation and administration guide

Version 1.0.0
2014-03-15

Legal Notice Copyright
© 2013-2014 Ingenjörsbyn AB.
This document is licensed by Ingenjörsbyn AB under the Creative Commons Attribution-ShareAlike 3.0 Unported License, http://creativecommons.org/licenses/by-sa/3.0/. If you distribute this document, or a modified version of it, you must provide attribution to Ingenjörsbyn AB and provide a link to the original.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
Nagios® is an official trademark of Nagios Enterprise Inc.
All other trademarks are the property of their respective owners.


Abstract
This guide provides information of the installation and administration of Bischeck. For more in-depth configuration options please see the “Bischeck - configuration guide”.
Table of Contents
Chapter 1: Getting help
Chapter 2: Installation and upgrading
Section 2.1: Installation
Subsection 2.1.1: Installation directory structure
Section 2.2: Upgrading
Subsection 2.2.1: Rollback upgrade
Section 2.3: Getting started
Chapter 3: Customization
Section 3.1: Jar customization
Section 3.2: Logging
Section 3.3: Integration with pnp4nagios
Chapter 4: Command line utilities
Section 4.1: Run Bischeck
Section 4.2: Validating configuration files
Section 4.3: Twenty4HourThreshold testing
Chapter 5: Bischeck internal surveillance
Section 5.1: Operations
Section 5.2: Timers
Chapter 6: Building Bischeck from source
Section 6.1: Developing with Bischeck
Section 6.2: Developing custom JEP functions
Chapter 7: System requirements
Section 7.1: Java
Section 7.2: Redis
Section 7.3: Jar dependencies
Chapter 8: Bischeck license
Chapter 9: Bug reports and feature requests
Chapter 10: Credits

1 Getting help

More information about Bischeck and related projects can be found on www.bischeck.org.

2  Installation and upgrading

2.1 Installation

The latest binary version of Bischeck is available here http://www.bischeck.org.
Download the distribution file and follow the steps below to install. Make sure you have root privileges when running the installation if you want the init scripts to be installed for automatic startup and shutdown. Before installing, please see the System requirements ↓ chapter. 
# tar xzvf bischeck-x.y.z.tar.gz
# cd bischeck-x.y.z
# chmod 755 install 
# ./install -u #Get usage
# ./install    #Install default
# service bischeckd start    #Redhat/Centos
# /etc/init.d/bischeck start #Debian/Ubuntu
To get a full list of available options to the install script use -u. 
# ./install -u 
Welcome to bischeck 1.0.0_RC2 installer 
==================================== 
Copyright Anders Håål, Ingenjörsbyn AB 2011-2013 
Licensed under GPL version 2
Usage for the bischeck installer 
-u show usage. 
-U the user name to install bischeck as and run the daemon as - default nagios. 
-J java home directory, no default value. 
-I installation directory for bischeck - default /opt/socbox/addons/bischeck. 
-R uninstall - permanently remove the installation. 
-X upgrade from current version - if possible. 
-d set the linux distribution name if the installer can not detected. Supported are    
   rh, rhel, redhatenterpriseserver, centos, debian and ubuntu 
-p the port number for the JMX RMI server, for example 3333. No default value. 
-i the IP address where the RMI port should run, for example 127.0.0.1. No default value. 
-a if authentication should be used applied on JMX connection. Default is false.
​
By default, the install script will install Bischeck in directory /opt/socbox/addons/bischeck, referred to as $BISHOME in this documentation and with the ownership of the user id nagios. Make sure that the user exists before running install.
The last commands start the Bischeck daemon with the effective user id of the user id set during the installation, default user nagios. The installation will configure bischeckd to start automatically in run level 3, 4 and 5.
The process id of the java process running Bischeck in daemon mode will be stored in a file, default in /var/tmp/bischeck.pid. [A]  [A] The location of the pid file can be changed by a setting in the Bischeck properties.xml file. More info in the “Bischeck configuration guideThis file is used by the bischeckd script to stop the java process running Bischeck and ensure that only one instance of Bischeck is started on the server.
If the installation is done as a none root user, effective user id is not root, Bischeck will be installed as the user running the install script. This means that the init script, bischeckd, will not be installed in /etc/init.d. To start Bischeck, the command line utility must be used, see Command line utilities↓.

2.1.1 Installation directory structure

When the installation is complete, the following directory structure exists: 
$BISHOME - 
         |- /bin        - Scripts and init scripts 
         |- /customlib  - Custom jar files, like jdbc drivers 
         |- /etc        - Bischeck XML based configuration files
         |- /lib        - All jar files requiered for Bischeck 
         |                including the Bischeck jar
         |- /resources  - Different resources files, like XML schema 
         |                files, logback.xml, etc.
         |- svninfo.txt - Subversion commit number for the version  
         |- version.txt - Bischeck version

2.2  Upgrading

Download the new version as described in the previous chapter and to upgrade run the install script with the option -X: 
# ./install -X
The upgrade will automatically stop the currently running Bischeck and save the current installation in a directory parallel to the new version named bischeck_x.y.z, where x.y.z is the version of the old installation. 
# ./install -I /usr/local/bischeck -X
The -I switch must be used if the currently installed version of Bischeck is not in the default installation directory.
The file migrationpath.txt describes the supported upgrade paths and the migration scripts that will run by the install script.
If the upgrade is successful Bischeck can be started manually by executing: 
# service bischeckd start

2.2.1 Rollback upgrade

In the event of an upgrade failure or if the new version of Bischeck do not work properly a simple rollback can be conducted with the following step. The process assume that the installation directory is /opt/socbox/addons/bischeck:
  1. Go to the parent directory of where Bischeck has been installed:
    # cd /opt/socbox/addons/
  2. Remove the newly installed Bischeck:
    # rm -rf bischeck
  3. Move the old version, saved by the upgrade to bischeck_x.y.z where x.y.z is the version number of the previous installed version, to the install directory:
    # mv bischeck_x.y.z bischeck
  4. Replace the init script bischeckd with the old version:
    # cp bischeck/bin/bischeckd /etc/init.d/bischeckd
# chmod 755 /etc/init.d/bischeckd

2.3  Getting started

In the $BISHOME/etc directory there are working examples of all the configuration files. These files serve as an examples to get started with a simple Bischeck setup of monitoring the response time of a tcp ping on the ssh port, port 22, on localhost. You will find this familiar if you have been using Nagios before and the example uses the well known Nagios check command, check_tcp. The example assumes that you have the Nagios plugins installed. To get the example running you need to perform the following steps:
  1. Go to the $BISHOME/etc directory, default is /opt/socbox/addons.
  2. Edit the bischeck.xml at line 16 with the path to the check_tcp command. If you have check_tcp installed in /usr/lib/nagios/plugins you do not need to edit the file.
  3. Edit the servers.xml file to add the correct host name, line 12, and password, line 22, for your NSCA server. Even if you do not have an NSCA server defined, Bischeck will still run, but with a connection error when trying to send data to NSCA.
  4. Now you are done to start Bischeck and check out what happens in the log file. See Logging↓ for more information about Bischeck logging. 
$ sudo /etc/init.d/bischeckd restart
$ tail -f /usr/tmp/bischeck.log
Maybe you ask yourself why you should use Bischeck to do a check_tcp? This is just a simple example, but when you have this running you can check out the 24thresholds.xml file to get some more understanding about the power of dynamic and adaptive thresholds.

3 Customization

3.1  Jar customization

To enable support for custom jar files, please place them in the directory $BISHOME/customlib. This can typically be jdbc drivers, custom threshold classes, etc. Any jar file in the $BISHOME/customlib will automatically be class loaded by Bischeck.

3.2  Logging

Bischeck use logback, http://logback.qos.ch/ for log management. The logback configuration is described in the logback.xml file located in the $BISHOME/resources directory. By default Bischeck writes log information at level INFO to file /var/tmp/bischeck.log.

3.3  Integration with pnp4nagios

pnp4nagios can create graph layouts depending on the check command used for the service on the Nagios server. Since Bischeck uses a passive check, we need to create a unique check command that matches the pnp4nagios layout for Bischeck. Create a link in the libexec directory on the nagios server: 
nagios$ ln -s check_dummy check_bischeck
When describing the service, always use the check_bischeck as the check command in the Nagios configuration. The check_bischeck.php that controls the pnp4nagios layout must be copied to the directory pnp4nagios/share/templates on the Nagios server.

4  Command line utilities

There are a number of command line utilities available in Bischeck. All of which can be run through the bischeck script located in the $BISHOME/bin directory. To use the bischeck script, add it to your PATH variable. 
$ PATH=$BISHOME/bin:$PATH

4.1 Run Bischeck

The normal way to run Bischeck is as a daemon using the init.d script bischeckd, but it is also possible to start Bischeck in continuous running mode by executing: 
$ bischeck Execute -d
Running in this way have limitations since the execution will not automatically be placed as a background process and the effective user id will be the user starting the process which may not have all necessary permissions according to the installation. Neither will the pid files be updated correctly. For production systems, always use the init.d script. 
$ sudo /etc/init.d/bischeckd start
or 
# service bischeckd start
For testing purposes, it can be beneficial to run Bischeck once and make sure that everything is executing as expected. This is done by executing: 
$ bischeck Execute
This will override all scheduling definitions and execute everything directly, but only once.
To show the pid file used for the Bischeck daemon running: 
$ bischeck configuration.ConfigurationManager -p
This command is used in the init script bischeckd to retrieve the current pid.

4.2 Validating configuration files

To validate the correctness of the xml configuration files, use the following command, which will return 0 if files are correct. Use $? to display return status. 
$ bischeck configuration.ConfigurationManager -v; echo $?

4.3 Twenty4HourThreshold testing

To verify which threshold rule that will be used for a specific host, service and service item for a specific date, run: 
$ bischeck threshold.Twenty4HourThreshold -h erphost -s orders -i ediorders -d 20141207
The above command will display the threshold definitions that will be used for host erphost, service orders and serviceitem ediorders on the seventh of December 2014. If -d is omitted, the current date will be used.

5  Bischeck internal surveillance

Bischeck uses the Java JMX standard for internal monitoring. Bischeck exposes a number of JMX MBeans that enables controlling and instrumentation of the running Bischeck daemon. For example, Bischeck can be shutdown or reloaded through JMX. A reload operation is used to force Bischeck to re-read and deploy its configuration files but without restarting the process.
To allow remote JMX connections the following JMX settings are used by the bischeck script located in the $BISHOME/bin directory. 
jmxport=-Dcom.sun.management.jmxremote.port=<set by the installer script>
jmxrmiserver=-Djava.rmi.server.hostname=<set by the installer script>
jmxauth=-Dcom.sun.management.jmxremote.authenticate=<set by the installer script>
jmxssl=-Dcom.sun.management.jmxremote.ssl=false
jmxpasswd=-Dcom.sun.management.jmxremote.password.file=$bishome/etc/jmxremote.password
jmxaccess=-Dcom.sun.management.jmxremote.access.file=$bishome/etc/jmxremote.access
If authentication should be used, please change the configuration of the roles and passwords in the files jmxremote.access and jmxremote.password. For more information on how to configure JMX, please visit
http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html.
JMX is only enabled when the bischeck script is called with the argument "Execute -d", which is the way the bischeckd init script call the script bischeck to start Bischeck in daemon mode.

5.1 Operations

Bischeck enables a number of attributes and operations that can be managed over JMX. Please read the javadoc for the MBean classes to understand the methods available or just start jconsole or equivalent tool to browse available MBeans.
The key MBeans are:

5.2 Timers

Different execution times of the Bischeck daemon are also exposed through JMX. All timers are based on the http://metrics.codahale.com/ and shows the following parameters for each defined timer:
The timers used in Bischeck are:

6  Building Bischeck from source

To build Bischeck from source, check out the Bischeck trunk from gforge.ingby.com: 
$ svn checkout --username anonymous http://gforge.ingby.com/svn/bischeck/bischeck/trunk bischeck
To build a Bischeck distribution, run ant from the directory where you checked out the Bischeck code: 
$ ant dist
This will create a compressed tar file in the target directory, named bischeck-x.y.z.tar.gz where x.y.z is the version number. Different versions of Bischeck can be checked out from the tags directory located in http://gforge.ingby.com/svn/bischeck/bischeck/tags
The dist directive will also convert all documentation in lyx format to pdf. This requires the lyx [C]  [C] On Redhat/Centos lyx are part of the EPEL repos and should be version 2 or higher. and imagemagick packages to be installed.
To run all unit tests: 
$ ant testng
To set the Bischeck version number, the build.xml file must be edited or the ant variable app.version can be overridden: 
$ ant -Dapp.version=X.Y.Z dist

6.1  Developing with Bischeck

It is simple to develop your own service, serviceitem, threshold and server classes. To develop your own you must follow the interface that exists for each type. For service and serviceitems, an abstract class exists with default implementation of most of the methods described in the interfaces. For more information, check out the Bischeck javadoc, that is generated using ant and that is part of the distribution in the $BISHOME/docs/javadoc directory.

6.2 Developing custom JEP functions

To develop custom mathematical JEP functions that operate on cached data, please see the JEP documentation, http://www.cse.msu.edu/SENS/Software/jep-2.23/doc/website/doc/doc_usage.htm. To make a function recognized by Bischeck, it should be put in a jar and copied to the $BISHOME/customlib directory. The class must then be mapped to a name that will be used in the Bischeck. This is done in the file jepextfunctions.xml located in the $BISHOME/resources directory. The jepextfunctions.xml file is a standard Java XML property file where the key is the name of the function and the value is the class name that implements it.

7  System requirements

7.1 Java

Bischeck should run on any operating system that supports Java 6 or higher. The installation and init scripts are supported on Redhat and Debian equivalent Linux distributions. Running on Microsoft Windows has been tested successfully but is not supported by the installation scripts.

7.2 Redis

Since version 1.0.0 of Bischeck, redis, http://www.redis.io, is the default memory cache. Redis provides a different range of persistence options like point-in-time snapshots and persistence logs. Redis also supports replication and high-availability options.
Bischeck has been tested with redis version 2.6.7. Installation of redis is not part of the Bischeck installation and should be installed prior to installing Bischeck. Redis is not required to run on the same server as Bischeck.
The connection to redis is configured in the $BISHOME/etc/property.xml file. The following properties control the connection:
There is also an additional property that can be used with the new redis based cache and that is cache.provider.redis.fastCacheSize. This parameter defines the number of cache items that for each service definition (host-service-serviceitem) should be held in parallel with the redis cache in the bischeck heap space. If Bischeck finds the data in the fast cache on a read, access to redis is not needed. Data writes are always written to redis storage. The default value for the option is 0, meaning that the fast cache is size 0. Depending on how the cached data is used in your configuration, increasing the fast cache can improve performance but will also increase the heap space.
With redis it is possible to access the data in the cache at runtime. Redis provides many client libraries for different programming languages, http://redis.io/clients. The quickest way is just to use the redis-cli command line utility.
The cache prior to version 1.0.0, should be regarded as deprecated. Many of the the new features available as of version 1.0.0 are not supported in the old implementation, such as e.g. aggregations.

7.3 Jar dependencies

The following jar packages are distributed as part of the Bischeck distribution. All these packages have their own open source licenses.
All jar files are distributed as part of Bischeck and located in the lib directory.

8  Bischeck license

Bischeck is licensed under GNU license version 2. For more info please visit http://www.gnu.org/licenses/gpl-2.0.htm

9 Bug reports and feature requests

Please submit bug reports and feature requests on www.bischeck.org in the Forge section.

10  Credits

Thanks to all people who have developed all the great software that Bischeck depends on and especially all who made Nagios and the Nagios community a success. A special thanks goes to Peter Johansson and his colleagues at DHL Freight Sweden for sponsoring the development and providing ideas and important feedback.
The Bischeck project would like to thank the following companies that sponsor the Bischeck project with valuable commercial tools and development environments: