Graphite

Graphite auf Ubuntu 16.04 installieren

Die folgende Installationsanleitung stammt größtenteils aus dem Dokument [to Setup Graphite with Statsd on an Ubuntu 16.04]. Die im Folgenden dargestellten Befehle müssen als superuser ausgeführt werden.

Hi, today we are going to show you installation and configuration of Graphite and Statsd on Ubuntu 16.04 LTS. In short Graphite is a highly scalable real-time graphing system which provides real-time visualization and storage of numeric time-series data. This means that Graphite requires other applications to collect and transfer the data points. The Graphite project itself is composed of a few different components, each of which has a specific and focused purpose where the most visible and dynamic component of a Graphite installation is Graphite web application. Graphite-Web, a Django-based web application that renders graphs and dashboards. It gives a flexible interface to design graphs. You can combine different types of metrics, control labelling, fonts, colors and line properties, you can resize and manipulate the data where required. The second major component of Graphite is the Carbon metric processing daemons. A single Graphite configuration will have one or more Carbon daemons that are responsible for handling data that is sent over by other processes that collect and transmit statistics. While the third major component of Graphite is the Whisper time-series database library, which is very flexible and allows time-series data to be stored in great detail. It creates different archives at different levels of detail, so that in practical usage, the information gracefully degraded into lower resolutions when it passes certain configured ageing thresholds.

Graphite itself does not concern itself with data collection.So, for this purpose we will be using a collection program that is Statsd, that will be used be send other data to Graphite for processing. The StatsD service will collect all data as it comes, aggregate it, and then send summarized data points to Graphite in the time frame it expects.

Prerequisites

Before moving towards installation of Graphite and Statsd, we need to need to make sure the system is fully updated as many of the required Graphite packages are available in the default repository of Ubuntu.

Let's login to your Ubuntu 16.04 LTS server using root or sudo user credentials and run the command below to update your system.

apt-get update -y

Once your system is updated, move to the installation of Graphite packages first.

Installing Graphite on Ubuntu 16

The installation of Graphite has been made fairly simple because of its availability of repository in Ubuntu. Let's run the following command to install Graphite along with its all the components that is web application, a storage backend called Carbon, and the database library called whisper.

apt-get install graphite-web graphite-carbon

Once you press 'y' and hit the Enter key, your installation process will begin, while during the installation process you will be asked for whether you want Carbon to remove the database files or not. Let's choose "No" here not to destroy your stats.

Let the installation process completes, after that we have to configure some necessary parameters for the Graphite to make it function properly.

Setup Database for Django

The Graphite web application is a Django Python application that needs to store its data instead of carbon and the whisper database library. By default its configured to use SQLite3 database, but it would be much better to use PostgreSQL. Let's install PostgreSQL along with its dependent packages by using below command.

apt-get install postgresql libpq-dev python-psycopg2

PostgreSQL packages along with required python libraries to interact with Graphite will be installed after your type 'Y' and hit the "Enter" key.

Now connect to the PostgreSQL using 'postgres' user and create a new user and database to be used for Graphite by using the following commands.

sudo -u postgres psql

postgres=# CREATE USER graphite WITH PASSWORD 'passw0rd';

postgres=# CREATE DATABASE graphite WITH OWNER graphite;

postgres=# \q

Configuring the Graphite Web Application

After setting up database, now we will configure Graphite by opening its web application configuration file to update its 'SECRET_KEY' that should be set to a long, random unique string to use as a secret key for this install. This key is used for salting of hashes used in auth tokens, CRSF middleware, cookie storage, etc. This should be set identically among instances if used behind a load balancer.

vim /etc/graphite/local_settings.py

SECRET_KEY = 'secure_key'

Set your local timezone (Django's default is America/Chicago) If your graphs appear to be offset by a couple hours then this probably needs to be explicitly set to your local timezone.

TIME_ZONE = 'Europe/London'

UN-Comment the REMOTE_USER authentication

USE_REMOTE_USER_AUTHENTICATION = True

Then under the database configurations, update the parameters according to the previously created user and database in PostgreSQL as shown.

DATABASES = {
  'default': {
  'NAME': 'graphite',
  'ENGINE': 'django.db.backends.postgresql_psycopg2',
  'USER': 'graphite',
  'PASSWORD': 'passw0rd',
  'HOST': '127.0.0.1',
  'PORT': 
  }
}

Save and close the configuration file but DO NOT FORGET TO RUN 'manage.py syncdb' AFTER SETTING UP A NEW DATABASE.

Graphite Database Sync

Run the following command to sync database, where the following number of operation swill be performed as shown in the image and you will be asked to create a superuser account for the database.

graphite-manage migrate auth

Operations to perform:
Apply all migrations: auth
Running migrations:
Rendering model states... DONE
Applying contenttypes.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0001_initial... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK

This will prevent from the user authentication error, now we can run below command to sync database.

graphite-manage syncdb

Operations to perform:
Synchronize unmigrated apps: account, cli, render, whitelist, metrics, url_shortener, dashboard, composer, events, browser
Apply all migrations: admin, contenttypes, tagging, auth, sessions
Synchronizing apps without migrations:
Creating tables...
Creating table account_profile
Creating table account_variable
Creating table account_view
Creating table account_window
Creating table account_mygraph
Creating table dashboard_dashboard
Creating table events_event
Creating table url_shortener_link
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying admin.0001_initial... OK
Applying sessions.0001_initial... OK
Applying tagging.0001_initial... OK

You have installed Django's auth system, and don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'root'):
Email address: keshifs@gmail.com
Password:
Password (again):
Superuser created successfully.

Configuring Carbon

Now we are going to start configuration of Carbon, the Graphite storage backend by opening the service configuration file. But before that let's enable the carbon service to start at boot by making changed in the following file.

vim /etc/default/graphite-carbon

# Change to true, to enable carbon-cache on boot
CARBON_CACHE_ENABLED=true

Save and close this file and then open the Carbon configuration file using your favorite editor to make some minor changes in it.

vim /etc/carbon/carbon.conf

# Enable daily log rotation. If disabled, carbon will automatically re-open
# the file if it's rotated out of place (e.g. by logrotate daemon)
ENABLE_LOGROTATION = True

Save and close the file and move to the next step to configure storage Schemas.

Storage Schemas Configurations

Now, open the storage schema file to tells Carbon how long to store values and how detailed these values should be there.

vim /etc/carbon/storage-schemas.conf

There are two default section are present, let add another one with below mentioned parameters that we will be using lateron.

[test]
pattern = ^test\.
retentions = 5s:5m,1m:1h,5m:1d

So, when we request information from Graphite, it will return information from the most detailed archive that measures the time frame we're asking for. Let's save and close the file and come to the storage aggregation methods. We can define the way we want aggregation to occur in a file called storage-aggregation.conf . Copy the file from the Carbon examples directory into our Carbon configuration directory using command below.

cp /usr/share/doc/graphite-carbon/examples/storage-aggregation.conf.example /etc/carbon/storage-aggregation.conf

vim /etc/carbon/storage-aggregation.conf

[min]
pattern = \.min$
xFilesFactor = 0.1
aggregationMethod = min

[max]
pattern = \.max$
xFilesFactor = 0.1
aggregationMethod = max

[sum]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum

[default_average]
pattern = .*
xFilesFactor = 0.5
aggregationMethod = average

Save and close the file editor using ':wq!' and start Carbon service by executing the below.

systemctl start carbon-cache

Setup Apache Web server

Here we are going to install Apache Web server to use the web interface of Graphite, let's run the following command to install Apache and its dependent graphite components.

apt-get install apache2 libapache2-mod-wsgi

a2enmod wsgi  # [JHA] testen, ob das Modul aktiviert ist.

Once the installation is complete, disable the virtual host file, so that it may not conflicts.

a2dissite 000-default

Site 000-default disabled.
To activate the new configuration, you need to run:
  service apache2 reload

Before reloading Apache2, copy the Graphite Apache virtual host file to the available sites, enable virtual host file and then reload Apache web service by using below command.

cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available

a2ensite apache2-graphite

systemctl reload apache2

Install and Configure Statsd ([JHA] dies ist optional)

Graphite along with its major components has been installed and setup successfully, now we will be install StatsD to use statsd, we need to install node.js using below command.

apt-get install nodejs -y

Then install 'git' and clone a Statsd repository on your system '/opt' directory.

apt-get install git

cd /opt

git clone git://github.com/etsy/statsd.git

Once installed, create a new file within the cloned statsd directory and insert the following lines in it.

vim /opt/statsd/localConfig.js

{
graphitePort: 2003,
graphiteHost: "127.0.0.1",
port: 8125
}

Save and close the file and restart following services.

systemctl restart carbon-cache

Then move to statsd Directory and run below command.

cd /opt/statsd

node ./stats.js ./localConfig.js

That's it, let's move to its web interface.

Accessing Graphite Web Interface

All the required packages has been install with required configurations in place. Now its time to open your web browser to interact with the Graphite web interface. Simply use your FQDN or Server's IP and you will see its web page like shown below.

http://your_server_ip

Login from the top right corner, using the credentials you have setup earlier. Next drop down the tree and expand the matrices, see find and check your desired graph like shown below.

From the Graphite Composer you can choose a range of the graph that you wish to view. Similarly from the dashboard you can add or remove your custom graphs of your choice.

Conclusion

In this article you have learned about the installation and configuration of Graphite and StatsD on Ubuntu 16.04. Graphite is a highly scalable real-time graphing system. As a user, you write an application that collects numeric time-series data that you are interested in graphing, and send it to Graphite’s processing backend, carbon, which stores the data in Graphite’s specialized database. The data can then be visualized through graphite’s web interfaces. Graphite is designed to handle numeric time-series data. So, anybody who would want to track values of anything over time is highly recommended to use Graphite. Thank you for following our post , feel free to get back to us in case of any difficult or leave your valuable suggestions.

[JHA] Test

PORT=2003
SERVER=graphite.your.org
echo "local.random.diceroll 4 `date +%s`" | nc -q0 ${SERVER} ${PORT}