Quick set-up

In the following the set-up process is detailed how to remedy an incident that is described on use case page (please click on the Short explanation button below the video).

Installation

engine

First install the engine then alter the following settings in the [install dir]/conf/reaction-engine-application.yml config file:

• database counfiguration details: the settings in spring.datasource.XXX
• mail counfiguration details: the settings in spring.mail.XXX
• endpoint of the management app: the reaction.management_web_app.endpoint setting

Then set up the passwords of the workers and the management web application in the [install dir]/security/credentials file. After the configuration start the engine with ./reaction_engine.sh start.

management web application

First install the management web application then alter the following settings in the [install dir]/management_app/settings_reaction.py config file:

• DATABASES: change the values as per the database access details
• REACTION_ENGINE_REST_URL: the endpint of the reaction engine, e.g. http://ACME-SRV01:7003/reaction-engine
• REACTION_REST_AUTH_PUBLIC_KEY: the user name that will be used in the credential file of the engine
• REACTION_REST_AUTH_PRIVATE_KEY: the password that will be used in the credential file of the engine

After the configuration start the apache server.

worker

After the installation of the worker on ACME00 please set the following two configuration settings in the /local/reaction/worker-1.1/conf/worker.yml:

• host_name: use the value ACME00, this value will be used in the execution flow to refer to the worker
• rest_call.engine_endpoint: the endoint of the engine, e.g. http://ACME-SRV01:7003/reaction-engine

After the configuration start the reader worker with ./reaction_reader.sh start and the the executor worker with ./reaction_executor.sh start. Do the same on ACME01 too.

Configuration in the management web application - creating the reference data

What? How? When? All has to be done is to answer to these 3 questions: What to observe? How to fix the problem? When to start the remedy process? What to observe? It is simple, the log file of the business application has to be observed. The business application (the system) is modeled in the management web application so that is the reason why the type of the reference data is called system. How to fix the problem? A chain of OS commands can be specified that is called the execution flow. When to start the remedy process? When an error (in our case the java.lang.OutOfMemoryError: Java heap space) appears in the observed log. A so-called error detector has to be created that will bind the system reference data with the execution flow and the message pattern.

creating the systems

The 2 running instances of Hermes application are almost the same so it is recommended to create a common parent (the management web application can handle hierarchy of systems) where the matching properties can be specified. Hermes [GROUP] - specify the log file location, maintenance window
    Hermes Host 0 [LOG FILE] - specify the host name (ACME00)
    Hermes Host 1 [LOG FILE] - specify the host name (ACME01)

creating the execution flow

The execution flow defines the OS (Linux/Window) commands that will be executed to fix the issue. The following flow is created that contains the OS commands (green), email sending (gray), and if-else operations (yellow).

creating the error detector

The error detector binds the system reference data with the execution flow and the message pattern (it is a regular expression, like .*java\.lang\.OutOfMemoryError: Java heap space.*). A hierarchy of systems is created in the first step of the reference data creation so only the parent system (Hermes [GROUP]) has to be assigned to the detector. It means that all its children systems will be observed too (the log files of host ACME00 and ACME01 will be monitored). So when the text java.lang.OutOfMemoryError: Java heap space appears in the log file on either host then the flow will be executed.