Applications Handling

The purpose of this document is to give Griaule application/services and required components basic knowledge to all users. After reading the document, the user should be able to manage the applications and perform basic troubleshooting.

Architecture

It is possible to simplify the GBDS architecture to four main components and the Integrator application. Each of those components communicates with each other as shown in the figure below. Those components are:

• GBDS: API, Matcher, and Notifier. Griaule main Applications.

• Services: ETR, Cardscan, and BEST. Client and Web services that will interact directly with Griaule apps.

• RDB: Relational Database. Used to store reference metadata.

• Hadoop: Collection of Open-source software designed for distributed storage and big data processing. Also, handles the communication between Griaule apps.

• Integrator: Client application used to perform specific activities or customizations using Griaule apps.

Directories and Commands

This section will show the directories where the files are located, the initialization commands, and how to perform health checks.

GBDS Management

The GBDS is composed of the GBDS Api, the Marcher, and the Notifier.

API Handling

Matcher Handling

Notifier Handling

Health Checks

This section describes some methods to monitor or troubleshoot the applications.

• Confirming the application started successfully.

At the terminal, you should see lines similar to:

API:

{date} {time},257 [INFO ] Http11NioProtocol - Starting ProtocolHandler ["http-n io-8085"]
{date} {time},262 [INFO ] NioSelectorPool - Using a shared selector for servlet write/reading
{date} {time},823 [INFO ] API - Started API in {time} seconds (JVM running for {time})

Matcher:

{date} {time},180 [INFO ] TaskConsumer - Starting consumer using consumer group: spt-02-gbds-tasks
{date} {time},182 [INFO ] PriorityConsumer - subscribed to gbds-tasks-GOD_PRIORITY
{date} {time},184 [INFO ] PriorityConsumer - subscribed to gbds-tasks-HIGHEST_PRIORITY
{date} {time},186 [INFO ] PriorityConsumer - subscribed to gbds-tasks-HIGHER_PRIORITY
{date} {time},187 [INFO ] PriorityConsumer - subscribed to gbds-tasks-HIGH_PRIORITY
{date} {time},189 [INFO ] PriorityConsumer - subscribed to gbds-tasks-DEFAULT_PRIORITY
{date} {time},191 [INFO ] PriorityConsumer - subscribed to gbds-tasks-LOW_PRIORITY
{date} {time},192 [INFO ] PriorityConsumer - subscribed to gbds-tasks-LOWER_PRIORITY
{date} {time},197 [INFO ] PriorityConsumer - subscribed to gbds-tasks-LOWEST_PRIORITY

Notifier:

{date} {time},418 Notifier - group-ETR - notifier-$a - creating notifier
{date} {time},424 NotificationConsumer - group-ETR - notification-supervisor-group-ETR is ready. Start consuming

• Check if application procces is running.

ps -ef | grep gbsapi | grep -v grep
ps -ef | grep matcher | grep -v grep
ps -ef | grep gbsnotifier | grep -v grep

• Check if logs are being updated properly with the latest/current timestamps.

tailf /var/log/griaule/gbsapi/gbsapi.log
tailf /var/log/griaule/gbds/gbds.log
tailf /var/log/griaule/gbsnotifier/notifier.log

• Check if API is processing properly (CURL/HTTP).

API:

curl -v http://localhost:8085/gbds/v2/operations/ping

DB Coneection:

curl -v http://localhost:8085/gbds/v2/exceptions/EndDate=1400000000000

Services Management

Composed by Best Server, CardScan Web, and ETR Web.

Best Handling

CardScan Handling

ETR Handling

Health Checks

This section describes some methods to monitor or troubleshoot the services.

• Confirming the services started successfully.

At the terminal, you should see lines similar to:

BEST:

{date} {time} spt02 server[5860]: {date} {time} org.apache.coyote.AbstractProtocol start
{date} {time} spt02 server[5860]: INFO: Starting ProtocolHandler ["http-nio-8123"]
{date} {time} spt02 server[5860]: {date} {time} org.apache.catalina.startup.Catalina start
{date} {time} spt02 server[5860]: INFO: Server startup in {time} ms
{date} {time} spt02 java[5860]: CaptureManager: waiting for some USB event...
{date} {time} spt02 server[5860]: {date} {time} [Pooling thread] INFO com.griaule.best.service.SearchService - Start pooling verification of pending/processing searchs
{date} {time} spt02 server[5860]: {date} {time} [Pooling thread] INFO com.griaule.best.service.SearchService - End of polling: # searches checked.

CardScan:

{date} {time} spt02 server[3657]: {date} {time} org.apache.coyote.AbstractProtocol start
{date} {time} spt02 server[3657]: INFO: Starting ProtocolHandler ["http-nio-8087"]
{date} {time} spt02 server[3657]: {date} {time} org.apache.catalina.startup.Catalina start
{date} {time} spt02 server[3657]: INFO: Server startup in (time) ms
{date} {time} spt02 java[3657]: CaptureManager: waiting for some USB event...
{date} {time} spt02 server[3657]: {date} {time} [Profile queue thread] INFO com.griaule.cardscan.service.ProfileService - Get Next [GBDS_READY, GBDS_OK] profiles to process

ETR:

{date} {time} spt02 server[790]: {date} {time} org.apache.coyote.AbstractProtocol start
{date} {time} spt02 server[790]: INFO: Starting ProtocolHandler ["http-nio-8089"]
{date} {time} spt02 server[790]: {date} {time} org.apache.catalina.startup.Catalina start
{date} {time} spt02 server[790]: INFO: Server startup in {time} ms

• For ETR and CardScan web, if you just installed the service or if you removed the service server directory for a redeploy, after a successful start, run the setup script for some additional configuration.

/var/lib/tomcats/setup.sh <app>

Where <app> can be etr or cardscan.

Relational Database Management

Composed by MySQL or MariaDB.

Database Handling

Health Check

This section describes some methods to monitor or troubleshoot the database operation.

• Confirm service is up and running.

Execute the status operation in the terminal. It is expected something similar to:

[user] service mysqld status
Redirecting to /bin/systemctl status mysql.service
 mariadb.service - MariaDB {version} database server
    Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/mariadb.service.d
            migrated-from-my.cnf-setting.conf
    Active: active (running) since {date} {time}; {past time}
    Docs: mand:mysql(8)
        https://mariadb.com/kb/en/library/systemd/
 Maind PID: 15061 (mysqld)
    Status: "Taking your SQL request now..."
    Cgroup: /system.slice/mariadb.service
            15061 /usr/sbin/mysqld --basedir=/usr

• Check if service process is runniong.

ps -ef | grep mysqld | grep -v grep

• Check if there are any errors in the logs.

less /var/log/messages | grep mysqld

GBDS will NOT automatically reconnect.

If any configuration change was made, restart it to take place.

Hadoop Management

Hortonworks Data Platform (HDP) services used in GBDS is composed of 6 services, each with a specific function:

• HDFS - Distributed Filesystem.

• HBASE - Non-Relational database for Big Data.

• KAFKA - Queue System. Handles data transfer between apps.

• ZOOKEEPER - Handle all HDP services management and synchronization.

• RANGER + SOLR - Security layer. Handle user authentication and audit.

All these services are managed by a graphical user interface (GUI) named AMBARI.

AMBARI Server runs in the master node, AMBARI Agents (Client) run on all servers.

HDP Services Handling

This section will describe how to handle the services with the AMBARI GUI. It is important to notice that all services are sensible, and some critical services like HBASE and HDFS may take some time to stabilizes, especially in big clusters. This is important when initializing or restarting the services.

• Access AMBARI GUI by http://<hostname>:8080/#/login

  The default username and password are both admin.

• Service Information is displayed in the left sidebar.

  

  • Green means the service is up and running.

  • Red means the service is stopped.

  • A red dot with numbers on the left side means alerts were found for the specified service.

  • If a service is in maintenance mode (suitcase with + symbol in the middle), the alerts will be suppressed.

• Start, Stop and Restart services.

  To start/stop/restart the service, the user can click on the ... symbol at the left sidebar, options to start, stop, or restart all services will be displayed.

  

  To start, stop, or restart one service at a time, click in the service and the upper right, click on the ACTIONS button, then select the desired action.

  

• Following Background Process.

  To follow the background processes, click on the engine icon in the top menu bar.

AMBARI Handling

To start, stop or verify server or agent status, follow the command described in the table below.

Logs

Since HDP Services are open-source, the user can find the documentation online. If there is an issue, to help troubleshooting, follow some logs locations: