Indexima as Windows Service

Setup Indexima as a service

You can set up your Indexima console and your Indexima Core Engine to run as windows services.

Running as a Windows service allows Indexima to run in the background without interfering with the GUI. Running as a service permits also to be resilient to a reboot and to automatically restart INDEXIMA components.

This process involves a 3rd party program free of charge called NSSM. You can download this program here: http://nssm.cc/release/nssm-2.24.zip

Download from the above line the zip file and extract from nssm-2.24.zip\nssm-2.24\win64 the file nssm.exe in a suitable location. You can run the file interactively or within a script to adjust and start services.

Ensure your Indexima console and core engine are running in the foreground on all concerned nodes to qualify batch files such as start-nodes.bat and start-visualdoop.bat.

To ease the installation, we describe a scripted installation, assuming Indexima console was installed in c:\indexima\visualdoop and Indexima core engine in c:\indexima\galactica.

Indexima console

Open a Windows command prompt box with elevated rights. Run the following commands assuming NSSM.EXE is reachable

CODE

nssm install indexima-console c:\indexima\visualdoop\start-visualdoop.bat
nssm set indexima-console AppDirectory c:\indexima\visualdoop
nssm set indexima-console DisplayName Indexima Console
nssm set indexima-console Description Indexima Console
nssm set indexima-console AppStdout c:\indexima\visualdoop\admin-svr.log
nssm set indexima-console AppStderr c:\indexima\visualdoop\admin-svr.log

All the other NSSM parameters are left to default.

You can review your setting interactively using the GUI of NSSM by using the following command:

CODE

nssm edit indexima-console

Then start the Indexima console by typing the following command:

CODE

nssm start indexima-console
indexima-console: START: The operation completed successfully.

You can review the admin-svr.log file in your directory c:\indexima\visualdoop as shown below:

CODE

c:\indexima\visualdoop>set VISUALDOOP_REVISION=1.6.9.389 
Using war file visualdoop-webapp-1.6.9.389.war from arguments

Indexima Core Engine

Run the following commands assuming NSSM.EXE is reachable

CODE

nssm install indexima-core  c:\indexima\galactica\start-node.bat
nssm set indexima-core AppDirectory c:\indexima\galactica
nssm set indexima-core DisplayName Indexima Core Engine
nssm set indexima-core Description Indexima Core Engine
nssm set indexima-core AppStdout c:\indexima\galactica\cluster-svr.log
nssm set indexima-core AppStderr c:\indexima\galactica\cluster-svr.log

On the server having the role of INDEXIMA MASTER, add the following command to instruct this instance to start the HIVE2 service.

CODE

nssm set indexima-core AppParameters --master

All the other NSSM parameters are left to default.

You can review your setting interactively using the GUI of NSSM by using the following command:

CODE

nssm edit indexima-core

Then start the Indexima Core Engine by typing the following command:

CODE

nssm start indexima-core
indexima-core: START: The operation completed successfully.

You can review the cluster-svr.log file in your directory c:\indexima\galactica. The given example is the log file of the INDEXIMA master.

CODE

c:\indexima\galactica>set GALACTICA_REVISION=1.6.9.389 
Log4j2 directory 'C:\indexima\log'
11:03:31,510 [       main] apache.hadoop.util.NativeCodeLoader.WARN  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting Indexima HiveServer2...
11:03:31  INFO [main] INDEXIMA_CONSOLE - [HIVE] Initializing HiveServer ...
11:03:31  INFO [main] INDEXIMA_CONSOLE - [HIVE] ThriftCLIService added...[BINARY]
11:03:36  INFO [main] INDEXIMA_CONSOLE - [HIVE] HiveServer running..

Standard Windows tools such as Computer Management can be used to manage such Indexima services.

Always use the command c:\indexima\galactica\stop.bat when stopping an Indexima Cluster

Troubleshoot Windows-based cluster

Winutils.exe has stopped working

Winutils.exe is the Hadoop binary required to run basic Hadoop calls when running the Indexima cluster on Windows.

The module in error or missing is MSVCR100.dll. The best way to fix it is to download again Microsoft Visual C++ 2010 Redistributable Package available at Microsoft: https://docs.microsoft.com/en-US/cpp/windows/latest-supported-vc-redist?view=msvc-160. The required file is vcredist_x64.exe

Wrong directories for logs and warehouse

The conf/galactica.conf configuration file has been designed by default for Java Linux usage which uses the character / as a directory separator. Under Windows, make sure you designate all your paths using a double backslash \\ as shown below:

galactica.conf

CODE

# Root of Indexima warehouse
warehouse = C:\\indexima\\warehouse

# Query history directory
history.dir = C:\\indexima\\history
# Log directory
log.dir = C:\\indexima\\log
# Hive log directory
hive.log.dir = C:\\indexima\\hivelog

However, this path separator is still a simple backslash \ in the batch file describing the INDEXIMA environment as shown below:

galactica-env.bat

CODE

REM Path to our java/jre
set JAVA_HOME="C:\Program Files (x86)\Java\jre1.8.0_201"

REM Path to your Hadoop installation
set HADOOP_BASE="C:\hadoop-2.8.3"

Hive Server won't start

When starting INDEXIMA using the command start-node.bat --master, the console restarts permanently the INDEXIMA HIVE server as shown below:

CODE

Starting Indexima HiveServer2...
INFO [main] INDEXIMA_CONSOLE - [HIVE] Initializing HiveServer ...
INFO [main] INDEXIMA_CONSOLE - [HIVE] ThriftCLIService added...[BINARY]
INFO [main] INDEXIMA_CONSOLE - [HIVE] Initializing HiveServer ...
INFO [main] INDEXIMA_CONSOLE - [HIVE] ThriftCLIService added...[BINARY]
INFO [main] INDEXIMA_CONSOLE - [HIVE] Initializing HiveServer ...
INFO [main] INDEXIMA_CONSOLE - [HIVE] ThriftCLIService added...[BINARY]

By setting hive in debug mode in galactica.conf as shown below:

galactica.conf

CODE

# Hive log directory
hive.log.dir = C:\\indexima\\hivelog
# Hive log thresold
hive.log.threshold = INFO

The resulting log file displays the following errors:

CODE

09:16:55,087 INFO  [main]: server.HiveServer2 (HiveServer2.java:stop(313)) - Shutting down HiveServer2
09:16:55,091 INFO  [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(376)) - Exception caught when calling stop of HiveServer2 before retrying start
09:16:55,091 WARN  [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(384)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds
java.lang.RuntimeException: Error applying authorization policy on hive configuration: java.lang.RuntimeException: Error while running command to get file permissions
Error while running command to get file permissions

This is due to the fact that INDEXIMA does not have write permissions to write in a temporary directory.

Add the following property in conf/hive-site.xml . Note the double backslash \\ when defining the path of the scratch directory.

CODE

    <property>
        <name>hive.exec.scratchdir</name>
        <value>C:\\indexima\\scratchdir</value>
        <description>Scratch space for Hive jobs</description>
    </property>

Then add write permissions to such directory

CODE

C:\>cd \hadoop-2.8.3\bin
C:\hadoop-2.8.3\bin>winutils.exe ls \indexima\scratchdir
drwx------ 1 BUILTIN\Administrators DESKTOP-CALI\robert 0 Jan 22 2039 \indexima\scratchdir

C:\hadoop-2.8.3\bin>winutils.exe chmod 777 \indexima\scratchdir

C:\hadoop-2.8.3\bin>winutils.exe ls \indexima\scratchdir
drwxrwxrwx 1 BUILTIN\Administrators DESKTOP-CALI\robert 0 Jan 22 2039 \indexima\scratchdir

Restart the INDEXIMA cluster to have modifications take effect.