Indexima as Windows Service
Setup Indexima as a service
You can set up your Indexima console and your Indexima Core Engine to run as windows services.
Running as a Windows service allows Indexima to run in the background without interfering with the GUI. Running as a service permits also to be resilient to a reboot and to automatically restart INDEXIMA components.
This process involves a 3rd party program free of charge called NSSM. You can download this program here: http://nssm.cc/release/nssm-2.24.zip
Download from the above line the zip file and extract from nssm-2.24.zip\nssm-2.24\win64
the file nssm.exe
in a suitable location. You can run the file interactively or within a script to adjust and start services.
Ensure your Indexima console and core engine are running in the foreground on all concerned nodes to qualify batch files such as start-nodes.bat and start-visualdoop.bat.
To ease the installation, we describe a scripted installation, assuming Indexima console was installed in c:\indexima\visualdoop
and Indexima core engine in c:\indexima\galactica.
Indexima console
Open a Windows command prompt box with elevated rights. Run the following commands assuming NSSM.EXE
is reachable
nssm install indexima-console c:\indexima\visualdoop\start-visualdoop.bat
nssm set indexima-console AppDirectory c:\indexima\visualdoop
nssm set indexima-console DisplayName Indexima Console
nssm set indexima-console Description Indexima Console
nssm set indexima-console AppStdout c:\indexima\visualdoop\admin-svr.log
nssm set indexima-console AppStderr c:\indexima\visualdoop\admin-svr.log
All the other NSSM parameters are left to default.
You can review your setting interactively using the GUI of NSSM by using the following command:
nssm edit indexima-console
Then start the Indexima console by typing the following command:
nssm start indexima-console
indexima-console: START: The operation completed successfully.
You can review the admin-svr.log
file in your directory c:\indexima\visualdoop
as shown below:
c:\indexima\visualdoop>set VISUALDOOP_REVISION=1.6.9.389
Using war file visualdoop-webapp-1.6.9.389.war from arguments
Indexima Core Engine
Run the following commands assuming NSSM.EXE
is reachable
nssm install indexima-core c:\indexima\galactica\start-node.bat
nssm set indexima-core AppDirectory c:\indexima\galactica
nssm set indexima-core DisplayName Indexima Core Engine
nssm set indexima-core Description Indexima Core Engine
nssm set indexima-core AppStdout c:\indexima\galactica\cluster-svr.log
nssm set indexima-core AppStderr c:\indexima\galactica\cluster-svr.log
On the server having the role of INDEXIMA MASTER, add the following command to instruct this instance to start the HIVE2 service.
nssm set indexima-core AppParameters --master
All the other NSSM parameters are left to default.
You can review your setting interactively using the GUI of NSSM by using the following command:
nssm edit indexima-core
Then start the Indexima Core Engine by typing the following command:
nssm start indexima-core
indexima-core: START: The operation completed successfully.
You can review the cluster-svr.log
file in your directory c:\indexima\galactica
. The given example is the log file of the INDEXIMA master.
c:\indexima\galactica>set GALACTICA_REVISION=1.6.9.389
Log4j2 directory 'C:\indexima\log'
11:03:31,510 [ main] apache.hadoop.util.NativeCodeLoader.WARN - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting Indexima HiveServer2...
11:03:31 INFO [main] INDEXIMA_CONSOLE - [HIVE] Initializing HiveServer ...
11:03:31 INFO [main] INDEXIMA_CONSOLE - [HIVE] ThriftCLIService added...[BINARY]
11:03:36 INFO [main] INDEXIMA_CONSOLE - [HIVE] HiveServer running..
Standard Windows tools such as Computer Management
can be used to manage such Indexima services.
Always use the command c:\indexima\galactica\stop.bat
when stopping an Indexima Cluster
Troubleshoot Windows-based cluster
Winutils.exe has stopped working
Winutils.exe is the Hadoop binary required to run basic Hadoop calls when running the Indexima cluster on Windows.
The module in error or missing is MSVCR100.dll
. The best way to fix it is to download again Microsoft Visual C++ 2010 Redistributable Package
available at Microsoft: https://docs.microsoft.com/en-US/cpp/windows/latest-supported-vc-redist?view=msvc-160. The required file is vcredist_x64.exe
Wrong directories for logs and warehouse
The conf/galactica.conf
configuration file has been designed by default for Java Linux usage which uses the character /
as a directory separator. Under Windows, make sure you designate all your paths using a double backslash \\
as shown below:
galactica.conf
# Root of Indexima warehouse
warehouse = C:\\indexima\\warehouse
# Query history directory
history.dir = C:\\indexima\\history
# Log directory
log.dir = C:\\indexima\\log
# Hive log directory
hive.log.dir = C:\\indexima\\hivelog
However, this path separator is still a simple backslash \
in the batch file describing the INDEXIMA environment as shown below:
galactica-env.bat
REM Path to our java/jre
set JAVA_HOME="C:\Program Files (x86)\Java\jre1.8.0_201"
REM Path to your Hadoop installation
set HADOOP_BASE="C:\hadoop-2.8.3"
Hive Server won't start
When starting INDEXIMA using the command start-node.bat --master
, the console restarts permanently the INDEXIMA HIVE server as shown below:
Starting Indexima HiveServer2...
INFO [main] INDEXIMA_CONSOLE - [HIVE] Initializing HiveServer ...
INFO [main] INDEXIMA_CONSOLE - [HIVE] ThriftCLIService added...[BINARY]
INFO [main] INDEXIMA_CONSOLE - [HIVE] Initializing HiveServer ...
INFO [main] INDEXIMA_CONSOLE - [HIVE] ThriftCLIService added...[BINARY]
INFO [main] INDEXIMA_CONSOLE - [HIVE] Initializing HiveServer ...
INFO [main] INDEXIMA_CONSOLE - [HIVE] ThriftCLIService added...[BINARY]
By setting hive in debug mode in galactica.conf
as shown below:
galactica.conf
# Hive log directory
hive.log.dir = C:\\indexima\\hivelog
# Hive log thresold
hive.log.threshold = INFO
The resulting log file displays the following errors:
09:16:55,087 INFO [main]: server.HiveServer2 (HiveServer2.java:stop(313)) - Shutting down HiveServer2
09:16:55,091 INFO [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(376)) - Exception caught when calling stop of HiveServer2 before retrying start
09:16:55,091 WARN [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(384)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds
java.lang.RuntimeException: Error applying authorization policy on hive configuration: java.lang.RuntimeException: Error while running command to get file permissions
Error while running command to get file permissions
This is due to the fact that INDEXIMA does not have write permissions to write in a temporary directory.
- Add the following property in
conf/hive-site.xml
. Note the double backslash\\
when defining the path of the scratch directory.
<property>
<name>hive.exec.scratchdir</name>
<value>C:\\indexima\\scratchdir</value>
<description>Scratch space for Hive jobs</description>
</property>
- Then add write permissions to such directory
C:\>cd \hadoop-2.8.3\bin
C:\hadoop-2.8.3\bin>winutils.exe ls \indexima\scratchdir
drwx------ 1 BUILTIN\Administrators DESKTOP-CALI\robert 0 Jan 22 2039 \indexima\scratchdir
C:\hadoop-2.8.3\bin>winutils.exe chmod 777 \indexima\scratchdir
C:\hadoop-2.8.3\bin>winutils.exe ls \indexima\scratchdir
drwxrwxrwx 1 BUILTIN\Administrators DESKTOP-CALI\robert 0 Jan 22 2039 \indexima\scratchdir
- Restart the INDEXIMA cluster to have modifications take effect.