EDIT 2014 - Server Installation Guide

/ EUROPEAN COMMISSION
DIRECTORATE-GENERAL
INFORMATICS
Information systems Directorate

European Commission

Specific Contract Fixed Price012038

"EDIT 2014 - Server Installation Guide"
under Framework Contract DI/06770-00

Date: / 19/05/2016
Version: / 02.50
Authors: / Gheorghe MORARU, Bogdan TUDOR
Revised by: / Gheorghe MORARU, Charis MICHOPOULOS
Approved by: / Theodoros MIGLAKIS
Public: / No
Reference Number: / D2.2 & D2.7

TABLE OF CONTENTS

1. Scope Introduction......

1.1. Purpose of the Document......

1.2. Scope of the Document......

1.3. Intended Readership......

2. Reference and Applicable documents......

2.1. Reference Documents......

2.2. Applicable Document......

3. Environment Requirements......

3.1. Server Operating System......

3.2. Application Server......

3.3. Database......

4. Preparation of the Installation......

4.1. Installation Files......

4.2. Database Setup......

4.2.1. Scripts Overview......

4.2.2. User Setup......

4.2.3. Folder and Packages Setup......

4.2.4. Table and Sequences Setup......

4.2.5. Procedures......

4.2.6. Reference Data Setup......

4.3. Application Server Configuration......

4.3.1. Auxiliary Folders......

4.3.2. Application Server Settings......

4.3.3. Create a datasource......

4.3.4. Create the JMS Resources......

4.3.5. JNDI Security......

5. EDIT configuration......

6. Deployment......

7. Build Instructions......

Document History

Version / Date / Comment / Modified Pages
00.10 / 14/12/2011 / Initial Version / -
00.20 / 16/12/2011 / Internal Review / -
01.00 / 21/12/2011 / Initial formal release for EBB2011 / -
01.10 / 08/06/2012 / Updated for EBB2012 application
01.20 / 21/11/2012 / Updated ebb-edit.properties for SMS, External contacts / 8,9
01.30 / 14/06/2013 / Updated for EDIT 2013, removed EBB naming to EDIT
01.40 / 16/12/2013 / Updated with JMS properties, executor-app
01.50 / 13/03/2014 / Updated with confidentiality disclaimer and news section / 10
01.51 / 26/03/2014 / Updated with reference to WLST scripts / 4
01.60 / 07/04/2014 / Added edit.contentImagePath / 6, 8
01.61 / 08/04/2014 / Added missing properties from ebb-edit.properties (edit.maxVolume – maxUploadSize) and ebb-database.properties (jms/db) / 8, 9
02.00 / 02/07/2014 / Updated for EDIT 2014 / Title, Footer,Referemce documents, minor phrasing changes in pages 8-12
02.10 / 02/12/2014 / Added sdmx keyfamilies property, index file retry / 11
02.20 / 05/03/2015 / Added jvm minimum required version for https / 4
02.30 / 11/09/2015 / Added WLST scritps / 7
02.40 / 29/04/2016 / Updated Environment Requirements, small modification in the WLST scripts / 5, 8
02.50 / 19/05/2016 / Added section 7 Build Instructions / 14

"EDIT 2014 - Server Installation Guide" - Page 1

Document Version 02.50.date 19/05/2016

1.Scope Introduction

1.1.Purpose of the Document

This document depicts in detail all steps necessary for the installation of the EDIT 2014 application in server mode. This application has two components: Web Client (edit-web-client) and Web Services (edit-services-web).

1.2.Scope of the Document

This document provides the necessary details in order to install the EDIT 2014 application.

1.3.Intended Readership

• The Customer
• Technical Managers
• Software Architects
• Developers

2.Reference and Applicable documents

2.1.Reference Documents

ID / Reference / Title / Version
R1 / DI/06770-00 / Framework Contract ESP DESIS II / 14/09/2010

Table 2- 1: Reference documents

2.2.Applicable Document

ID / Reference / Title / Version
A1 / 11131.2013.999-2013.539 / Specific Contract Fixed Price No 012038 – CC DI.06770 / 04/12/2014

3.Environment Requirements

Before getting started with the installation, the following requirements concerning the server’s environment must be met. The usage of those products/technologies, documented subsequently, is recommended, as all applications have been implemented and tested with these products/technologies. Other products or system environments have not been evaluated and can therefore not be recommended for application. Due to the usage of well known products, technologies and standards, a high degree of compatibility to other environments is possible.

3.1.Server Operating System

Any OS that supports running a Java 7 Virtual Machine.

3.2.Application Server

Weblogic Application Server 12.1.2.

3.3.Database

A standard Oracle 10g database is needed. The database should use AL32UTF8 as character set (NLS_CHARACTERSET) and AL16UTF16 as national character set (NLS_NCHAR_CHARACTERSET).

4.Preparation of the Installation

4.1.Installation Files

Deployable:

• edit-web-app.ear
• edit-services-app.ear
• edit-executor-app.ear

Deployment configuration (JAR file):

• edit-properties.jar

DB Scripts: these are described in the chapter 4.2 (Database Setup).

WSLT Scripts:

• datasource.py – setup the database data source
• jms.py – setup the JMS resources
• users.py – setup the JNDI credentials
• policies.py – setup the security policies

Dependencies Folder (<EDIT Repository>/deps as 4.3.1):

• edit-services-1.0-SNAPSHOT.jar
• edit-services-impl-1.0-SNAPSHOT.jar
• log4j-1.2.14.jar
• slf4j-api-1.5.8.jar
• slf4j-log4j12-1.5.8.jar

Lib Folder (Weblogic domain lib folder):

• edit-properties.jar – to be added after the environment specific configuration is done
• log4j-1.2.14.jar
• ojdbc-14.jar
• slf4j-api-1.6.1.jar
• slf4j-log4j12-1.6.1.jar

4.2.Database Setup

4.2.1.Scripts Overview

The following scripts are used to initialize the Edit database:

• 01_oracle_grants.sql
• 02_oracle_create_tables.sql
• 03_oracle_create_sequences.sql
• 04_oracle_create_packages.sql
• 05_oracle_create_package_bodies.sql
• 06_oracle_create_procedures.sql
• 07_oracle_init.sql
• 08_oracle_init_permissions.sql
• 09_oracle_init_translations.sql
• 10_oracle_init_templates.sql
• 20_oracle_10g_wm_concat_function.sql

4.2.2.User Setup

A user (EDIT Oracle user) should be created in order to allow the application to access the database. The user should havethe following privileges:

• connect to the database
• create, alter and drop for the tables, views, indexes, foreign keys and sequences in his default schema.
• select, insert, update, delete on any table in his default schema

4.2.3.Folder and Packages Setup

A sys should modify and run the oracle_grants.sql in order to create the Oracle directory and grant the rights over this directory and the UTL_FILE Oracle package. The grants are granted to the edit2013 Oracle user. The referenced directory must be already present in the system and the name in Oracle must be EDIT_DATA_DIR. All files the data pump location folder must be accessible with read/write permissions by system users that launch the oracle database.

• 01_oracle_grants.sql

4.2.4.Table and Sequences Setup

For setting up the database tables, the database administrator must run the following scripts:

• 02_oracle_create_tables.sql
• 03_oracle_create_sequences.sql

4.2.5.Procedures

Before setting up the reference data, a set of stored procedures need to be defined in the database to simplify the insertion of translations and permissions:

• 04_oracle_create_packages.sql
• 05_oracle_create_package_bodies.sql
• 06_oracle_create_procedures.sql
• 20_oracle_10g_wm_concat_function.sql

4.2.6.Reference Data Setup

Before executing the script, the database administrator must set the NLS_LANG environment variable, so that the sql scripts containing UTF8 characters execute correctly. In order to populate the tables with the reference data, the database administrator or the EDIT DB user(created in the User Setup paragraph) must run the following scripts:

• 07_oracle_init.sql
• 08_oracle_init_permissions.sql
• 09_oracle_init_translations.sql
• 10_oracle_init_templates.sql

4.3.Application Server Configuration

4.3.1.Auxiliary Folders

A file repository folder must be accessible by the web services application in order to store data files used for the import/export process and to resolve dependencies for the compilation process during the job execution.

The following folder structure needs to be created:

directory / Description
<EDIT Repository> / root directory of the files repository
<EDIT Repository>/img / Folder storing the image files for the news section
<EDIT Repository>/repo / Folder storing the import/export data files
<EDIT Repository>/deps / Folder storing the dependencies necessary for the compilation process
<EDIT Repository>/generated / Folder storing the java classes which are generated by the compilation process
<EDIT Repository>/tramo / Folder storing the tramo executable file

The files from the distribution folder <File Deps> need to be copied into the newly created file deps folder (e.g. /edit/deps).

The tramo executable file from the distribution folder <Tramo> needs to be copied into the newly created tramo folder (e.g. /edit/tramo).

The location of these folders needs to be configured in the application as described in the configuration chapter below.

Note: the following configurations below can be also created by executing the provided scripts located under .. Software Release (Packaged files & Sources)\Server\WLST.

In order to execute .py files from that location the following commands should be used (Windows environment example):

call %WL_HOME%/server/bin/setWLSEnv

java weblogic.WLST users.py

java weblogic.WLST policies.py

java weblogic.WLST datasource.py

java weblogic.WLST jms.py

where %WL_HOME% should point to the Oracle Middleware installation folder (eg: C:\Middleware\wlserver)

4.3.2.Application Server Settings

The files from 4.1 lib folder must be copied into the lib folder of the ebb Weblogic server.

The file named bin/startWebLogic.sh must be updated with the following content:

MEM_ARGS="-Xmx2000m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=4

The –Xmx value should be customize in accordance to the number of paraller jobs intended to run on the server. The actual value to be set depends on many factors and should be fine-tuned, but a value of 1G per job is recommended.

4.3.3.Create a datasource.

A JDBC Datasource needs to be created with the following settings:

Name: Oracle Datasource for EDIT

JNDI Name: editDB

Database Type: Oracle

Database Driver: Oracle Thin Driver

The default options should be selected (Supports Global Transactions and One-phase Commit), and the following connection properties filled in:

Property / Description / Example value
database name / name of the database / edit
host name / IP address of the database / 10.30.40.5
port / database connection port / 1521
database user name / user name of the user created during database setup / edit
password / password of the user created during database setup / edit

4.3.4.Create the JMS Resources.

The following JMS resources need to be created:

Name / Type / Description
ConnectionFactory-0 / Connection Factory / EditConnectionFactory
Queue-0 / Queue / EditExecutionQueue
Queue-1 / Queue / Dataset Instance ManagerQueue
Topic-0 / Topic / EditCancellationTopic
Topic-1 / Topic / JobMeasureTopic

4.3.5.JNDI Security.

All JNDI resources should be protected using security policies. The credentials for a user with access to the JNDI tree must be then configured in the properties files of the application as described later on in the documentation.

WSLT Scripts are provided as part of the installation files.

5.EDITconfiguration

The following properties files are used by the EDIT application:

• ebb-database.properties
• ebb-dbSpecific.properties
• ebb-edit.properties
• ebb-server.properties
• ebb-webclient.properties
• ecas-config-edit-services-web.properties
• ecas-config-edit-web-client.properties

The following table contains all configuration options for the file ebb-database.properties:

property name / description / example/allowed values
jndi.name / database jndi / java:comp/env/jdbc/editDB
jndi.db.user / Jndi username for the database / jndiUsername
jndi.db.password / Jndi password for the database / jndiPassword
jndi.db.initial / Initial Context factory Class for the database / weblogic.jndi.WLInitialContextFactory
jndi.jms.user / Jndi username for the JMS resources / jndiUsername
jndi.jms.password / Jndi password for the JMS resources / jndiPassword
jndi.jms.initial / Jndi password for the JMS resources / weblogic.jndi.WLInitialContextFactory

The ebb-dbSpecific.properties file is configured for deploying with Oracle and should not be changed unless deploying with a PostgreSQL database.

The following table contains all configuration options for the file ebb-edit.properties:

property Name / description / example/allowed values
edit.repositoryPath / Data files repository path / /edit/repo
edit.contentImagePath / News images repository path / /edit/contentImages
grammar.path / path where the classes are generated / /edit/generated
grammar.prefix / prefix of the generated classes
(do not change it ) / ec.estat.edit.engine.
codemanager.generated
grammar.dependency
Path / path where are the dependencies / /edit/deps
grammar.javacPath / path to the JDK / /eurostat/documents/wl/EbbNonSec/javac
edit.batchSize / how many sql inserts are done in a batch / 50
edit.pageSize / how many records are being fetched at a time from the database / 25000
edit.maxRowsGeneratedPerBlock / The maximum number of records in the memory before the result is written in the database / 25000
edit.verticalBlockFetchSize / how many rows are fetched for vertical data blocks / 10000
edit.parallelThreads / how many parallel jobs are executed / 2
edit.maxExecutionTime / Maximum execution time for a job in seconds / 3600
edit.maxRowsGeneratedInCustomOperation / Number of rows cached before writing to the DB in the execution of a custom dataset operation / 10000
edit.maxVolume / Index used to compute the size of the insert batches / 2000000
edit.partitionSizeCounter / Define interval of partitions when the memory is checked / 1000
edit.minFreeMemory / Threashold where the system will force garbage collection and check if the system is running out of memory / 2000000
edit.maxUploadSize / Maximum size allowed for a file upload / 2000000000
tramo.path / Path to the tramo folder / /edurostat/documents/wl/ebb/tramo
tramo.program / Tramo executable file / Tramo.sh
grammar.precision / BigDecimal divide precision / 10
edit.db.import.pagesize / Page size for reading from external DB / 10000
edit.graphics.dot.limitation / Maximum number of dots for a graphic / 300
edit.graphics.partition.limitation / Maximum number of values inside a partition / 100
edit.detail.statistical.rep.endline / Detailed statistics report end line / ======END REPORT ======
edit.edamis.url / EDAMIS Feedback WS URL /
edit.edamis.service / for the real service use edamisFeedbackClientServiceTarget
for the mock service use edamisFeedbackClientServiceMock / edamisFeedbackClientServiceTarget
edit.edamis.organization / Value to be set in the EDAMS WS call / ESTAT
edit.edamis.sender / Value to be set in the EDAMS WS call / ESTAT
edit.edamis.applicationName / Value to be set in the EDAMS WS call / EBB
edit.export.edamis.filetype / Type of file to be generated for the EDAMIS export / CSV
edit.email.server.host / Host of the email server / mail.server.estat
edit.email.server.port / Port of the email server / 25
edit.email.server.user / User for the email server / User
edit.email.server.pass / Password for the email server / Password
edit.email.from / From address to be used for the notification /
sms.enabled / Flag indicating whether sms services are available / 0 (disabled)
1 (enabled)
sms.emailDomain / The domain of the sms server / ecsms.eu
external.contacts.sms.sender / The sender name for the smses / EUROSTAT
external.contacts.email.sender / The sender of the emails sent to external contacts /
external.contacts.verificationEmail.subject / The subject of the emails with the verification code. / Email address verification
external.contacts.passwordEmail.subject / The subject of the emails with the password for opening
edit.sdmxregistry.url / URL for the SDMX Registry /
edit.sdmxregistry.service / for the real service use sdmxRegistryPortTypeTarget
for the mock service use sdmxRegistryPortTypeTargetMock / sdmxRegistryPortTypeTarget
edit.sdmxregistry.registryAgencyId / Value to be set in the SDMX Registry WS call / ESTAT
edit.sdmxregistry.registrySenderId / Value to be set in the SDMX Registry WS call / ESTAT
edit.sdmxregistry.registryUser / SDMX Registry user / User
edit.sdmxregistry.registryPassword / SDMX Registry password / Password
edit.sdmxregistry.registryDomain / SDMX Registry user domain / Circa1
edit.executorMode / The mode of the executor. Executor mode must be set to master for the main execution deployment / master
edit.messaging.brokerURL / Message broker URL, to be used only in ActiveMQ configurations / vm://localhost?broker.persistent=false&broker.timeBeforePurgeTempDestinations=20000&broker.cacheTempDestinations=true&jms.prefetchPolicy.queuePrefetch=0&jms.watchTopicAdvisories=false
edit.messaging.queueNameSelector / Queue Names served by the executor, separated by comma / DEFAULT_INSTALL_QUEUE
edit.messaging.executionQueueName / The physical name of the Message Broker Queue
Note: weblogic syntax is JMSServer/JMSModule!QueueName / JMSServer-0/SystemModule-0!Queue-0
edit.messaging.cancellationTopicName / The name of the Message Broker Topic for cancellation messages
Note: weblogic syntax is JMSServer/JMSModule!TopicName / JMSServer-0/SystemModule-0!Topic-0
edit.messaging.jobMeasureTopicName / The name of the Message Broker Topic for measurement messages.
Note: weblogic syntax is JMSServer/JMSModule!TopicName / JMSServer-0/SystemModule-0!Topic-1
edit.messaging.dsiManagerQueueName / The name of the Message Broker Queue for managing access to dataset instances.
Note: weblogic syntax is JMSServer/JMSModule!QueueName / JMSServer-0/SystemModule-0!Queue-1
edit.messaging.replyTo / Specific settings for the reply queue mechanism. To be used only in activemq configurations / ?replyTo=dsiManagerReplyQueue$amp;replyToType=Exclusive
edit.messaging.provider / The Message Broker provider / jms
edit.messaging.jndi.connectionName / The jndi connection name, to be used only in weblogic configurations / editConnectionFactory
edit.messaging.jms.jndi.factory / The jndi connection factory for the JMS resources / URL=t3://10.20.20.45:7101
edit.messaging.db.jndi.factory / The jndi connection factory for the database connection / URL=t3://10.20.20.45:7101
execution.queue.change.request.sender / From address to be used in ‘change queue request’ emails /
execution.queue.change.request.subject / Subject to be used for ‘change queue request’ emails / High priority job execution requested
edit.imageMaxSize / Maximum size allowed for a news image / 1048576
edit.ext.db.timeout / Timeout to be used for the connection to external databases in ms / 600000
edit.jobs.pendingErrorThreshold / Threshold of number of pending jobs at which the system can be considered to be having some probable internal errors / 100
edit.jobs.pendingWarningThreshold / Threshold of number of pending jobs at which the system should be monitored further to check for errors / 50
archive.path / Settings specific for the postgresql setup
Folder to store the dataset archives / /tmp
edit.dps.domain / Domain on which XTNET/Edit (DPS) specific behaviour should be activated / DPS
edit.embargoLocation / DPS Specific
Location where the embargo data is stored / /embargo
edit.briOutputLocation / DPS Specific
Location where the confidential outputs should be published / /BRI
edit.breOutputLocation / DPS Specific
Location where the non-confidential outputs should be published / /BRE
edit.dps.testdatasets.prefix / DPS Specific
Prefix to be used for the test dataset / TestDS
edit.dps.embargodata.header / DPS Specific
Header to be used for the embargo DAT files / EMBARGO
edit.sdmx.key.families / Directory where SDMX key families are / /edit/repo/keyfamilies
edit.bulkFilesUpload.maxRetries / Maximum retries for checking if a data file described in an index file has arrived / 50

The following table contains all configuration options for the file ebb-server.properties:

property Name / description / example/allowed values
server.host / server ip / 127.0.0.1
server.port / server ip port / 8080
server.protocol / server protocol / http

The following table contains all configuration options for the file ebb-webclient.properties: