Operation guidelines

This applies to RedHat family Linux distributions (RedHat, CentOS, Fedora, ...) and to the Tomcat® application server. It can easily be transposed to other technical platforms.

Services restarting

When required the involved services may need to be stopped in the following order:

Then clean up the Tomcat work folders (and optionally the technical logs folder):

rm -fr $TOMCAT_ROOT/work/Catalina $TOMCAT_ROOT/conf/Catalina
rm -fr $TOMCAT_ROOT/logs/*

Then the services are to be restarted in the following order:

Logs reviewing

To help with diagnostics, several kind of logs can be usefull:

Web console logs

In some cases, the web console logs can be usefull. Check your browser's documentation to figure out how to open the web console. Make sure the logs are persisted when you change page, and reproduce the issue. When connected as designer, the Simplicité logs will be displayed in the web console as well.

HTTP Archive Logs

The http archive file (.har) contains all the HTTP requests made by the application and their results. Check your browser's documentation to figure out how to generate a .har file. Make sure the logs are persisted when you change page, and reproduce the issue.

Tomcat logs

The Tomcat server technical logs are located in $TOMCAT_ROOT/logs.

The content of these logs is managed at Tomcat configuration level, please refer to Tomcat documentation for details (e.g. this document for Tomcat 9)

Simplicité logs

The application-specific technical logs are located in $TOMCAT_ROOT/webapps/ROOT/WEB-INF/log (template-based packaging) or in custom location depending on your installation.

The content of these logs is managed by Log4J and can be customized by overriding the default log4j.xml file provided in $TOMCAT_ROOT/webapps/ROOT/WEB-INF/classes.

For each logging event, an event code is associated, and depending on the configuration in Operation / Events, the message will be either:

Housekeeping

Database logs

Some application log events are recording a significant amount of logs entries into the m_log table of the default database.

Make sure to purge this table on a regular basis:

Import supervisions

Any file import submitted to an application results in a supervision record.

Each of these supervision records includes several attached documents:

They are thus consuming a significant amount of storage.

Make sure to purge these records on a regulrar basis:

Asynchronous jobs records

Any job launched asynchronously by the internal crontable result in a job supervision record (with a link to associated database logs).

Make sure to purge these records on a regulrar basis:

Temporary and exported files

In your code you may use temporary or export directories/files that you may not delete properly.

Make sure to empty these directories/files on a regular basis:

Recycle bin

Removed attached document are not physically deleted, they are simply moved to a recycle bin directory.

Make sure to empty this directory on a regular basis:

Out of sync documents

Some document processing in your code may result in inconsitencies between the document table and the actual documents stored.

Make sure to resynchronise documents on a regular basis:

Save and restore

For a given application a comprehensive save consist in:

These two operations needs to be done exactly at the same time to avoid any data inconsistencies.

To restore the application, the database dump and the archive must be restored in their initial locations.

Note: as of version 3.2 the documents can be stored in the database as BLOBs, in this case saving the database is sufficient (no need to save the document data directory)

Monitoring

All technical components may need to be monitored (especially Tomcat and the database engine) using any convenient tool.

Application-level tools are available (from the UI with an operator profile or from command line) to do basic technical and applicative monitoring. Typically the health check page/service can be called and parsed on a regular basis:

curl -b cookies.txt -c cookies.txt "<base URL>/health[?format=json]"

Note: the -b and -c argument are required to reuse the same Tomcat session

Typical output is:

{
  "platform": {
    "status": "OK",
    "version": "3.2.M05",
    "builton": "2016-10-12",
    "encoding": "UTF-8",
    "systemdate": "2016-10-13 11:51:04"
  },
  "application": {
    "applicationversion": "1.0.0",
    "contextpath": "",
    "contexturl": "<base URL>",
    "activesessions": 1,
    "enabledusers": 12,
    "totalusers": 14
  },
  "os": {
    "name": "Linux",
    "architecture": "amd64",
    "version": "3.10.23-xxxx-grs-ipv6-64",
    "systemencoding": "UTF-8"
  },
  "disk": {
    "diskfree": 10741,
    "diskusable": 9741,
    "disktotal": 19841
  },
  "javavm": {
    "version": "1.8.0_91",
    "vendor": "Oracle Corporation",
    "vmname": "OpenJDK 64-Bit Server VM",
    "vmversion": "25.91-b14",
    "vmscriptengine": "rhino",
    "heapfree": 117231,
    "heapsize": 210432,
    "heapmaxsize": 466432,
    "totalfreesize": 373231
  },
  "cache": {
    "grantcache": 14,
    "grantcachemax": 0,
    "grantcacheratio": 0,
    "objectcache": 10,
    "objectcachemax": 10000,
    "objectcacheratio": 0,
    "processcache": 0,
    "processcachemax": 10000,
    "processcacheratio": 0
  },
  "database": {
    "vendor": "3",
    "productname": "HSQL Database Engine",
    "productversion": "2.3.2",
    "drivername": "HSQL Database Engine Driver",
    "driverversion": "2.3.2",
    "dbdate": "2016-10-13 11:51:04",
    "dbdateoffset": 0
  },
  "healthcheck": {
    "date": "2016-10-13 11:51:04",
    "elapsedtime": 2
  }
}

If this page does not return response or returns a HTTP status different from 200 or if the response value for platform.status is not OK, something needs to be investigated/fixed.

System updates

System updates must be applied regularly, especially in case of critical security updates.

On Linux CentOS, you can check if there are pending system upgrades packages by:

sudo yum check-update

You can then apply the updates by:

sudo yum update

You should then clean up update packages by:

sudo yum clean all

If there is an OpenJDK update you must stop the running Tomcat instances prior to the update and restart them after the update.

If there are other running services impacted by the updates (e.g. database service), you must restart the services.

If there is a kernel update you must reboot the server after the update.

To clean up old kernels you can do:

sudo yum install yum-utils
sudo package-cleanup --oldkernels --count=2

Tomcat updates

The Tomcat server must be updated regularly, depending on the way it has been installed the processus may vary.

Platform updates

The Simplicité® platform must be updated regularly, at least on its maintenance branch (see versions), depending on the way it has been installed the processus may vary.