Operation guidelines

This applies to RedHat family Linux distributions (RedHat, CentOS, Fedora, ...) and to ythe Tomcat® application server. It can easily be transposed to other technical platforms.

Services restarting

When required the involved services may need to be stopped in the following order:

Then clean up the Tomcat work folders (and optionally the technical logs folder):

rm -fr $TOMCAT_ROOT/work/Catalina $TOMCAT_ROOT/conf/Catalina
rm -fr $TOMCAT_ROOT/logs/*

Then the services are to be restarted in the following order:

Logs reviewing

Technical logs

The Tomcat server technical logs are located in $TOMCAT_ROOT/logs.

The content of these logs is managed at Tomcat configuration level, please refer to Tomcat documentation for details (e.g. this document for Tomcat 9)

Application logs

The application-specific technical logs are located in $TOMCAT_ROOT/webapps/ROOT/WEB-INF/log (template-based packaging) or in custom location depending on your installation (e.g. $TOMCAT_ROOT/data/simplicite/<my application>/log/ for legacy packaging).

The content of these logs is managed by Log4J and can be customized by overriding the default log4j.xml file provided in the confSimplicite-x.y.jar (classical packaging) or in $TOMCAT_ROOT/webapps/ROOT/WEB-INF/classes (sandbox packaging)

Housekeeping

Database logs

Some application log events are recording a significant amount of logs entries into the m_log table of the default database.

Make sure to purge this table on a regular basis (Operation > Logs > and use the Delete list list action or use the I/O PURGELOGS service).

Import supervisions

Any file import submitted to an application results in a supervision record.

Each of these supervision records includes several attached documents:

They are thus consuming a significant amount of storage.

Make sure to purge these records on a regulrar basis (Operation > Import supervisor and use the Purge list action or use the the I/O PURGESUPERVISION service).

Asynchronous jobs records

Any job launched asynchronously by the internal crontable result in a job supervision record (with a link to associated database logs).

Make sure to purge these records on a regulrar basis (Operation > Asynchronous jobs and use the Delete jobs list action or use the the I/O PURGEJOBS service).

Temporary and exported files

In your code you may use temporary or export directories/files that you may not delete properly.

Make sure to empty these directories/files on a regular basis (use the I/O PURGETEMPFILES service and the PURGEEXPORTS service).

Recycle bin

Removed attached document are not physically deleted, they are simply moved to a recycle bin directory.

Make sure to empty this directory on a regular basis (use the I/O PURGERECYCLEBIN service).

Out of sync documents

Some document processing in your code may result in inconsitencies between the document table and the actual documents stored.

Make sure to resynchronise documents on a regular basis (use the I/O SYNCDOCS service).

Save and restore

For a given application a comprehensive save consist in:

These two operations needs to be done exactly at the same time to avoid any data inconsistencies.

To restore the application, the database dump and the archive must be restored in their initial locations.

Note: as of version 3.2 the documents can be stored in the database as BLOBs, in this case saving the database is sufficient (no need to save the document data directory)

Monitoring

All technical components may need to be monitored (especially Tomcat and the database engine) using any convenient tool.

Application-level tools are available (from the UI with an operator profile or from command line) to do basic technical and applicative monitoring. Typically the health check page/service can be called and parsed on a regular basis:

curl -b cookies.txt -c cookies.txt "<base URL>/health[?format=json]"

Note: the -b and -c argument are required to reuse the same Tomcat session

Typical output is:

{
  "platform": {
    "status": "OK",
    "version": "3.2.M05",
    "builton": "2016-10-12",
    "encoding": "UTF-8",
    "systemdate": "2016-10-13 11:51:04"
  },
  "application": {
    "applicationversion": "1.0.0",
    "contextpath": "",
    "contexturl": "<base URL>",
    "activesessions": 1,
    "enabledusers": 12,
    "totalusers": 14
  },
  "os": {
    "name": "Linux",
    "architecture": "amd64",
    "version": "3.10.23-xxxx-grs-ipv6-64",
    "systemencoding": "UTF-8"
  },
  "disk": {
    "diskfree": 10741,
    "diskusable": 9741,
    "disktotal": 19841
  },
  "javavm": {
    "version": "1.8.0_91",
    "vendor": "Oracle Corporation",
    "vmname": "OpenJDK 64-Bit Server VM",
    "vmversion": "25.91-b14",
    "vmscriptengine": "rhino",
    "heapfree": 117231,
    "heapsize": 210432,
    "heapmaxsize": 466432,
    "totalfreesize": 373231
  },
  "cache": {
    "grantcache": 14,
    "grantcachemax": 0,
    "grantcacheratio": 0,
    "objectcache": 10,
    "objectcachemax": 10000,
    "objectcacheratio": 0,
    "processcache": 0,
    "processcachemax": 10000,
    "processcacheratio": 0
  },
  "database": {
    "vendor": "3",
    "productname": "HSQL Database Engine",
    "productversion": "2.3.2",
    "drivername": "HSQL Database Engine Driver",
    "driverversion": "2.3.2",
    "dbdate": "2016-10-13 11:51:04",
    "dbdateoffset": 0
  },
  "healthcheck": {
    "date": "2016-10-13 11:51:04",
    "elapsedtime": 2
  }
}

If this page does not return response or returns a HTTP status different from 200 or if the response value for platform.status is not OK, something needs to be investigated/fixed.

System updates

System updates must be applied regularly, especially in case of critical security updates.

On Linux CentOS, you can check if there are pending system upgrades packages by:

sudo yum check-update

You can then apply the updates by:

sudo yum update

You should then clean up update packages by:

sudo yum clean all

If there is an OpenJDK update you must stop the running Tomcat instances prior to the update and restart them after the update.

If there are other running services impacted by the updates (e.g. database service), you must restart the services.

If there is a kernel update you must reboot the server after the update.

To clean up old kernels you can do:

sudo yum install yum-utils
sudo package-cleanup --oldkernels --count=2

Tomcat updates

The Tomcat server must be updated regularly, depending on the way it has been installed the processus may vary.

Platform updates

The Simplicit√©® platform must be updated regularly, at least on its maintenance branch (see versions), depending on the way it has been installed the processus may vary.