Platform Installation Guide (printable)
1 Overview
This guide provides information on installation planning, configuration, and tuning of the platform for optimum performance and security. It is intended for installers who are building a fully-optimized, production installation.
Additional Resources
- For a searchable and printable version of this guide, see the Platform Installation Guide (printable)
- To learn more about the latest release, see the All Release Notes
- See also these additional resources for installation and configuration:
2 Installation Requirements
2.1 Hardware Requirements
| Application Server |
|
| Document storage |
|
| Database Server |
|
| Database backup |
|
2.2 Software Requirements
2.2.1 Installing MySQL Version 8.0.xx
| Operating System |
| ||||
| MySQL Server |
If you are a platform user, see Configure the MySQL Server.
| ||||
| Libraries | Download the third-party libraries and save them in a folder. You may want to integrate them into the system during the installation.
scrollTolerantForwardOnly=true
|
2.2.2 Installing MySQL Version 5.7.xx
| Operating System |
| ||
| MySQL Server |
If you are a platform user, see Configure the MySQL Server. | ||
| Libraries | Download the third-party libraries and save it in a folder. You may want to integrate them into the system during the installation. |
2.2.3 Required for Basic Platform Functionality
| Mail Server |
An email message should be received at the specified address. If the confirmation message does not arrive, check the mail log to discover the cause.
|
| Cache |
Ehcache is available in the Common Tomcat profile. For more information, see Configuring Ehcache. |
| Web Server |
The Apache web server can be placed in front of the Tomcat appserver to deliver static content more efficiently.
For more information, see Installing and Configuring Apache for Use with the Platform. |
| Java |
The installation and operation of the AgileApps Cloud platform requires Java 8. However, the custom code in Java Class continues to be validated against Java 6 syntax as the instrumentation engine responsible for monitoring resource utilization works only on Java 6 byte codes. |
| Browser Support |
As part of the security processes, the Remember me on this computer option is removed from the browser's login page.
|
2.2.4 Required for Additional Functionality
| OpenOffice (optional) |
| ||
| HornetQ (optional) |
The Messaging Server is an optional platform component that:
Get HornetQ from http://www.jboss.org/hornetq/downloads.html. For more information, see Installing the Messaging Server. | ||
| Charting Libraries (optional) |
Additional libraries are needed to email a chart or a report. For more information, see Install the Chart Handling Libraries. |
2.3 License Requirements
About licenses:
- One platform license is required for each instance of the Application Server.
- A short term license is included in the initial installation.
- An email notification is sent out 30 days before a license is due to expire.
- A grace period extends for an additional seven days after the expiration date.
Daily notifications are sent during that period, until the license is renewed or the grace period ends. - To obtain a long-term license, contact your Software AG representative. You will receive an XML license file.
To install the new license:
- Copy the XML license file to {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN
- Restart tomcat
3 Installation Planning
Platform Hardware Stack Platform Software Stack
3.1 Platform Deployment Options
This section describes deployment options that increase scalability and reliability.
- Multiple Application Servers
- Instances of the platform can be deployed on different servers to improve response times.
- Separate servers for Apache httpd
- When the Apache httpd server is used to serve static content, the Apache httpd instances can be deployed on separate servers, while platform instances run on their own servers. Learn More: Installing and Configuring Apache for Use with the Platform
- Separate Messaging Server
- To further improve performance, the Messaging Server can be installed on a standalone system. Or it can run the same system as an appserver.
- Web App Accelerators
- Web Application acceleration services like those provided by Akamai can also be used to improve response times and throughput. Learn More: Web Application Accelerator.
- Multiple memcached servers
- The MemCache Daemon can also be run on one or more separate servers. In addition to the performance improvement, that configuration makes it somewhat easier to add application servers to scale capacity later. Learn More: Configuring memcached
- Certificate-authentication Server
- Adding a certificate-validation server to the network architecture, allows the platform to be accessed by browsers or by REST APIs without a login step, by supplying an identifying certificate.
- Learn more: Client Certificate Authentication in a private AgileApps Cloud (pdf)
- Separate Database Server
- Running the platform and the Database on different servers is highly recommended for a production deployment. Learn More: Configuring MySQL to Run on a Separate Server
- Separate Servers for Backend Processes
- The backend processes used by the application server can be (and should be) deployed to at at least one "backend server" to minimize response time in the customer-facing server they would otherwise be running on. Beyond that, heavily-used services can be deployed to servers of their own, to maximize performance. Learn More: Managing Backend Services
3.2 Prototype Deployment
The following diagram shows the kind of architecture that is typical for a production system pre-10.11 release and post 10.11 release. With the 10.11 release, AgileApps uses the common Tomcat that comes along with the IS installation and does not use the stand-alone Tomcat anymore. This allows AgileApps access to the flow services on the co-hosted IS. The following images allow you to identify the difference in the architecture for pre and post 10.3 release.
- Architecture for pre-10.16 Release
- Architecture for post-10.16 Release
- Architecture for post-10.16 Release
The architecture is explained as follows:
- Node 1 and Node 2 are the AgileApps server instances available in the front-end.
- Node 3 serves as the backend AgileApps Server instance. The critical backend processes shown here (import, export, and scheduling, which uses quartz) are all run from a single platform instance. However, you can employ additional servers, as load demands.
Learn more: Managing Backend Services. - Document storage (which includes pictures and image files) is managed separately from the database. In the example above, it is housed as part of the backend server instance. The Document storage communicates with all the nodes.
Learn more: See the Document Server section in Service Provider Settings. - The user logs into AgileApps and the request comes to the Load Balancer which distributes the traffic across the web servers.
- The web server can be an Apache server or an Nginx server.
Learn more: Installing and Configuring Apache for Use with the Platform. - A Memcached server reduces response time by caching data in memory. Memcached servers are accessed by all Application servers, backend as well as front-end.
Learn more: Configuring memcached - All the nodes are connected to the primary database server. The primary database is running on its own server for better performance. It can also reside within one of the nodes VM1 or VM2 along with the AgileApps server. The database could be MySQL, Amazon RDS, or so on.
Learn more: Configuring MySQL to Run on a Separate Server - The replicated database is running separately on its own server. This ensures more reliability and for performing read-intensive operations like taking backups, generating reports, and exporting data.
Learn more: Running Reports and Storage Checks On a Replicated Database Server. - Requests that access and update the database, whether coming from a user or a backend process, go to the primary database server.
3.3 Choosing a MySQL Replication Strategy
MySQL supports several replication formats:
- Statement-Based Replication (SBR), which replicates entire SQL statements
- Row-Based Replication (RBR), which replicates only changed rows.
- Mixed-Based Replication (MBR), which is a combination of the two.
With Statement-Based Replication, SQL statements are propagated using the standard statement-based, binary logging format. That is the default replication format in the version of MySQL that ships with the platform.
Row-based binary logging is a mechanism that logs changes in individual table rows. With row-based replication, the master writes events to the binary log that indicate how individual table rows are changed.
When mixed format is in effect, statement-based logging is used by default, but automatically switches to row-based logging when it is necessary to do so.
Learn More:
- Replication: http://dev.mysql.com/doc/refman/5.5/en/replication.html
- Replication formats: http://dev.mysql.com/doc/refman/5.5/en/replication-formats.html
- Comparison of Statement-Based and Row-Based Replication:
http://dev.mysql.com/doc/refman/5.5/en/replication-sbr-rbr.html - How To Set Up Database Replication In MySQL:
http://www.howtoforge.com/mysql_database_replication
3.4 Choosing a MySQL High-Availability Option
MySQL can be configured for High Availability using the following schemes supported by MySQL:
- MySQL Replication
- Replication provides data safety. It also lets you run reports and backups from the secondary server, reducing the load on the primary server and improving the response time seen by users. There is some exposure to data loss, since recently-seen data may not have been replicated when a failover occurs--but on the other hand, transactions don't incur the lag time added by serialized (synchronous) replication of data.
- MySQL Replication + Heartbeat
- Using Linux HA capabilities, MySQL can be configured for automatic IP-address failover.
- Learn More: http://dev.mysql.com/doc/refman/5.5/en/ha-overview.html
- DRBD + MySQL Heartbeat
- While the previous two schemes do statement-based replication, Distributed Replicated Block Devices (DRBD) does synchronous replication of modified disk blocks. The advantage is consistently fast failover times. And because it is synchronous, incoming data is virtually guaranteed to be replicated, for maximum protection against data loss. The disadvantage is that traffic-handling volume may be decreased, due to the serial writes (replication must complete before the original data write can finish). This mechanism is also expensive to set up and maintain.
- Learn More: http://dev.mysql.com/doc/refman/5.5/en/ha-drbd.html
- MySQL Replication + DRDB + MySQL Heartbeat
- Replication and DRDB are not mutually exclusive. They can be used in combination with one another.
3.4.1 Comparing the Options
The following table compares those replication schemes:
Requirements Replication Replication + Heartbeat DRBD + Heartbeat Replication + DRBD + Heartbeat Availability Automated IP failover No Yes Yes Yes Automated database failover No No Yes Yes Typical failover time User/script-dependent Varies < 30 seconds < 30 seconds Automatic resynchronization of data No No Yes Yes Geographic redundancy support Yes Yes No Yes Scalability Supports Read-intensive applications Yes Yes No Yes Supports Write-intensive applications No No Yes Yes Maximum number of nodes per group One master, multiple slaves One master, multiple slaves One active (primary), one passive (secondary) node Maximum number of slaves Unlimited (reads only) Unlimited (reads only) One (failover only)
- Note: This table is based on information obtained from http://dev.mysql.com/doc/refman/5.5/en/ha-overview.html
3.4.2 MySQL Cluster Not Supported
MySQL Cluster has functionality limitations that prevent it from working with the platform. Some of the more significant limitations include:
- Limitation on number of objects in an instance (around 20,000)
- Temporary tables are not supported
- Limitations on Index size
- Savepoint and Rollback to Savepoint are not supported
- Replication is not supported
- Online schema changes are not supported
4 Platform Installation
4.1 Installation Files
4.1.1 Post-Installation File Hierarchy
When the platform is installed, these additional folders are created:
- docs
- Store the documents, attachments, and other files uploaded to the platform.
- tmp
- A work folder used for temporary files.
4.1.2 Configuring the MySQL Server
These instructions apply to the version of MySQL specified in the Software Requirements.
Note: MySQL configuration is a pre-requisite for the AgileApps installation.
Terminology:
- MySQL Server
- The database server which has MySQL Server software installed, configured, and running.
- MySQL Client
- Any other computer which connects to the database server and has the MySQL Client software installed, configured, and running.
- The client connects to the database server to access data, for example, webserver.
- Configuration file
- my.cnf, a MySQL configuration file that may be edited and used in the MySQL client or MySQL Server installation.
4.1.2.1 MySQL Settings
- Storage Engine
- default-storage-engine = innodb
- Server System Variables Configuration
- It is observed that READ_UNCOMMITTED does not work well with Row-Based Replication. Use one of the following configurations:
- a) Either STATEMENT-BASED or MIXED replication with the Transaction Isolation level dictated by business needs.
- b) ROW-BASED replication with transaction_isolation = READ-COMMITTED
- User Configuration
-
- Create a user account with password in MySQL
The Application Server will use the user details.
- The user account should have all MySQL privileges enabled on all databases
- Use the default root account that is created during the installation of MySQL. You can also create a non-root MySQL user to run Agile Apps. For more information, see Creating a non-root MySQL User to Run Agile Apps.
- Create a user account with password in MySQL
- sql-mode Configuration
-
- Add the following code to the my.cnf file located in the MySQL Client:
- sql-mode="PIPES_AS_CONCAT,ANSI_QUOTES"
- Add the following code to the my.cnf file located in the MySQL Server:
- sql-mode="PIPES_AS_CONCAT,ANSI_QUOTES"
- MySQL Configuration for UTF-8 Unicode character set for 5.7 Version
-
- Add the following code to the my.cnf file, located in the MySQL Client:
- default-character-set=utf8
- Add the following code to the my.cnf file, located in MySQL Server:
- character-set-server=utf8
- collation-server=utf8_general_ci
- The character set defines how records can be alphanumerically ordered (or grouped, sorted, filtered, indexed). The list of supported languages is determined by the character set available at the server level. Hence, the UTF-8 Unicode character set configuration is required as part of MySQL configuration.
- To specify character settings at MySQL configuration time:
For MySQL Version 5.7, see https://dev.mysql.com/doc/refman/5.7/en/charset-applications.html
- To specify character settings at MySQL configuration time:
- MySQL Configuration for UTF-8mb4 Unicode character set for MySQL 8.0 Version
-
Note: For MySQL configuration v5.7 to v8 upgrade, refer Upgrading to MySQL v8.
- Add the following code to the my.cnf file, located in the MySQL Client:
- default-character-set=utf8mb4
- Add the following code to the my.cnf file, located in MySQL Server:
- character-set-server=utf8mb4
- collation-server=utf8mb4_0900_ai_ci
- The character set defines how records can be alphanumerically ordered (or grouped, sorted, filtered, indexed). The list of supported languages is determined by the character set available at the server level. Hence, the UTF-8 Unicode character set configuration is required as part of MySQL configuration.
-
Note: If user does not specific in my.cnf or my.ini file then default MySQL 8 character set and collation are utf8mb4 and utf8mb4_0900_ai_ci taken by server.
- To specify character settings at MySQL configuration time:
For MySQL Version 8.0, see http://dev.mysql.com/doc/refman//8.0/en/charset-applications.html
- To specify character settings at MySQL configuration time:
- max_allowed_packet Configuration
- Add max_allowed_packet = 64M to my.cnf or my.ini
- For MySQL Version 5.7, see https://dev.mysql.com/doc/refman/5.7/en/charset-applications.html
- For MySQL Version 8.0, see https://dev.mysql.com/doc/refman/8.0/en/charset-applications.html
- Restart the MySQL server to implement the changes
- regexp_time_limit Configuration
The regular expression limit is expressed as the maximum permitted number of steps performed by the match engine, and thus affects execution time only indirectly. It is defined in milliseconds.
- Add regexp_time_limit=2048 to my.cnf or my.ini
- For MySQL Version 5.7, see https://dev.mysql.com/doc/refman/5.7/en/charset-applications.html
- For MySQL Version 8.0, see https://dev.mysql.com/doc/refman/8.0/en/charset-applications.html
- Restart the MySQL server to implement the changes
4.1.2.2 Populate the Timezone Tables
- The platform uses MySQL's timezone tables for timezone conversions. These tables are not automatically populated when MySQL is installed, so it necessary to do so after installation.
- Run the following program to initialize the MySQL timezone tables:
- mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql
- where /usr/share/zoneinfo is the standard Linux location for the time zone files. (Your system may differ.)
- For MySQL Version 5.7, see https://dev.mysql.com/doc/refman/5.7/en/mysql-tzinfo-to-sql.html
- For MySQL Version 8.0, see https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html
- mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql
4.1.3 MySQL Tuning Parameters
These section provides some typical MySQL configuration values that are appropriate for a production system. These values assume:
- 100Mb Primary Ethernet Card
- 32 GB Memory
- 2 Processors
- 2 Cores per Processor
- Switched Ethernet Port
- 64-bit Red Hat Enterprise Linux ES 4
Important:
- Before changing parameters, take the server and MySQL down.
Do not adjust parameters while MySQL is running. Doing so can potentially cause the database to be corrupted. - The settings shown here are an example. Consult the latest MySQL document for information on the best settings to use.
- Before changing parameters, take the server and MySQL down.
4.1.3.1 my.cnf / my.ini Options
interactive-timeout = 7200 wait-timeout = 7200 back_log = 200 max_connections = 350 max_connect_errors = 10 table_cache = 8192 max_allowed_packet = 48M binlog_cache_size = 8M max_heap_table_size = 120M sort_buffer_size = 2M join_buffer_size = 2M thread_cache = 200 thread_concurrency = 24 query_cache_size = 500M query_cache_limit = 1M ft_min_word_len = 4 thread_stack = 256K tmp_table_size = 120M
4.1.3.2 InnoDB-Specific Options
innodb_additional_mem_pool_size = 48M innodb_buffer_pool_size = 23G innodb_data_file_path = ibdata1:10M:autoextend innodb_file_io_threads = 4 innodb_thread_concurrency = 24 innodb_flush_log_at_trx_commit = 1 innodb_log_buffer_size = 8M odb_log_file_size = 256M innodb_log_files_in_group = 3 innodb_max_dirty_pages_pct = 90 #innodb_flush_method=O_DSYNC innodb_lock_wait_timeout = 300
4.2 Installing and Upgrading the Application Server
4.2.1 New Installations
- Review the Installable Version Release Notes.
- Ensure that the Hardware Requirements are met.
- Ensure that the Software Requirements#Required for Installation are met.
- Configure the MySQL Server so it is ready for use.
- Ensure that all database servers and web servers are configured to be in the same timezone, regardless of where they are geographically located.
- Use the standard SoftwareAG webMethods installer to install the platform of your choice.
Learn more:- From the webMethods documentation page consult the respective Installation document.
- For the very first server that is installed, Start the Application Server to verify that everything is working.
- For subsequent servers, Configure Backend Services before startup.
- Install the Platform License
- Once installed, proceed to Configuring the Application Server
4.2.2 Upgrades
-
Important:
These are instructions are for version 10.5 and later.
If you are upgrading to 10.5, use the 10.5 Upgrade Process, instead.
- If you have not already done so:
- Review the Installable Version Release Notes.
- Ensure that the Hardware Requirements are met.
- Ensure that the Software Requirements#Required for Installation are met.
- Backup the platform databases.
- Use the standard SoftwareAG installer to install the 10.5 platform.
Learn more: In webMethods documentation page see the 9.9 Installation document. - Start the Application Server
- When done, proceed to Configuring the Application Server
4.3 License Requirements
About licenses:
- One platform license is required for each instance of the Application Server.
- A short term license is included in the initial installation.
- An email notification is sent out 30 days before a license is due to expire.
- A grace period extends for an additional seven days after the expiration date.
Daily notifications are sent during that period, until the license is renewed or the grace period ends. - To obtain a long-term license, contact your Software AG representative. You will receive an XML license file.
To install the new license:
- Copy the XML license file to {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN
- Restart tomcat
5 Platform Security
5.1 Managing SSL Certificates
5.1.1 Obtaining an SSL Certificate
The platform provides a default self-signed certificate which is used by the Application Server.
To obtain and install your own SSL Certificate, make a request to a Certificate Authority (CA). An SSL certificate authenticates a website to a web browser, part of a security protocol to manage secure data exchange.
The CA will accept your Certificate Signing Request and generate a certificate which identifies your website as a secured website.
To create a Certificate Signing Request (CSR)
- 1. Create a keystore and a private key:
cd {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN keytool -genkey -alias tomcat -keyalg RSA -keysize 2048 -keystore {keystore_filename}
- 2. Create a CSR from the keystore
keytool -certreq -keyalg RSA -alias tomcat -file certreq.csr -keystore {keystore_filename}
- 3. Submit the resulting file, certreq.csr, to the CA to obtain a certificate.
(When the certificate arrives, you are ready for the next step of steps.)
To Install the Certificate Obtained from the CA
Once you have obtained a certificate, you need to import it into the keystore.
But first, in addition to your certificate, the CA might provide a Chain/Root Certificate, which must also be imported. If you have received a chain certificate from the CA, then:
- 1. Copy the contents of the chain certificate into a file called chain
- 2. Import the chain certificate into your keystore:
keytool -import -alias root -keystore {keystore_filename} -trustcacerts -file chain
When the chain certificate (if any) has been imported, you are ready for the final step:
- 3. Import the certificate received from the CA:
keytool -import -alias tomcat -keystore {keystore_filename} -trustcacerts -file {certificate_filename}
-
Note: If you have SSL certificate and private key, follow the below steps:
- 1. Convert the private key and certificate to PKCS#12 format using OpenSSL. Assuming you have the private key file in .key format (private.key) and the certificate file in .crt format (VMNX-AALIND22.crt), use the following command:
openssl pkcs12 -export -inkey <private.key> -in <certificate.crt> -out <keystore.p12> -name <alias>
The default alias is set to 1.
- 2. Replace <alias> with the desired alias for the key entry.
- 3. Import the PKCS#12 file into the Java keystore using the keytool command:
keytool -importkeystore -srckeystore <keystore.p12> -srcstoretype PKCS12 -destkeystore <keystore.jks> -destalias <alias>
- 4. Replace <alias> with the alias used in the previous step.
5. Enter the appropriate passwords when prompted, including the source keystore password for the PKCS#12 file and the destination keystore password for the Java keystore.
- 6. Once you have successfully completed these steps, the certificate and private key should be imported into the Java keystore with the specified alias.
5.1.2 To update a Customer SSL Certificate in AgileApps
- 1. Stop the Application server.
- 2. Update keystoreFile and keystorePass values in “com.softwareag.catalina.connector.https.pid-agileappsHttps-8284.properties” file available under
- {install-dir}/profiles/IS_default/configuration/com.softwareag.platform.config.propsloader folder.
-
Note: The keystorePass value provided by you in plain text is encrypted automatically when you restart the AgileApps application server.
- 3. After updating the properties, place the certificate in the {install-dir}/profiles/IS_default/configuration/tomcat/conf folder.
- 4. Restart the memcached server and start the AgileApps application server.
5.1.3 Learn More
- Certificate Signing Request (CSR) Generation Instructions-Tomcat, at
https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&id=AR227
5.2 Controlling Port Access
5.2.1 Firewall Ports
Platform Application Servers are typically deployed behind a Firewall. The firewall needs to open the ports those servers use.
The default ports are:
- Non SSL port 80
- SSL port 443
The SSL port always needs to be open. If the application is to be accessed only using https, the http port can be blocked. If the platform's Sites capability will be used to provide a public URL, then the http port needs to be open.
-
Note: For secure communication, you should always access the platform using SSL (https://yourdomain/networking/Service). The platform provides a default self-signed certificate which is used by the Application Server. This certificate can be replaced with your own certificate, purchased from a certificate-signing authority. For more instructions on that process, see Managing SSL Certificates.
5.2.2 Changing Port Assignments
To change port assignments:
- Edit the following files available in {install-dir}/profiles/IS_default/configuration/com.softwareag.platform.config.propsloader as follows:
- com.softwareag.catalina.connector.http.pid-agileappsHttp-8283.properties - In this file, set the port value to 80 and redirectPort value to 443.
- com.softwareag.catalina.connector.https.pid-agileappsHttps-8284.properties - In this file, set the port value to 443.
Note: If you change the http port (80), you will need to specify the port number in the URLs for the document servers recorded in the database. To see those values:
Mysql> SELECT document_server,import_document_server,public_document_server FROM relationals.NETWORK_GLOBAL_PROPERTIES;
Those URLs are read when an application server starts, and used as the forwarding-target for document-access requests.
5.3 Running the Platform as a Non-Root User
As a good security practice, it is recommended that the platform and the various software components that it required by the platform are run as a users other than root. This section explains about running the platform as a non-root user. For information on creating a non-root MySQL user in Unix Common Tomcat Platform installations, see Creating a non-root MySQL User to Run Agile Apps.
-
Important: The installer needs the same privileges as the root user, without actually being the root.
5.3.1 memcached
memcached accepts the command line argument –u {username}. With that argument, memcached assumes the identity of the specified user when running, after being launched by the root user. It is recommended that this username is set to nobody in the memcached startup command. For example:
- <syntaxhighlight lang="java" enclose="div">
/usr/local/bin/memcached -m 250 -p 11211 -d -u nobody </syntaxhighlight>
Here, memcached is configured to run as a daemon using 250MB of cache, listening on port 11211, and running as the user nobody.
5.3.2 MySQL
A typical mysql installation has the mysql server running as the user mysql. You can verify that setting using the ps command:
- <syntaxhighlight lang="java" enclose="div">
ps auxwww | grep mysql </syntaxhighlight>
5.3.3 Apache httpd server
Set the User and Group directives in httpd.conf to values other than root.
For example:
- Create a group called apache and add the user apache to that group.
- Set the values for User and Group in httpd.conf to apache
5.3.4 Platform
The AgileApps Cloud platform is based on Apache Tomcat. It can be run as a standalone AppServer or deployed behind the Apache httpd server. When run as a standalone server, the platform listens to ports 80 and 443 for http requests. To be run as a non-root user, the platform must be deployed behind Apache, and must be listening on ports > 1023 (since only the root user can bind to low-numbered ports).
When installing the AgileApps Cloud platform:
- 1. Login as the user tomcat instead of user root.
- That way, the ownership and permissions for the installation folders are set up properly.
After configuring the platform and Apache to work together:
- 2. Create the group tomcat and add the user tomcat to it:
- <syntaxhighlight lang="java" enclose="div">
/usr/sbin/groupadd tomcat /usr/sbin/useradd -g tomcat -d /home/tomcat tomcat </syntaxhighlight>
- 3. Change the password for the user tomcat:
- <syntaxhighlight lang="java" enclose="div">
passwd tomcat </syntaxhighlight>
- 4. Change the ownership in the folder where the platform is installed to the user tomcat.
- For example, if the installation folder is /opt/softwareag/agileapps:
- <syntaxhighlight lang="java" enclose="div">
chown -R tomcat:tomcat /opt/softwareag/agileapps </syntaxhighlight>
- 5. Restrict permission on the configuration folder to the user tomcat:
- <syntaxhighlight lang="java" enclose="div">
chmod -R 700 {agileapps}/tomcat/conf </syntaxhighlight>
- 6. Ensure that the user tomcat has sufficient permissions on the Temp Directory
- If the Temp directory configured in the Service Provider Settings is /tmp (the default), ensure that everyone has read and write permissions on that folder:
chmod –R 777 /tmp </syntaxhighlight>
- <syntaxhighlight lang="java" enclose="div">
- If upgrading an installation in which the Temp Directory configured in the Service Provider Settings is not /tmp, and is used exclusively by the platform, change the ownership of the folder to the user tomcat. For example, if the location is /var/platform_tmp:
chown -R tomcat:tomcat /var/platform_temp </syntaxhighlight>
- <syntaxhighlight lang="java" enclose="div">
- 7.Make it possible for the non-root user to run the startup command
- to become user tomcat before starting the platform.
- To do that, change this line:
- <syntaxhighlight lang="java" enclose="div">
{install-dir}/profiles/IS_default/bin/startup.sh </syntaxhighlight>
- to
- <syntaxhighlight lang="java" enclose="div">
sudo {install-dir}/profiles/IS_default/bin/startup.sh </syntaxhighlight>
6 Backend Services
6.1 Managing Backend Services
The AgileApps Cloud platform has several backend services. Each of them can be run on a different server, to improve performance and scalability. They can be disabled, as well, if they are not used.
-
Important:
All backend services are enabled by default. But...- Backend services should not be enabled on a customer-facing web server, to avoid performance problems.
(The exception would be a single-user server used for development or proof of concept.) - With the exception of memcached, a backend service should not be enabled on more than one server, to avoid errors.
Immediately after installing a server, then, it is necessary to disable all backend services, except for those services that are intended to run on the current server--and then only if the current server is intended to be a backend server.
- Backend services should not be enabled on a customer-facing web server, to avoid performance problems.
6.1.1 About the Backend Services
An Application Server instance can run one or more of the following services. Enabling and disabling them determines which instance they run in--or whether they are available at all.
Service Default Status Description Report Scheduler Enabled Runs scheduled reports when they're due Import Enabled Imports of data into the database Export Enabled Exports data from the database Memcached Enabled Data caching mechanism (installed separately) Quartz Enabled Time keeper for all scheduled events.
6.1.2 Enabling and Disabling Backend Services
During installation, the networking.properties file is created. That file can be used to manually enable and disable backend services for an existing installation.
-
Important: Each background service should run on only one server.
- To view the server list
- Open {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN/networking.properties
- Servers are listed in this format: xxxxx_manager_instance=1
where "xxxxx" is one of "import", "report", etc.
- Servers are listed in this format: xxxxx_manager_instance=1
- To Disable a service
- Add a "#" character before the xxxxx_manager_instance=1 command
- # report_manager_instance=1
- Save the file
- Restart the Application Server
- To Enable a service
- Remove the "#" character from the xxxxx_manager_instance=1 command
- report_manager_instance=1
- Save the file
- Restart the Application Server
6.1.3 Configuring Backend Services
During installs and upgrades, it is only necessary to enable or disable a service on a platform instance. The install/upgrade process takes care of making the connections.
To change a configuration for an existing installation, changes need to be made manually, by adjusting the networking.properties file(s) and restarting the Application Server(s).
-
Note: A change in the configuration for either memcached, quartz, or the document service requires a restart of all Application Servers.
The general process is:
- Make the changes to the networking.properties files for all Application Servers.
- Restart the Application Servers.
These sections describe the changes to make:
Manually Configuring Backend Services
6.1.4 Configuring Document Services
-
Important: Changing the document service configuration requires a restart of all Application Servers.
Document uploads and downloads are handled by the document service. Managed documents include:
- Individual documents accessed through the documents tab
- Attachments to records
- File and image fields in records
The document service ensures that documents can be accessed from any application server. One of the App Servers should be designated as the document server. That server needs access to the storage device that will hold the documents.
-
Recommendation: A reliable storage device should be used for document storage--something on the order of a SAN or a RAID enabled disk array.
The URL of the document server can be configured by updating the following columns in the table relationals.NETWORK_GLOBAL_PROPERTIES:
- document_server - Stores documents that can be accessed only by authenticated users
- public_document_server – Stores public documents that can be accessed without logging into the platform
- import_document_server – Stores documents that are uploaded using data import
- Notes:
- The typical value for those properties is http://ip_address_of_document_server/networking
- Changing the document server location requires a restart of all platform instances.
-
Recommendation: All three values should be the same. In other words, exactly one server should be responsible for all of the document services.
6.1.5 Configuring the Quartz Scheduler
-
Important:
- Quartz must be enabled on at least one platform instance, or the platform won't run.
- It is recommended that you run Quartz on the back-end servers to avoid any performance issues in the front-end servers.
- If you have multiple instances, Quartz should run on only one of them, unless you follow the Quartz clustering instructions below.
- Changing the quartz configuration requires a restart of all Application Servers.
6.1.5.1 Enabling Quartz on a Platform Instance
- 1. Enable qrtz_manager_instance = 1. This is available at {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN/networking.properties
- 2. Edit {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN/AALSchedulers.xml
- 4. Add the following information:
- a. Database user name (an admin user)
- b. Password for that user
- Example:
<attribute name="org.quartz.dataSource.myDS.URL" value="jdbc:mysql:"/> <attribute name="org.quartz.dataSource.myDS.user" value=""/> -- root user <attribute name="org.quartz.dataSource.myDS.password" value=""/> -- password
-
Note: Do not change the value of org.quartz.scheduler.instanceName property.
6.1.5.2 Moving Quartz to Another Platform Instance
Suppose you have one application server instance, and you have decided to add another to handle the traffic volume. And let's say you also decide to move quartz to the new instance, to further reduce the load on the original server. To that, you disable Quartz on the first instance, and start it on the second.
- 1. Disable qrtz_manager_instance = 0. This is available {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN/networking.properties
- 2. For the second instance, follow the steps above, in Setting up Quartz on a Platform Instance
6.1.5.3 Setting up a Quartz Cluster
By default, Quartz is pre-configured to run as a cluster of instances running on different servers. To run successfully, however, each server must have the settings shown here. Otherwise, a Quartz instance will not know that the others exist, and will not lock job records that are in progress. Perform this configuration on any server you later add to the cluster.
- 1. Open {install-dir}/profiles/IS_default/configuration/tomcat/conf/RN/AALSchedulers.xml
- 2. Check this line:
<attribute name="org.quartz.scheduler.instanceId" value="AUTO" [VERIFY THIS LINE]
- 3. Under # Configure JobStore, check these lines:
... # Configure JobStore <attribute name="org.quartz.jobStore.misfireThreshold" value="60000"/> [VERIFY THESE LINES] <attribute name="org.quartz.jobStore.isClustered" value="true"/> <attribute name="org.quartz.jobStore.clusterCheckinInterval" value="20000"/>
In addition, MySQL must be configured to set the appropriate transaction-isolation level and to specify row-based replication:
- Edit {MYSQL_INSTALLATION}/my.cnf
- Use the following settings:
transaction-isolation = READ-COMMITTED binlog-format = ROW
6.1.6 Configuring memcached
The MEMory CACHE Daemon is a high-performance, distributed-object caching system that minimizes user response time by caching application data and other elements required by programs running on the application server.
The platform uses memcached to store meta information which so that they can be obtained from the cache without accessing the database e.g. Object and field definitions, layout definitions, data policies etc. (Transactional customer data is not stored in memcached--that data is cached by the database.)
Data is stored in memcached using a lazy read mechanism. When some meta information is required, the platform checks to see if it is available in memcache. If it is available, the data is retrieved. If it is not available, a database read is performed, which in turn populates the entry in memcached. When the meta information changes due configuration changes by the user, the corresponding entry is flushed from memcached.
-
Important: The configuration for every instance must have the same memcached settings--otherwise, the caches won't be synchronized. Changing the memcached configuration therefore requires a restart of all Application Servers.
6.1.6.1 Managing memcached Servers
- To install memcached
- See http://memcached.org/ for installation instructions.
- To start memcached
-
- Execute the memcached script located in the /bin directory:
./memcached -d start -p 11211 -u root -m 25
- where:
- -p is the port number (11211)
- -u is the user (root)
- -m is the amount of memory in MB (25)
- (Change the parameters to the values used for your installation.)
- where:
- To stop memcached
./memcached -d stop</tt>
6.1.6.2 Configuring Application Servers to Use memcached
- To enable memcached
- Edit networking.properties
- Remove the leading # character from the line
#MemCachedServers=localhost:11211
- To change the default port
- The default port is 11211. To change the port assignment, modify the value(s) for MemCachedServers in networking.properties.
- To configure multiple memcached instances
- Multiple memcached instances are recommended, so that cache access is distributed. Do that by specifying the IP address and port for each memcache server in the networking.properties file.
-
Important: If there are multiple memcached servers, then:
- Every instance should have an identical list of servers.
- The servers should be in the same order in each list.
- If the configuration changes, all application servers should be restarted.
- Learn more: Adding Additional memcached Servers
- To bring up the servers
- After making the changes to the networking.properties files for all Application Servers, (re)start servers in the required Server Restart Sequence.
7 Performance Optimizations
7.1 Configuring MySQL to Run on a Separate Server
This procedure sets up database connection pooling between the two servers.
- Enable permissions on MySQL, so web servers can access the database server:
GRANT ALL ON *.* to 'root'@'web_server_IP_Address' IDENTIFIED BY 'password'
- Direct the database server to the application server:
- Edit {install-dir}/profiles/IS_default/configuration/tomcat/conf/server.xml
- Replace url="jdbc:mysql://{yourDomain}/relationals" with
url="jdbc:mysql://IP_address_of_database_server/relationals" - Save the file
- If the Application server is currently running, then Restart the Application Server.
Learn More: http://dev.mysql.com/doc/refman/5.5/en/replication-solutions-backups.html
7.2 Installing and Configuring Apache for Use with the Platform
Apache can be installed in front of the tomcat server to serve static content like gif and jpeg images, Javascript files and stylesheets more efficiently, leaving the Application Server free to process incoming requests. This section tells you how to configure Apache to serve the platform's static content.
7.2.1 Overview
With this arrangement, Apache will be in front of Tomcat. Any request that comes to the platform first goes to Apache. Apache determines if the request is a valid request and, if so, will forward the request to Tomcat for processing.
Since Apache will be in front of Tomcat, it needs to be configured to listen on port 80 and port 443 for http and https traffic respectively. Apache also needs to be configured to forward traffic coming on those ports to the port on which Tomcat is listening.
7.2.2 Requirements
The following Apache modules must be enabled when installing Apache. They can be compiled as a static binary into apache or loaded as dynamic modules.
- mod_proxy
- mod_proxy_connect
- mod_proxy_http
- mod_ssl
- mod_expires
- mod_headers
7.2.3 Configuration Process
In outline, the process looks like this:
- Configure Apache Virtual host.
- Start Apache, Tomcat and memcached.
-
Important: Ensure that all database servers and web servers are configured to be in the same timezone, regardless of where they are geographically located.
7.2.4 Configure Apache Virtual Hosts
- 1. Add two virtual host entries
- - One in httpd.conf for Apache to listen on port 80 for HTTP traffic.
- - One in httpd-ssl.conf for Apache to listen on port 443 for HTTPS traffic.
- Any traffic meant for the platform coming on these ports is forwarded to Tomcat. This request forwarding is achieved using the mod_proxy module.
- The httpd.conf and httpd-ssl.conf configuration files can be found in the $APACHE_HOME directory. For each virtual host entry, make sure the following are set appropriately:
ServerAdmin root@localhost DocumentRoot "/usr/local/apache/htdocs/" ServerName example.com ServerAlias www.example.com
- 2. Configure certificates for SSL virtual host on port 443
- Make sure the following points to your certificate and private key.
- httpd-ssl.conf:
SSLCertificateFile "{install_dir}/tomcat/conf/RN/<yourdomain>.cer" SSLCertificateKeyFile "{install_dir}/tomcat/conf/RN/<yourdomain>_private_key"
- 3. Enable secure communications
- Set up a secure communications channel between Apache and Tomcat. The URLs for the Proxy are then https://, rather than http://, and all traffic is encrypted.
SSLProxyEngine on
ProxyPreserveHost On
- 4. Configure Proxy Paths in Apache
- Proxy paths need to be configured to enable mod_proxy to recognize the request and forward it to Tomcat appropriately as shown in the following examples:
ProxyPass /networking/* https://tomcat-host-name:8284/networking/* ProxyPassReverse /networking/* https://tomcat-host-name:8284/networking/*
ProxyPass /networking/* http://tomcat-host-name:8283/networking/* ProxyPassReverse /networking/* http://tomcat-host-name:8283/networking/*
For ace-lib components, make the updates as per the following examples:
ProxyPass /ace-lib/* https://tomcat-host-name:8284/ace-lib/* ProxyPassReverse /ace-lib/* https://tomcat-host-name:8284/ace-lib/*
ProxyPass /ace-lib/* http://tomcat-host-name:8283/ace-lib/* ProxyPassReverse /ace-lib/* http://tomcat-host-name:8283/ace-lib/*
- Note: These instructions assume that Tomcat and Apache are running on the same server. If they are running on different servers, change the settings appropriately.
- 5.Configure virtual hosts
- For each virtual host in httpd.conf and httpd-ssl.conf for ports 80 and 443, copy the settings from {install_dir}/httpd-proxy.conf, changing the standard port assignment of 8283 and 8284, if needed.
For more information on changing ports, see Controlling Port Access.
In case, you are trying to access AgileApps server without an Apache server and if you use any port number other than 80 and 443, you have to access the AgileApps URLs along with the port numbers. For example: https://{hostname}:8284/networking/servicedesk/index.jsp </noinclude>
7.3 Adding Additional Application Servers
Adding additional instances of the platform can help to load balance traffic and improve performance.
7.3.1 Considerations
- Paths: It's a good practice to use the same paths for platform installation folders and files on each server.
- Distribution of backend services: Running backend services on different servers can help to balance the load. For example, the Report, Bounce, and Import services can run on one server, while and the rest of the services run elsewhere.
- Learn More: Enabling and Disabling Backend Services
- Single-instance services: Each backend service should run on only one server. It is therefore important to ensure that no service is enabled on more than one server.
- An additional install behaves like an upgrade: When the installation process finds that the database already exists, it does an "upgrade", rather than a "fresh install". That consideration has several implications:
- The version of the platform installed on the new server must match the version installed on the existing servers. Otherwise, the database will be automatically upgraded, and the existing instances will find that they can no longer access the database.
- It is necessary to Configure Backend Services after the install.
(If the service is already running on an another server, you should Disable the Backend Service first.)
- Quartz is enabled on every install: The Scheduler service is required for platform operations, so it is enabled on every install, whether it is a fresh install or an upgrade. In consequence, you should Disable the Service on the existing server, before doing the new install.
7.3.2 Overview of the Process
- Disable conflicting services on the original server(s). (In particular, Quartz which will always be in "conflict".)
- Install the new server.
- Manually configure backend services it will be hosting.
- Manual configure changes to the original server(s), as needed, and restart them.
7.3.3 To Install an Additional Instance
- Disable all services on the original server(s).
- Install the platform on the additional servers.
- Verify database connections:
- Make sure the database server can be accessed from the new server.
- Verify the login credentials to the database server from the new instance.
- Configure Backend Services that will be running on the new server.
- Make the corresponding adjustments to the backend-services configuration on the original instances.
- Start/restart all servers.
7.4 Adding Additional memcached Servers
Adding an additional memcached servers helps to distribute the cache and balance the load across the memcached servers.
- To add an additional memcached server
- Edit networking.properties on all instances.
- Update the MemCachedServers entry
- Restart all servers, following the Server Restart Sequence
Note: The sequence is important!
- Example
There is one memcached server running on serverA (192.168.0.1) on port 11211. On each instance, the networking.properties has this:
MemCachedServers=192.168.0.1:11211
A second memcached server is brought up on serverB (192.168.0.2) on port 11211. The networking.properties file is edited on all instances so it looks like this:
MemCachedServers=192.168.0.1:11211 192.168.0.2:11211
After following the Server Restart Sequence, all instances are using the new, distributed cache.
If a 3rd memcached server is added (192.168.0.3), the networking.properties file will look like this:
MemCachedServers=192.168.0.1:11211 192.168.0.2:11211 192.168.0.3:11211
Again, the Server Restart Sequence is followed to enable the new configuration.
8 Application Server Administration
8.1 Server Restart Sequence
When an installation employs memcached or the Messaging Server, it is important to follow this sequence when restarting servers:
- <syntaxhighlight lang="bash" enclose="div">
- STOP THE MESSAGING SERVER, if one is running:
/etc/init.d/messaging stop {hornetq-folder}/bin/stop.sh
- STOP ALL APPLICATION SERVERS
- On each server:
{install-dir}/profiles/IS_default/bin/shutdown.sh
- STOP ALL memcached SERVERS
- On each server:
/bin/memcached -d stop
- START ALL memcached SERVERS
- On each server:
/bin/memcached -d start -p {port} -u {user} -m {MB_of_memory}
# Typical values: # Port: 11211, User: root, MB of memory: 25
- START ALL APPLICATION SERVERS
- On each server:
{install-dir}/profiles/IS_default/bin/startup.sh
- START THE MESSAGING SERVER, if you're running one:
{hornetq-folder}/bin/start.sh /etc/init.d/messaging start </syntaxhighlight>
- Considerations
-
- Stopping application servers ensures that they aren't adding entries to the cache.
- Stopping memcached makes sure that the cache is flushed.
- Those two steps can occur in either order. It is the next two for which order is critical:
- Restarting memcached first makes sure that a clean copy of the cache is available.
- When the application servers come up, they use the clean cache.
8.2 Start the Application Server
- Login as root
- {install-dir}/profiles/IS_default/bin/startup.sh
8.3 Restart the Application Server
- Login as root
- {install-dir}/profiles/IS_default/bin/restart.sh
8.4 Stop the Application Server
To stop the application server, you kill the Apache Tomcat container it's running in.
- Login in as root
- {install-dir}/profiles/IS_default/bin/shutdown.sh
8.5 Accessing the Application Server
After the application server has been started:
- 1. Access the Service Provider URL: http://{yourDomain}/networking/Service?t=1&targetpage=ViewPort.jsp:
- Where:
- {yourDomain} is the platform domain. For example: yourcompany.com
- Default username and password: Use the credentials you typed into the Software AG installer.
-
Note: These credentials are only applicable to the on-premise fresh installations of AgileApps and not for AgileApps cloud.
- The platform will request that you change your password after you login the first time.
- 2. Setup the Service Configuration with these required parameters:
- Configure the Service Settings and specify Service and Domain names:
Parameter Description Typical Value Service Name Name of the service provider Financiocorp Services Prefix for Service Domain Optional subdomain name Allowed characters: a-z, A-Z, 0-9, - (alphanumeric, plus hyphen)
Example: service
Service Domain The Domain Name part of the URL mydomain.com Domain URL Read Only
Automatically populated as:- Prefix + Service Domain
service.mydomain.com
8.6 Creating a Tenant
- After Login, create a new tenant from a web browser using the URL as follows:
- https://yourdomain.com/networking/Service?t=2308
- Where yourdomain.com is the name of your domain
9 Database Administration
9.1 Recognizing the Platform Databases
The platform installation contains the following databases:
- relationals - Holds information about all tenants and a global authentication table for all users.
- db1000 – A database-template used to create new tenants.
- db892085391 – The "administration tenant" used to administer other tenants. Admins for the platform log in here.
- db##### - Databases for other tenants, where the number in the name is the tenant's ID. (In the platform GUI, use the Accounts tab to find a tenant's ID number.)
9.2 Database Administration Tips
- Ensure that all database servers and web servers are configured to be in the same timezone, regardless of where they are geographically located.
- You should have different user accounts with different access permissions. These are typical:
- A root user with all permissions, accessed from a specific host only.
- For access from other web servers, one root user for each host.
- An admin user with limited privileges to do a DB dump, monitor the DB server, and/or monitor the replication server.
- A read only user who cannot modify the database for reports, backups, and exports.
- A replication user that is used only by the replication process.
- Always do DB dumps, reporting, and exports from the replication server, as those activities introduce locks that interfere with users.
- Analyze queries from the replication server. Do not login as root.
- Do not login as root on the production server unless executing a DML or DDL.
- Always use transactions when executing a DML (begin- and end-rollback).
9.3 Database Backup and Recovery
MySQL can be backed up using the mysqldump command - http://dev.mysql.com/doc/refman/5.5/en/mysqldump.html
Tip: The replicated database server should be used for backups.
To set up for it, see Configuring MySQL to Run on a Separate Server
9.3.1 Standard Database Backup
- Dump the database
mysqldump –uroot –pxxx –all-databases –quick –routines –result-file=dumpfile.sql
- Dump the database at regular intervals, using a Linux cron job
-
- Put the mysqldump command in a shell script - say, xyz.sh
- Setup cron job e.g. every day at 6 pm
0 18 * * * /yourscriptlocation/xyz.sh > /somedirectory/xyz.out 2>&1
- Use tar to compress the resulting dump file, to save space.
- Maintain daily backups for ten days or so, to reclaim the space they use.
- Maintain monthly backup. Save the dump of last day of the month, for example, and retain the dump file for a year or so.
- Store long-term backups offsite.
9.3.2 Standard Database Restore
- Restoring all data from a dump file
- Drop all the databases.
- Restart mysqld
- Start the restore process and run in background:
mysql –uroot –pxxxx < dumpfile.sql > dumpfile.out 2>&1 &
- Restoring a single database or table from a dump file
- Drop the old database/table:
mysql –uroot –pxxxx $database < dumpfile.sql > dumpfile.out 2>&1 &
9.3.3 Learn More
For more detailed information on database backup and recovery in MySQL, see:
Managing Email Credits with Vertical Response
10 Housekeeping
10.1 Regular Maintenance Ensures Top Performance
Regular housekeeping helps to keep the platform running at peak efficiency.
You'll know that housekeeping is necessary when your Platform Monitoring reveals a decreased level of performance, despite your Performance Optimizations and Platform Tuning.
But regular, planned maintenance can prevent such slowdowns from occurring in the first place.
As part of that process, you will be:
10.2 Purging and Archiving Data on the Platform
To keep storage requirements to a minimum, perform these actions at regular intervals, using the platform GUI:
- Audit logs are purged after a retention period which can be configured.
- Debug logs used for development are retained for a day.
- When a record is deleted, it is moved to the Recycle Bin where it is purged after 30 days.
- The platform provides a data export mechanism which can be used for data archiving.
- Data can also be read using the REST API and saved elsewhere.
10.3 Cleaning Up Temporary Files on the Server
Every few months, it's a good idea to remove old temporary files, to keep them from accumulating on the server. To do that:
- Delete old CSV files in the documents folder.
For example, delete *.csv files that are more than 5 days old. - Delete old files in the temp folder.
For example, delete files more than 5 days old.- To find the location of the temp folder:
- From the Service Provider URL: http://{yourDomain}/networking/Service?t=1&targetpage=ViewPort.jsp
- Go to Settings > Service Provider Settings > Service Configuration.
- To find the location of the temp folder:
- Delete old tomcat logs.
For example, delete files more than 10 days old in {install-dir}/profiles/IS_default/logs - Delete old apache logs.
For example, delete files older than 10 days in {apache-install-dir}/apache/logs
11 Platform Monitoring
11.1 Heartbeat Check
The Heartbeat Check checks to make sure that the platform is still running. It's accomplished by visiting a URL which gets processed by Tomcat. (This lightweight URL gets an gets an immediate response from Tomcat, if it is running.)
-
Note: If Tomcat runs out of memory is not responding for some other reason, you won't be able to tell by monitoring the port or the process. The Heartbeat Check is the only way to know.
If you happen to be logged in, you can visit https://{yourDomain}/networking/rest in your browser. If Tomcat is running, you get a list of REST APIs supported by the platform.
To run the same test without logging in, use a REST client to visit the URL and specify the HEAD method (rather than GET, PUT, or POST). If Tomcat is running, you get an http return code of 200.
11.2 Monitoring Servers
Any server in the system can become a bottleneck, at some point, so it's a good idea to continually monitor the health of the critical processes that run on them. This section lists the components to monitor on each server.
11.2.1 Monitoring Application Servers
- tomcat availability and CPU utilization. Check threads, connection pool size, sticky sessions, KeepAliveRequests, etc.
- GC – Allocation and de-allocation of memory on the JVM. (Monitoring and Tuning Garbage Collection)
- OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics)
11.2.2 Monitoring Web Servers
- apache-httpd availability
- OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics)
11.2.3 Monitoring Database Servers
- mysql availability. Check caches, buffers, connections, timeouts. (Monitoring and Tuning MySQL)
- OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics)
- Note: If replication is employed, scripts can be written to check the health of the replication and report replication lag times.
11.2.4 Monitoring memcached Servers
- memcached availability
- OS (Linux) CPU utilization, IO activity, swap ratio, context switches, etc. (Monitoring OS Statistics)
11.2.5 Monitoring the Network
- Dropped packets
- Socket wear and tear
11.3 Monitoring Processes
These pages tell you how to monitor the critical processes on each server:
11.3.1 Monitoring and Tuning Garbage Collection
To monitor the frequency of Garbage Collection inside the JVM:
$java_home/bin/jstat -gcutil pid <intervalmillis> [root@web1]# /usr/local/java/bin/jstat -gcutil 12929 5000 S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 42.66 37.32 12.82 26.96 4845 145.414 26 1.745 147.159 0.00 42.66 37.78 12.82 26.96 4845 145.414 26 1.745 147.159 0.00 42.66 57.70 12.82 26.96 4845 145.414 26 1.745 147.159
Tuning parameters are set in catalina.sh under CATALINA_OPTS
Learn More: http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html
11.3.2 Monitoring and Tuning MySQL
- Running on a 64-bit machine, there should be plenty of memory.
- 20G of memory or more should be allocated to MySQL (highly recommended).
- Allocate a good amount of memory to innodb_buffer_pool_size - the combined cached for data and the index.
- TimeOuts
- interactive_timeout, wait_timeout,innodb_lock_wait_timeout.
- Number of file descriptors (number of files that MySQL can have open)
- Open_files_limit.
- Also: Check that you have a sufficient number of file descriptors available in the OS.
- To monitor the number of active threads, live threads, and open threads:
- show processlist or show full processlist
- After modifying tuning parameters in my.cnf, always restart mysql.
Learn More: http://dev.mysql.com/doc/refman/5.5/en/server-parameters.html
11.3.3 Monitoring OS Statistics
Before tuning any stack, whether for a web server or a database server, you should know how heavily system resources are being used. To get the information, use the following commands:
- top
top - 15:50:31 up 309 days, 21:10, 1 user, load average: 0.06, 0.16, 0.15 Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12929 root 18 0 3680m 1.2g 12m S 17.6 32.3 76:19.02 java
- mpstat
03:51:30 PM CPU %user %nice %system %iowait %irq %soft %idle intr/s 03:51:30 PM all 1.79 0.00 0.29 0.43 0.01 0.06 97.42 425.80 03:51:30 PM 0 3.22 0.00 0.43 0.47 0.02 0.21 95.65 151.15 03:51:30 PM 1 1.08 0.00 0.21 0.68 0.00 0.01 98.02 103.64 03:51:30 PM 2 1.65 0.00 0.28 0.38 0.00 0.02 97.68 96.45 03:51:30 PM 3 1.24 0.00 0.23 0.19 0.00 0.01 98.33 74.56
- vmstat
procs -----------memory----------- --swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 22264 1110200 168460 1120660 0 0 1 39 0 0 2 0 97 0
- free –m
total used free shared buffers cached Mem: 3945 2862 1083 0 164 1094 -/+ buffers/cache: 1603 2342 Swap: 1027 21 1005
Learn More:'
11.4 Monitoring Ports
Experience has shown that monitoring the processes and doing a health check tends to be a lot more useful than monitoring ports. But if you need to, here are the ports to monitor:
- Apache-httpd - tcp/80 and tcp/443
- memcached - tcp/11211
- MySQL - tcp/3306
- Tomcat - The ports Tomcat is listening on are listed in the following files available under {install-dir}/profiles/IS_default/configuration/com.softwareag.platform.config.propsloader folder:
- com.softwareag.catalina.connector.http.pid-agileappsHttp-8283.properties
- com.softwareag.catalina.connector.https.pid-agileappsHttps-8284.properties
11.5 Monitoring Memcached
Performance statistics can be retrieved using memcached commands.
Learn More: http://code.google.com/p/memcached/wiki/NewCommands#Statistics
11.6 Monitoring Cluster--Enabled Quartz Instances
This process sets up an automatic check that sends an email alert when Quartz is down.
- Verify that these lines are in the AALSchedulers.xml file. (They are present in new installations, but are not automatically added during upgrades.)
- <syntaxhighlight lang="perl" enclose="div">
<attribute name="org.quartz.scheduler.instanceId" = value="AUTO"/> # Automatically assigns instance IDs to prevent conflicts ... <attribute name="org.quartz.jobStore.misfireThreshold" = value="60000"/> # Prevent infinite loop if a job can't start, but keeps retrying ... <attribute name="org.quartz.jobStore.isClustered" = value="true"/> # Turn on clustering <attribute name="org.quartz.jobStore.clusterCheckinInterval" = value="20000"/> </syntaxhighlight>
- Download the monitor_quartz.pl (Perl) script from the platform downloads area.
- Set these variable values for your installation:
- $mysql_username - Database user name.
- $mysql_password - Database password.
- $mysql_host - Name of server the database is running on.
- $config_file - Absolute path to tomcat/webapps/networking/WEB-INF/web.xml
- $recipients - Comma-separated list of email addresses who will receive email status messages
- Run the script in the background.
You'll get an email alert whenever the script discovers that Quartz isn't running.
- How it Works
- The lines in the properties file cause quartz to put an entry into the relationals.QRTZ_SCHEDULER_STATE table in the database. The script checks the timestamp to verify that the last stamp was within 5 minutes. (Quartz should be running once a minute. If it hasn't run in five minutes, it is considered to be down.)
12 Platform Tuning
12.1 Tuning Tomcat
The platform is a web application that runs in an Apache Tomcat container. The parameters listed in this section can be modified to optimize Tomcat's performance.
12.1.1 Connector Port Parameters
- Thread Allocation
- maxThreads, minSpareThreads, maxSpareThreads, maxkeepAliveRequests
- Timeouts
- connectionTimeOut, keepAliveTimeOut, connetionLinger
12.1.2 Connection Pooling Parameters
- initialSize, maxActive, maxIdle, maxWait, removeAbandoned, removeAbandonedTimeout
12.1.3 Learn More
12.2 Tuning Report Threads
The number of report threads can be configured in the networking.properties file:
- report_threads=2
- Assigns number of threads to be used by the report server.
Generally, two threads devoted to reporting is about right. If reports aren't arriving on time, consider incrementally expanding the number of threads to see if performance improves (while keeping an overall eye on system resources, to ensure that increasing the number of threads does't incur other bottlenecks.
For maximum reporting efficiency (if needed), setup a dedicated backend server for reporting. Learn More: Managing Backend Services
13 Troubleshooting
13.1 Watching Application Server Logs
The log files used by the platform are as follows:
- {install-dir}/profiles/IS_default/logs/relationals.log
- {install-dir}/profiles/IS_default/logs/sag-osgi.log
- {install-dir}/profiles/IS_default/logs/wrapper.log
- {install-dir}/profiles/IS_default/logs/platform.log
They contain informational messages as well as exceptions that help to troubleshoot specific problems. All messages are timestamped, with a prefix that indicates the module that recorded the message.)
- Examples
-
- A report scheduled to run at 2:00 is not created until much later. Examining the relationals.log for entries timestamped around that time can help to diagnose the problem. (The same applies to other scheduled events.)
- A java.lang.OutOfMemoryError exception in sag-osgi.log or wrapper.log indicates that tomcat has run out of memory.
Note: In general, the Heartbeat Check is the best way to monitor the availability of the application server.
13.2 Troubleshooting Quartz
If quartz stops processing jobs from the quartz queue, all scheduled jobs are listed in the NETWORK_SCHEDULE_POLICIES table. Ideally, there should be only 100 or so records/jobs scheduled in NETWORK_SCHEDULE_POLICIES.
Quartz lop entries can be viewed in relationals.log on the server where quartz is enabled. The log entries will look like this, where the worker threads are the ones that pick up and process jobs in the queue:
<timestamp> MasterScheduler ... <timestamp> [PlatForm_Worker-1] ... <timestamp> [PlatForm_Worker-2] ... <timestamp> RelationalsSchedulerWorker-1 ... <timestamp> RelationalsSchedulerWorker-2 ... ...
Here are some possible ways that quartz could get stuck:
- There are too many jobs/records in NETWORK_SCHEDULE_POLICIES
- Numbers in the 1,000's may cause quartz to become bogged down. You might want to determine what caused these many jobs to be inserted in the queue. If it is safe to delete them, they can be removed from the queue. Or they could be copied to a temporary table re-inserted later, at a time when there is no load on the server.
- Quartz might run into an error that causes the quartz thread to die.
- In that case, restarting the application server will bring the quartz thread back to life.
-
Tip: Set up a cron job that gets count of entries in the NETWORK_SCHEDULE_POLICIES. If the count hasn't changed after 20 mins or so, it usually means quartz is not picking up the jobs. (Alternatively use the REST APIs to create a task that monitors the status of that table.)