Securing Server-Agent Communications
Typically, the RHQ Server and the RHQ Agents talk to each other in the clear, meaning all communications traffic is unencrypted and no authentication is performed on either end. Many times, the environment in which you install your RHQ Servers and RHQ Agents do not warrant the extra setup time and runtime performance degradation you incur when enabling security on RHQ communications traffic (for example, if you already have a VPN and/or firewall protections in place that guard your RHQ Servers and RHQ Agents against intrusion). However, there are those that need or simply want the piece of mind of knowing their RHQ traffic is fully encrypted and authenticated. For those people, this section will describe the steps that you need to perform in order to fully secure the communications traffic between RHQ Servers and RHQ Agents. Please read this page which discusses the implications of running RHQ without security the communications channel.
 | There is a basic authentication mechanism employed by the server in which it assigns security tokens to its agents which are used to identify and "authenticate" registered agents. This token mechanism should not, however, be considered a strong authentication scheme for the purposes of protecting your RHQ network from infiltration. |
RHQ and SSL
RHQ utilizes SSL technology to perform both encryption and authentication. You can enable encryption (scrambling the data between server and agent to avoid someone eavesdropping on the traffic) and optionally enable authentication (which prohibits an intruder from attempting to spoof either a RHQ Server or RHQ Agent). It is recommended that you understand the basics of SSL and certificate-based security before attempting to secure your RHQ communications.
Encryption
By simply using a transport that uses SSL, you automatically get encryption. This means that when you configure your RHQ Server and RHQ Agent's communications layer, use an SSL-enabled transport to encrypt the traffic. An SSL-enabled transport includes https and sslsocket. You do not have to worry about setting up certificates if you want to use SSL-enabled transports just for encryption. The RHQ Server ships with a certificate and the RHQ Agent will create a self-signed certificate if it needs one.
Note that it is possible to just use SSL encryption without authentication. Some people may just wish to encrypt their RHQ traffic, without requiring the RHQ Servers and RHQ Agents authenticating each other. This setup, while less secure, is much easier to setup because it does not require creating and distributing trusted certificates.
Authentication
Authentication via SSL requires that you distribute trusted certificates to your RHQ Servers and RHQ Agents. You must also configure those RHQ Servers and RHQ Agents to reject any messages coming from remote clients that do not match any of those trusted certificates. In order to support authentication, you must use SSL-enabled transports (which include https and sslsocket). You must also obtain trusted certificates for all your servers and agents and package those certificates in a set of keystore and truststore files. You can configure the RHQ Server to authenticate RHQ Agents, RHQ Agents to authenticate the RHQ Server or both. RHQ provides this flexibility in case you only want to authenticate in one direction but not another. However, when people feel the need for authentication, they will usually enable it in both directions.
Authentication requires a bit more work to setup. Because true authentication requires a high degree of trust, you have to manually create and sign your certificates, create keystores and truststores that contain those certificates, then distribute your keystores and truststores in a highly secure manner to all your RHQ Servers and RHQ Agents. This may mean going so far as to physically hand-delivering your trusted certificates via CD and copying the certificates from the CD to all the computers hosting the RHQ Servers and RHQ Agents. Many times it isn't as highly paranoid as that - however, at some point along the way, you have to be able to place trust in whatever certificates you are using and distributing.
Setting Up Secure RHQ Communications
The following are the steps necessary to set up secure communications in RHQ. Remember that you have the option for encryption-only or encryption-with-authentication. If you choose the latter, you do not need to perform any of the steps dealing with the creation, packaging and distribution of certificates and keystores/truststores. If you wish to have full security with encryption-with-authentication, follow all the steps listed below. If you are setting up with encryption-with-authentication, these steps assume you want authentication in both directions (that is, the agents will need to authenticate with the server and the server will need to authenticate with the agents).
Step 1 - Use SSL Transport To Enable Encryption
The first thing you need to do is tell the RHQ Servers and RHQ Agents to use SSL when talking to each other. The RHQ low-level communications layer is based on JBoss/Remoting and uses what is known as a transport to ship messages back and forth between agents and servers. A transport can be either unencrypted (like the raw socket transport or http) or it can be encrypted (like sslsocket or https). We are going to use a raw socket transport since it is faster (HTTP is a text-based protocol and requires binary messages to be encoded and decoded into HTTP messages - RHQ can use raw sockets to send the binary messages without requiring it to build and parse HTTP headers). But, that said, you can easily replace https in place of sslsocket in these instructions if you wish to use HTTPS (you might wish to do this, for example, if you have a firewall that allows only HTTPS traffic to flow).
RHQ Server Instructions
- Shutdown the RHQ Server
- Go to the /bin directory under the location where your RHQ Server is installed
- In that directory, find the file rhq-server.properties and load it into your favorite text editor
- Find the following configuration preferences used to set the server's communications connector and set them appropriately:
where transport is the SSL transport you wish to use, bind-address is the IP address/hostname that the server will bind its listening connector to, bind-port is any free server socket port the server will listen to for incoming agent requests and transport-params are JBoss/Remoting-specific configuration parameters used to configure how the server and agent interact with this connector. You may notice that the bind-address uses "${jboss.bind.address}" - this is the value as defined by the RHQ Server's JBossAS bind address. If you wish to change the bind address the RHQ Server uses (and thus the address used by the connector to bind to), you can modify the jboss.bind.address setting in rhq-server.properties.
- (Optional) You may wish to explicitly define the secure socket protocol used by this connector, although the default (TLS) is usually good enough. If the default protocol of TLS is not what you want, find the following configuration preferences and set them appropriately:
Note that you can define one protocol when sending to the agent (rhq.server.client...) and a different protocol when receiving messages from the agent (rhq.communications.connector...). Of course, you must make sure your agents' protocols are configured in reverse (i.e. the agent must know what protocol the server expects when receiving and what protocol the server uses when sending). Again, you usually do not need to change the secure socket protocols, as the defaults are normally fine. The important thing to understand is that the transport must be an SSL-enabled transport, and of course the bind-port must be set to any port as long as it is unused on your server host.
- Save the configuration file
- If you do not plan on continuing on in order to setup SSL authentication, then you can now restart the RHQ Server
 | If the rhq.communications.connector.transport setting is changed to sslservlet, it tells the RHQ Server to piggyback on the secured embedded Tomcat connector to accept agent messages via HTTPS on the default port (which is 7443). On the agent, you don't have to change anything other than changing "RHQ Server URI" so it uses the sslservlet transport protocol, as opposed to servlet. When you do this, you use the RHQ Server's Tomcat certificate and the agent's auto-generated self-signed certificate for encryption. However, if you want to enable certificate-based authentication, then you must follow the steps outlined in this section; otherwise, you would need to configure Tomcat's keystore and truststores appropriately. We will not discuss this option here. See the Tomcat documentation if you wish to further explore this. |
RHQ Agent Instructions
- You must enter the RHQ Agent's advanced setup mode. You can do this in one of the following ways:
- Start the agent with the the command line option --setup --advanced
- If the agent is already running in non-daemon mode, go to the console, enter shutdown to make sure its down and then enter setup advanced
| While in setup mode, enter !? at any prompt to get help |
- Answer all prompts until you get to the prompt asking for the Agent Transport Protocol and enter sslsocket
- At the prompt asking for the RHQ Server IP Address, enter the bind address of the RHQ Server from the configuration settings from the previous section. That is, you must use the RHQ Server's bind address as defined by its rhq.communications.connector.bind-address setting.
- At the prompt asking for the RHQ Server Port, enter a bind port of the RHQ Server from the configuration settings from the previous section. That is, you must use the RHQ Server's new bind port as defined by its rhq.communications.connector.bind-port setting.
- At the prompt asking for the RHQ Server Transport Protocol, enter sslsocket (which is the RHQ Server's new transport as defined by its rhq.communications.connector.transport setting.
- At the prompt asking for the RHQ Server Transport Parameters, enter the parameters of the RHQ Server configuration settings from the previous section. That is, you must use the RHQ Server's new transport parameters as defined by its rhq.communications.connector.transport-params setting.
- (Optional) You may wish to explicitly define the agent connector's transport parameters. You may also wish to explicitly set the secure socket protocols used by the agent (the default protocol of TLS is usually sufficient, however). These settings can also be set when in advanced setup mode.
- Agent Transport Parameters are the optional JBoss/Remoting transport parameters that are used by the agent and server to interact with the agent's connector
- Incoming Secure Socket Protocol is the protocol used when accepting incoming messages from the RHQ Server (make sure this matches the RHQ Server's protocol setting rhq.server.client.security.secure-socket-protocol)
- Outgoing Secure Socket Protocol is the protocol used when sending outgoing messages to the RHQ Server (make sure this matches the RHQ Server's protocol setting rhq.communications.connector.security.secure-socket-protocol)
- Exit the agent (effectively shutting it down) and then restart it
At this point, you have now configured your RHQ Servers and RHQ Agents to encrypt their messages to each other via SSL. You can stop here and be assured that no one can effectively eavesdrop on the communications between them. However, if you wish to strengthen the security of the network traffic even further, continue on to the next series of steps which will enable certificate-based authentication between your RHQ Servers and RHQ Agents.
Step 2 - Prepare Your SSL Certificates
If you wish to have your RHQ Servers and RHQ Agents authenticate one another, you need something that "identifies" each one of them. SSL requires digital certificates for this purpose. If your company or organization can request and receive officially signed certificates from a trusted CA, you will need to obtain one certificate for each of your RHQ Servers and RHQ Agents that you plan on deploying in your RHQ environment. If you already have the certificates given to you
from your CA, you must place each of them in their own keystore file and combine all of them into a single truststore file. Otherwise, follow these set of instructions to create your own certificates.
The purpose of these instructions is to generate a keystore file and a truststore file for each RHQ Server and RHQ Agent. Each keystore file will contain a single self-signed certificate that belongs to one of the RHQ Server or RHQ Agent entities. Each truststore file contains all the certificates belonging to every RHQ Server and RHQ Agent.
| These instructions assume you have access to the keytool utility that comes with Sun Microsystem's Java distribution. If you are using another vendor's Java implementation, these instructions may or may not be exactly what you need. Consult your vendor's documentation on how to use their key generation tools. |
- For each RHQ Server or RHQ Agent, execute this command to generate its keystore file:
In this example, assume one of my RHQ Agents will be installed on a machine whose hostname is "myhost.mycorp.com". This command creates and self-signs a certificate and stores it under the alias "myhost" in the file "myhost-keystore.dat". The certificate is good for 10 years (3650 days). The certificate keys were generated using the DSA algorithm and were stored in the keystore using the JKS format. I've password-protected both the key itself and the keystore file as a whole with the password "rhqpassword". It is recommended that you at least choose your own passwords when generating your keystores.
The important part is to make sure you set the Common Name (CN) of the Distinguished Name (the -dname option) to the correct address where this keystore is to be installed. That's because as part of the SSL handshake, a remote client will attempt to verify that the issuer of the certificate (as listed in the CN) is the same name as where the certificate actually came from. Now that I have generated a self-signed certificate for this RHQ Agent and stored it in a file named "myhost-keystore.dat", I store that away for now and continue generating certificates/keystores for the rest of my machines until I have one keystore file for each and every machine that will host a RHQ Server or RHQ Agent. It is best to name the keystores so you remember which keystore file belongs to which machine (hence why, in the example above, the hostname was part of the filename). Same holds true with the alias names.
- Put each self-signed certificate you generated in the previous step in a single truststore file. You do this by exporting each certificate from each keystore and importing them all into a single truststore file.
- For each keystore file, export the self-signed certificate:
This extracts the self-signed certificate from the previously created myhost-keystore.dat file and stores the certificate in the file "myhost-cert".
- For each exported self-signed certificate, import them into a single truststore file:
This command is similiar to the -genkey command used to create the original keystore certificate. However, rather than asking the keytool to generate a new certificate, we are giving it an existing certificate and asking keytool to place it in the truststore file. The -keystore option defines the name of our truststore file. -alias is the name that we assign this certificate within the truststore file (note that for convenience, we give it the same alias under which it was found in its keystore file).
- Repeat these steps for each keystore file you created. You want to import every certificate into the same truststore - eventually having a single truststore file that contains all of your certificates. For example, if I had a total of 5 RHQ Servers and RHQ Agents in my RHQ environment, I would have 5 separate keystore files but a single truststore.dat file that contains all 5 certificates. When you are all done, you can use keytool to list the certificates in your truststore file, to make sure you did them all:
Step 3 - Distribute Your Keystores and Truststores
At this point, you have created a set of keystore files (one for each RHQ Server and RHQ Agent in your RHQ environment) and a single truststore file (a duplicate copy is to be given to each RHQ Server and RHQ Agent). You must now distribute those files to all the machines where your RHQ Servers and RHQ Agents live. You must do so in a secure fashion - be careful no one can steal, intercept or otherwise manipulate your keystore/truststore files. You must also make sure that you distribute the keystore files to the host machines that match the certificates' CN host addresses. If you mix them up and, for example, put the "myhost" keystore file on the "anotherhost.mycorp.com" machine, the SSL communications will fail for the RHQ Server or RHQ Agent running on "anotherhost".
RHQ Server Instructions
Each RHQ Server distribution has a JBossAS within it. That JBossAS's /data directory is the location to store the server's keystore/truststore files (but technically, you can put them anywhere that the server can access them - just make sure you remember the location because you will need it for the next step). For each RHQ Server, take its keystore file (make sure the keystore file has the appropriate CN value that matches the RHQ Server's hostname) and store it in the RHQ Server's directory on-container\server\default\data under the name keystore.dat. Make a copy of your truststore file and place it in that same data directory as well, under the name truststore.dat.
RHQ Agent Instructions
Each RHQ Agent distribution has a /data directory (the agent will create one if it doesn't yet exist). It is the logical choice to store the agent's keystore/truststore files (but technically, you can put them anywhere that the agent can access them - just make sure you remember the location because you will need it for the next step). For each RHQ Agent, take its keystore file (make sure the keystore file has the appropriate CN value that matches the RHQ Agent's hostname) and store it in the agent's /data directory. Make a copy of your truststore file and place it in the agent's /data directory as well.
Step 4 - Tell The RHQ Servers and Agents About Their Keystores/Truststores
Now you have to tell your RHQ Servers and RHQ Agents where your keystore and truststore files are in addition to providing other information about those files so they can be read properly. After completing this step, your RHQ Servers and RHQ Agents will be able to successfully authenticate themselves to each other.
RHQ Server Instructions
- Shutdown the RHQ Server
- Go to the /bin directory under the location where your RHQ Server is installed
- In that directory, find the file rhq-server.properties and load it into your favorite text editor
- Find the following configuration preferences used to set the server's security:
What is being configured here is the server-side SSL security settings (rhq.communications.connector.security...) to handle incoming messages from RHQ Agents and the client-side SSL security settings (rhq.server.client.security...) to handle outgoing messages to RHQ Agents. Because we are sharing the keystore and truststore for both directions, a lot of these values are the same.
Note that the locations of the keystore and truststore files are configured by default to be in the on-container/server/default/data location (which is what "${jboss.server.data.dir}" resolves to) under the names keystore.dat and truststore.dat. If, for whatever reason, you don't want to put your keystore/truststore files in that location, you must change those values to the location of those files making sure that the JBossAS process has read permissions to those locations.
 | As mentioned earlier, we will be setting up SSL authentication for both sides of the communications channel - if you do not want this, it is in the rhq-server.properties file that you would define which side you want to configure your SSL authentication and which side you do not (specifically you would enable/disable the server-auth-mode-enabled and client-auth-mode appropriately). Because it is rare that you would want to do this, we will not delve any deeper into this optional configuration. |
Since we want to enable both server-side and client-side with SSL authentication, we set client-auth-mode to "need" (meaning, in order to process an incoming request, that request must have a valid SSL certificate) and we set server-auth-mode-enabled to "true" (meaning any outgoing messages sent to a RHQ Agent will only be sent if that RHQ Agent has a valid SSL certificate).
The rest of the settings and their values should be fairly self-evident. You are simply telling the RHQ Server where it can find its keystore and truststore files, the passwords to access data in those files, the alias of the RHQ Server's own certificate as found in the keystore, etc.
- Save the configuration file
- Restart the RHQ Server
RHQ Agent Instructions
- You must enter the RHQ Agent's advanced setup mode. You can do this in one of the following ways:
- Start the agent with the the command line options --setup --advanced
- If the agent is already running in non-daemon mode, go to the console, enter shutdown to make sure its down and then enter setup advanced
- Answer all prompts until you get to the prompts asking about security. Because we are sharing the keystore and truststore for both directions, a lot of these values are the same. The prompts to look for and their new values are:
- Client Authentication Mode: need
- Server Authentication Mode Enabled?: true
- Incoming Secure Socket Protocol: TLS
- Server-side Keystore File: data\myhost-keystore.dat
- Server-side Keystore Algorithm: SunX509
- Server-side Keystore Type: JKS
- Server-side Keystore Password: rhqpassword
- Server-side Keystore Key Password: rhqpassword
- Server-side Keystore Key Alias: myhost
- Server-side Truststore File: data\truststore.dat
- Server-side Truststore Algorithm: SunX509
- Server-side Truststore Type: JKS
- Server-side Truststore Password: rhqpassword
- Outgoing Secure Socket Protocol: TLS
- Client-side Keystore File: data\myhost-keystore.dat
- Client-side Keystore Algorithm: SunX509
- Client-side Keystore Type: JKS
- Client-side Keystore Password: rhqpassword
- Client-side Keystore Key Password: rhqpassword
- Client-side Keystore Key Alias: myhost
- Client-side Truststore File: data\truststore.dat
- Client-side Truststore Algorithm: SunX509
- Client-side Truststore Type: JKS
- Client-side Truststore Password: rhqpassword
- Exit the agent (effectively shutting it down) and then restart it
At this point, you have now configured your RHQ Servers and RHQ Agents to both encrypt their messages to each other and to authenticate each other via SSL. You can stop here and be assured that no one can effectively eavesdrop on your RHQ communications nor can an infiltrator attempt to spoof itself as a bogus RHQ Server or RHQ Agent.
Step 5 - Test Your Setup
Once you are done with the preceding steps, you can finally restart your RHQ Servers and RHQ Agents. They should begin to talk to each other normally - if you've done everything correctly, you will not notice anything different! If you want to confirm that they really are using SSL authentication, simply remove a keystore or truststore file from either a RHQ Server or RHQ Agent and you should begin to notice errors appear in their log files. Removing a keystore file or truststore file from a RHQ Agent will prohibit that agent from being able to send and receive messages to/from the RHQ Server - which you can confirm by looking at the agent log file and looking for error messages. After you've finished testing, make sure you remember to restore the keystore/truststore files. You can test the SSL authentication by creating another keystore for one of your RHQ Agents, replace that keystore with the original keystore and try to see if that RHQ Agent can talk to the RHQ Server. Because this new keystore has a certificate that does not exist in the RHQ Server's truststore, the RHQ Server will no longer trust that agent and reject its messages. In effect, you simulated an infiltrator trying to spoof a RHQ Agent and the RHQ Server detected this security breach.
