RHQ, the common services project for infrastructure management

  Dashboard > RHQ-Project > ... > Configuring RHQ > Communications Configuration
  RHQ-Project Log In | Sign Up   View a printable version of the current page.  
  Communications Configuration
 Browse Space
Added by John Mazzitelli , last edited by John Mazzitelli on Mar 12, 2008  (view change)
Labels: 
(None)

Communications Configuration

Both the RHQ Server and the RHQ Agent use the same underlying communications layer for their incoming and outgoing messages (which is based on JBoss/Remoting). Their configuration files look similar because many of their preference settings configure this communications layer. This page will document those settings so you can be able to configure your RHQ Servers and RHQ Agents appropriately.

Index

RHQ Agent Communications Services

When The RHQ Agent starts for the first time, it enters setup mode. You can also manually enter this setup mode by using the --setup command line option or use the setup agent prompt command. Once in setup mode, you are asked to provide values for a series of preference settings. Some of these settings involve setting up the communications services of the agent, and some of these settings are only prompted if you enter "advanced" setup mode or "all" setup mode (those "advanced" and "all" settings are marked with a (a) below). To enter advanced setup mode and thus be able to set advanced settings, use the --advanced command line option with --setup or use the prompt command setup advanced. To enter the "all" setup mode (which allows you to set every preference available), you must use the prompt command setup all.

  • Agent IP Address - this is the address that the agent will bind to when listening for incoming messages. This usually is the same address that the agent's remote clients (aka RHQ Servers) will use when trying to connect to the agent. However, in some network setups, this may not always be the case (e.g. a remote client going through a router that forwards requests to a different host). If you want the RHQ Server to connect to this RHQ Agent via a different address, you need to set up some special transport params to indicate this, as described below.
  • Agent Port - this is the port that the agent will actually be listening to. This usually is the same port that the agent's remote clients (aka RHQ Servers) will use when trying to send messages to the agent. However, in some network setups, this may not always be the case (e.g. a remote client going through a router that forwards requests to a different port). If you want the RHQ Server to connect to this RHQ Agent via a different port, you need to set up some special transport params to indicate this, as described below.
  • Agent Transport Protocol (a) - this is the transport the agent expects incoming messages to adhere to. This is usually "socket" or "sslsocket" (for raw binary socket messages, either unsecured or secured). JBoss/Remoting has several different types of transports available, see its documentation if you wish to experiment with other types of transports.
  • Agent Transport Parameters (a) - these are additional parameters that are to be used when the agent creates its communications services and when its remote clients (aka RHQ Servers) needs to talk to the agent. See the section below, Transport Parameters, for the different types of transport parameters you can set.
  • RHQ Server IP Address - this is the IP address for the endpoint of the primary RHQ Server this agent will talk to. This RHQ Server IP Address value is dictated by the way the RHQ Server has configured its communications services (using similar settings such as these being described for the agent). Consult your RHQ Server's communications configuration to see what exact value this should be.
  • RHQ Server Port - this is the port that the primary RHQ Server is listening to. This RHQ Server Port value is dictated by the way the RHQ Server has configured its communications services (using similar settings such as these being described for the agent). Consult your RHQ Server's communications configuration to see what exact value this should be.
  • RHQ Server Transport Protocol (a) - this is the transport the primary RHQ Server will expect its incoming messages to flow over. This RHQ Server Transport value is dictated by the way the RHQ Server has configured its communications services (using similar settings such as these being described for the agent). Consult your RHQ Server's communications configuration to see what exact value this should be.
  • RHQ Server Transport Parameters (a) - this is additional transport parameters that are to be used when the agent connects to the primary RHQ Server. This RHQ Server Transport Parameters value is dictated by the way the RHQ Server has configured its communications services (using similar settings such as these being described for the agent). Consult your RHQ Server's communications configuration to see what exact value this should be. Particularly, you need to know what additional transport parameters the RHQ Server wants its clients to define. This is especially important if the RHQ Agent needs to connect to a different host and/or port than what the RHQ Server actually binds to.
  • Command Send Timeout (a) - this is the amount of milliseconds the agent will wait before aborting a command (i.e. the amount of time in milliseconds that the RHQ Server has in order to process commands and return its results). Make sure this value is the same as the timeout specified in the transport parameters of your RHQ Server URI (if specified), since both timeouts will be enforced. If this Command Send Timeout value is less than or equal to 0, the agent will not timeout its messages (note that if a timeout is specified in the RHQ Server URI transport parameters, that timeout will be enforced).
  • Command Send Retry Interval (a) - This is the minimum amount of time, in milliseconds, the agent will wait before trying to resend a guaranteed command that previously failed.
  • Command Send Max Retries (a) - If a guaranteed delivery message is sent, but the agent fails to connect to the server and deliver the message, it will
    always be retried. However, if the error was something other than a 'cannot connect' error, the command will only be retried this amount of times before the command is dropped (at which time it will be considered lost forever).
  • Maximum Commands To Concurrently Send (a) - This is the maximum number of commands the agent can send to the server at any one time. If you defined clientMaxPoolSize in your RHQ Server URI transport parameters, make sure its value is the same as this "Maximum Commands To Concurrently Send" value since you effectively can't have one higher than the other.

RHQ Server Communications Services

Settings analogous to those described above exist on the RHQ Server side as well. In order to set these communications configuration preferences on the RHQ Server, you edit their values found in the /bin/rhq-server.properties file. You will have to shutdown and restart the RHQ Server if you ever change values found within this properties file.

Many of these properties found in rhq-server.properties define system properties that are substituted in the comm overrides file found in jbossas/server/default/deploy/rhq.ear/server-comm-service.xml which override the defaults defined in server-comm-configuration.xml (found within the same directory). If you are unsure what a particular setting is for in the rhq-server.properties file, read Configuring the Server#StartupProperties or look at the server-comm-configuration.xml file which contains detailed comments for each of these settings.

Note that the rhq.communications.connector.transport-params setting allows you to set custom Transport Parameters just like you could do for the agent. Also note that the agent's RHQ Server URI preference value is a combination of the following RHQ Server settings:

  • rhq.communications.connector.transport
  • rhq.communications.connector.bind-address
  • rhq.communications.connector.bind-port
  • rhq.communications.connector.transport-params

So, for example, if the following settings are used to configure your RHQ Server:

rhq.communications.connector.transport=socket
rhq.communications.connector.bind-address=192.168.0.10
rhq.communications.connector.bind-port=16163
rhq.communications.connector.transport-params=enableTcpNoDelay=true&backlog=200

then when you setup your RHQ Agent, its configuration values would be:

RHQ Server IP Address=192.168.0.10
RHQ Server Port=16163
RHQ Server Transport Protocol=socket
RHQ Server Transport Parameters=enableTcpNoDelay=true&backlog=200

Transport Parameters

Both the RHQ Server and RHQ Agent use JBoss/Remoting as its underlying remoting framework. JBoss/Remoting defines remote endpoints (which identify how to connect to a RHQ Server or RHQ Agent) via InvokerLocator strings, which look like simple URLs. An example is socket://myhost.corp.com:16163/?transportParam1=value1&param2=value2. InvokerLocators consist of a transport protocol (socket:) as well as the host and port of the remote endpoint. Also as part of an InvokerLocator, are transport parameters which further customize the endpoint. They help define the behavior of the underlying communications connector (which is the thing that accepts incoming messages from remote clients). The RHQ Server and RHQ Agent can each define their own transport parameters via their rhq.communications.connector.transport-params preference setting (the agent's advanced setup mode will prompt you with Agent Transport Parameters when asking for this value).

Transport parameters are appended to the end of the InvokerLocator - in the same way a query string is appended to a URL. When you define your transport parameters, you must set them using the same syntax, specifically, multiple transport parameters are separated by ampersand characters. If you set transport parameters inside your XML config file, be sure you use the proper & string to represent the ampersand.

Below you will find listed all the possible transport parameters that can be defined in your RHQ Server or RHQ Agent rhq.communications.connector.transport-params setting. They are split into two groups: the first group are settings that affect the server-side socket behavior and the second group are settings that affect the client-side behavior. In this context, "server-side" refers to the connector that accepts incoming messages and "client-side" refers to the remote clients that send outgoing messages. RHQ Servers and RHQ Agents have both a server-side and client-side since they both send and receive messages from each other.

Refer to the JBoss/Remoting documentation if you wish to get additional information on these settings and their low-level implementation details.
  • Settings that affect the server-side
    • serverBindAddress - The address on which the server socket binds to listen for requests. The default is an empty value which indicates the server socket should be bound to the host provided by the InvokerLocator URL (the host).
    • serverBindPort - The port to listen for requests on.
    • timeout - The socket timeout value. The default on the server side is 60000 (one minute). If the timeout parameter is set, its value will also be passed to the client-side (see below).
    • backlog - The preferred number of unaccepted incoming connections allowed at a given time. The actual number may be greater than the specified backlog. When the queue is full, further connection requests are rejected. Must be a positive value greater than 0. If the value passed if equal or less than 0, then the default value will be assumed. The default value is 200.
    • numAcceptThreads - The number of threads that exist for accepting client connections. The default is 1.
    • maxPoolSize - The number of server threads for processing client requests. The default is 300.
    • socket.check_connection - indicates if the invoker should try to check the connection before re-using it by sending a single byte ping from the client to the server and then back from the server. This config needs to be set on both client and server to work. This is false by default.
  • Settings that affect the client-side
    • clientConnectAddress - the IP address or hostname the client will use to connect to the server-side socket. This would be needed in the case that the client will be going through a router that forwards requests made externally to a different IP address or hostname internally. If no clientConnectAddress or serverBindAddress is specified, the local host's address is used.
    • clientConnectPort - the port the client will use to connect to the server-side socket. This would be needed in the case that the client will be going through a router that forwards requests made externally to a different port internally.
    • timeout - The socket timeout value. The default on the client side is 1800000 (or 30 minutes).
    • enableTcpNoDelay - can be either true or false and will indicate if the client socket should have TCP_NODELAY turned on or off. TCP_NODELAY is for a specific purpose; to disable the Nagle buffering algorithm. It should only be set for applications that send frequent small bursts of information without getting an immediate response. The default is false.
    • clientMaxPoolSize - the client-side maximum number of active socket connections. This basically equates to the maximum number of concurrent client calls that can be made from the socket client invoker. The default is 50.
    • numberOfRetries - number of retries to get a socket from the pool. This basically equates to the number of seconds the client will wait to get a client socket connection from the pool before timing out. If the max retries is reached, a CannotConnectException will be thrown. The default is 30.
    • numberOfCallRetries - number of retries for making the invocation. This is unrelated to numberOfRetries in that when this comes into play is after it has already received a client socket connection from the pool. However, it is possible that the socket connection timed out while waiting within the pool. Since a connection check is not done by default, the connection is thrown away and an attempt to get a new one will be made. This will happen for however many numberOfCallRetries is (which defaults to 3). However, when (numberOfCallsRetries - 2) is reached, the entire connection pool is flushed under the assumption that all connections in the pool have timed out and are invalid and will start over by creating a new connection. If this still fails, a MarshalException is thrown.
    • socket.check_connection - Indicates if the invoker should try to check the connection before re-using it by sending a single byte ping from the client to the server and then back from the server. This config needs to be set on both client and server to work. This if false by default.

Here's an example that illustrates how a single InvokerLocator URL with transport parameters can configure both server-side and client-side behavior. Let's assume we are configuring a RHQ Server with the following settings:

  • transport: socket
  • bind-address: 192.168.0.5
  • bind-port: 56565
  • transport-params: backlog=300&enableTcpNoDelay=false

The full RHQ Server URI would then look like:

socket://192.168.0.5:56565/?backlog=300&enableTcpNoDelay=false

The backlog transport parameter is a server-side configuration and the enableTcpNoDelay transport parameter is a client-side configuration. When the RHQ Server creates its server socket and begins listening for incoming messages from RHQ Agents, it sets its backlog to 300. The RHQ Server ignores the enableTcpNoDelay parameter because it is only useful for clients that want to talk to its server socket. When a RHQ Agent sends a message to this RHQ Server, the RHQ Agent will ignore the backlog parameter because it is strictly a server-side setting but it will disable its TCP_NODELAY setting. As you can see, a single InvokerLocator can provide useful information for both ends of the communications channel (client and server).

Agent Communications Services Configuration

The previous sections described the configuration preferences for most of the features available in the agent. The main set of preferences not yet discussed are those configuration settings that define the communications services the agent will create in order to be able to receive commands from the RHQ Server.

There are two main components that are configurable:

  1. Multicast Detector
  2. Connector

Multicast Detector

The Multicast Detector is an optional service and is only needed if server auto-detection is enabled (rhq.agent.server-auto-detection=true). This service is responsible for listening for multicast heartbeat messages from remote RHQ Servers running on the network. As it detects the RHQ Server coming up and down, it will notify the agent. When the agent is told its RHQ Server has gone down, the agent will immediately stop sending messages. It will begin spooling guaranteed commands to disk and continue queueing volatile commands in case the RHQ Server comes back up. When the Multicast Detector sees the RHQ Server come back online, it will notify the agent which will then begin to start sending messages again.

You can configure the multicast address and port the detector will use. You can also configure the amount of time must pass without hearing from the RHQ Server before that server is considered dead.

Connector

The Connector is the server-side component responsible for opening up the agent's socket that will accept incoming commands from the RHQ Server. You can configure what transport the connector will use, what address the socket will bind to and what port it listens to. There are additional transport parameters you can define; such as what address the agent's client (aka RHQ Server) should use to connect to the agent, what the socket timeout should be, etc. These transport params are specific to the type of transport is being used.

These parameters are actually features supplied by the underlying JBoss/Remoting infrastructure used by the RHQ communciations layer - you can find more detail on all available transport params in the JBoss/Remoting configuration documentation.

Communications Security

See the Securing Communications page for detailed information on how you can secure the communications channel between RHQ Agents and RHQ Servers.
Powered by a free Atlassian Confluence Open Source Project License granted to Hyperic HQ. Evaluate Confluence today.
Powered by Atlassian Confluence 2.7.1, the Enterprise Wiki. Bug/feature request - Atlassian news - Contact administrators